WO2009098968A1

WO2009098968A1 - Imaging method and imaging system

Info

Publication number: WO2009098968A1
Application number: PCT/JP2009/051245
Authority: WO
Inventors: Koichi Sugiyama; Yuhsuke Satake; Hideo Uchida; Masato Iida
Original assignee: Sharp Kabushiki Kaisha
Priority date: 2008-02-07
Filing date: 2009-01-27
Publication date: 2009-08-13

Abstract

When a bluish illumination light is applied to an object (OBJ), each of sample points is shifted in the blue direction as a whole on the xy chromaticity coordinate system (lower left direction on the xy chromaticity coordinate system). That is, by changing an illumination environment, the sample point which has existed out of the color region (a point of yellow color of relatively high purity) is shifted to a yellow color of a lower purity and the point is contained in the color region which can be captured by an imaging device. As a result, all the colors of the object (OBJ) can be captured in the color region which can be captured by the imaging device. This enables accurate estimation of the spectral reflectance of the object (OBJ) and reproduction of the color out of the color region.

Description

Imaging method and imaging system

The present invention relates to an imaging method and an imaging system in which the spectral reflectance of a subject is taken into consideration, and more particularly to a technique for imaging in consideration of a color gamut which can be imaged in an imaging apparatus.

In recent years, techniques have been proposed for accurately reproducing the color of a subject imaged under various environments in an output device such as a display device or a printing device.

As a representative technology, color management technology based on the spectral reflectance of an object is known. This technology is realized by treating the color of the subject as a spectrum, and accurate color reproduction is possible without being affected by the illumination environment of the subject. About the color reproduction based on the spectral reflectance of such a subject, its principled processing is disclosed in “Mr. Yoichi Miyake,“ Introduction to spectral image processing ”, The University of Tokyo Press, February 24, 2006”. It is done.

By the way, the spectral reflectance of an object is generally measured using a spectral colorimeter or the like under a certain illumination environment. This is because when the subject is imaged by the imaging device, light needs to be emitted from the subject (that is, the illumination light is reflected by the subject), and the color of the subject is not illuminated at all. It is because it can not judge. In addition, in the case of estimating the spectral reflectance of the subject from the imaging data captured by the imaging device, it is necessary to accurately specify the illumination environment which has been irradiated to the subject at the time of imaging.

Japanese Patent Laid-Open No. 2000-046648 (Patent Document 1) discloses a configuration for estimating a spectral reflectance of a subject and generating a simulation image when the illumination light is changed.
Japanese Patent Laid-Open No. 2000-046648 Miyake Yoichi, "Introduction to Spectral Image Processing", The University of Tokyo Press, February 24, 2006

By the way, a general imaging device includes a plurality of imaging elements (typically, charge coupled devices (CCDs) or the like, which have different spectral sensitivities such as R (red), G (green), and B (blue) based on the three primary colors of light. It consists of a CMOS (Complementary Metal Oxide Semiconductor) image sensor). The color gamut that can be imaged by such an imaging device is determined by the spectral sensitivity, the dynamic range, and the like of each mounted imaging device. If a color outside the color gamut is imaged using such an imaging device, it will be falsely detected (clipping) as a color within the closest color gamut.

As described above, when the imaging apparatus outputs erroneously detected imaging data, the color reproduction of the subject can not be performed accurately. Therefore, in the conventional method, there is a problem that the color reproduction can not be accurately performed for an object having a color outside the color gamut that can be imaged by the imaging device.

Therefore, the present invention has been made to solve such problems, and an object thereof is an imaging method and an imaging system capable of accurately reproducing the color of an object even when the color gamut of the imaging device is deviated. To provide.

According to an aspect of the present invention, there is provided an imaging method comprising the steps of: acquiring first imaging data by imaging an object illuminated with illumination light having a specific wavelength characteristic using an imaging device; In contrast, the method includes the steps of generating color information in which the color of the subject is estimated using the wavelength characteristics of the illumination light, and regenerating data in which the subject is imaged based on the color information.

Preferably, in the step of generating color information, the spectral radiance of the illumination light is acquired as the first spectral radiance, and the spectral reflectance of the subject is estimated using the first spectral radiance and the first imaging data. Generating the first color data on the first color space indicating the color of the subject using the spectral reflectance and the first spectral radiance, and the one-color data is within a color range that can be captured by the device Evaluating whether or not one color data is not present in a possible color range, irradiating illumination light having a specific principal wavelength toward the subject, and illuminating light The method further comprises the steps of again photographing the subject during the irradiation and obtaining the second imaging data, and acquiring, as the second spectral radiance, the spectral radiance of the light irradiated to the subject during the irradiation of the illumination light. In the regenerating step, the spectral reflectance of the subject is estimated using the second spectral radiance and the second imaging data, and the first spectral radiance is used to indicate the color of the subject on the first color space. Regenerating the first color data.

More preferably, the evaluating step determines the illumination light so that the one-color data falls within a possible color range based on where the first color data is out of the color gamut on the first color space. Including the step of

Preferably, the method further includes the step of converting one-color data into second-color data on a second color space and outputting the converted data as image data.

More preferably, the first color space is an XYZ color system, and the second color space is an RGB color system.

An imaging system according to another aspect of the present invention includes an imaging device for outputting imaging data consisting of respective output values of an imaging device, and an illumination device capable of selectively irradiating a plurality of illumination lights having different principal wavelengths toward an object. And a spectral radiance acquisition unit for acquiring spectral radiance of light irradiated to the subject, spectral radiance and data to estimate the spectral reflectance of the subject, and then using the spectral reflectance to measure the color of the subject. A first color conversion unit for generating first color data on a first color space, and an evaluation unit for evaluating whether one-color data is present in a color range that can be captured by the device. The evaluation unit causes the lighting device to emit specific illumination light toward the subject when it is evaluated that one-color data is not present in a possible color range.

Preferably, the evaluation unit evaluates at which position on the first color space the one-color data is out of the color gamut, and specifies so that the one-color data falls within the possible color gamut according to the position. Select the illumination light of.

Preferably, the first color conversion unit uses the spectral radiance acquired in the second illumination environment when the first illumination environment to be applied when capturing an object is switched to the second illumination environment by the evaluation unit. The spectral reflectance is estimated, and further, the spectral reflectance and the spectral radiance obtained in the first illumination environment are used to generate first color data.

Preferably, the image forming apparatus further includes a second color conversion unit that converts the first color data after conversion in the first color conversion unit into second color data on a second color space and outputs the second color data as image data.

An imaging method according to still another aspect of the present invention comprises the steps of acquiring first imaging data by imaging an object under a first illumination environment using an imaging device, and irradiating the object under the first illumination environment Obtaining a first spectral radiance which is a spectral radiance of the light, and estimating the spectral reflectance of the object using the first imaging data and the first spectral radiance, and Generating first color data on a first color space indicating the color of the subject using the first spectral radiance; and imaging the subject under a second illumination environment using the imaging device, (2) acquiring imaging data, acquiring second spectral radiance which is spectral radiance of light irradiated to the subject under the second illumination environment, second imaging data and first spectral radiance Use Generating a second color data on a first color space indicating the color of the subject using the spectral reflectance and the first spectral radiance after estimating the spectral reflectance of the subject; Generating composite image data based on the first color data corresponding to the data and the second color data corresponding to the remaining pixels excluding the first color data.

Preferably, generating the second color data includes generating the second color data only for the remaining pixels.

Preferably, the imaging method further includes the step of evaluating whether or not the first color data for each pixel is within a color gamut that can be imaged by the imaging device. The step of evaluating classifies the pixels corresponding to the first color data present in the color range that can be imaged by the imaging device into the pixels of the first group, and the first color data that does not exist in the color range that can be imaged by the imaging device And categorizing the pixels corresponding to to a second group of pixels.

More preferably, in the evaluation step, when there is one of the first color data that exists outside the color gamut that can be captured by the imaging device, a plurality of illumination lights having different dominant wavelengths are selectively irradiated toward the subject. Forming a second illumination environment by directing specific illumination light from the possible illumination devices towards the subject.

More preferably, the step of evaluating selects a specific illumination light such that the color of the pixel corresponding to the first color data present outside the color gamut that can be captured by the imaging device falls within the color gamut that can be captured by the imaging device. Including the step of

More preferably, the step of selecting the illumination light is performed by evaluating at which position on the first color space the first color data present outside the color gamut that can be captured by the imaging device is out of the color gamut. Selecting the specific illumination light according to the position.

Preferably, the method further includes the step of converting the composite image data in the first color space into image data in the second color space.

Preferably, the first color space is an XYZ color system, and the second color space is an RGB color system.
In an imaging method according to another aspect of the present invention, a first imaging data is acquired by imaging an object using an imaging device, and the first color data included in the first imaging data can be imaged by the imaging device. A step of evaluating whether or not the color data is within the color gamut, and when it is evaluated that the first color data is not the color data within the imageable color gamut, illumination light having a specific wavelength is directed to the subject Corresponding to color data evaluated as not being color data within the color range of the first shooting data, and the second lighting step, the second shooting data capturing step, and the second shooting data capturing step. The step of generating third color data by estimating the color of the subject using the specific wavelength characteristic of the illumination light with respect to the second color data of the second imaging data, and imaging available included in the first imaging data Color within the color gamut By substituting the corresponding third color data of the first color data is evaluated not to be, and generating a combined image data.

An imaging apparatus according to still another aspect of the present invention obtains an imaging apparatus that outputs imaging data consisting of output values of a plurality of imaging elements in each pixel according to a subject, and acquires spectral radiance of light irradiated to the subject The spectral reflectance of the subject at each pixel is estimated using the spectral radiance acquisition unit, the first imaging data obtained by imaging the subject under the first illumination environment, and the first spectral radiance under the first illumination environment. A first color conversion unit that generates first color data on a first color space on a first color space that indicates the color of the subject using the spectral reflectance and the first spectral radiance, and the subject under a second illumination environment After the spectral reflectance of the subject at each pixel is estimated using the second imaging data captured at the second position and the second spectral radiance under the second illumination environment, and then using the spectral reflectance and the first spectral radiance The A second color conversion unit that generates, for each pixel, second color data on a first color space indicating a color of a subject; and a first color corresponding to a first group of pixels among a plurality of pixels included in the imaging device And a combining unit that generates combined image data based on the data and the second color data corresponding to the remaining second group of pixels excluding the first group of pixels.

Preferably, the second color conversion unit generates second color data only for the pixels classified into the second group.

Preferably, the imaging device further includes an evaluation unit that evaluates whether or not the first color data for each pixel is within a color range that can be imaged by the imaging device. The evaluation unit classifies the pixels corresponding to the first color data present in the color range that can be captured by the imaging device into the pixels of the first group, and sets the first color data that does not exist in the color range that can be captured by the imaging device. The corresponding pixels are classified into the second group of pixels.

More preferably, the imaging device further includes an illumination device capable of selectively illuminating a plurality of illumination lights having different dominant wavelengths toward the subject. The evaluation unit, when there is one of the first color data that exists outside the color gamut that can be imaged by the imaging device, forms a second illumination environment by irradiating specific illumination light from the illumination device toward the subject.

More preferably, the evaluation unit selects specific illumination light such that the color of the pixel corresponding to the first color data present outside the color gamut that can be captured by the imaging device falls within the color gamut that can be captured by the imaging device. .

More preferably, the evaluation unit evaluates at which position on the first color space the first color data present outside the color gamut that can be captured by the imaging device is out of the color gamut, and is specified according to the position Select the illumination light of.

Preferably, the imaging device further includes a third color conversion unit that converts composite image data in the first color space into image data in the second color space.

Preferably, the first color space is an XYZ color system, and the second color space is an RGB color system.

According to the present invention, the color of the subject can be accurately reproduced even when it is out of the color gamut of the imaging device.

FIG. 1 is a schematic configuration diagram of an imaging system according to a first embodiment of the present invention. It is a figure for demonstrating the outline of the color gamut evaluation process in the evaluation part shown in FIG. It is a figure for demonstrating the outline of the switching process of the illumination light in the evaluation part shown in FIG. It is a flow chart which shows the whole processing procedure in the imaging system according to a 1st embodiment of this invention. It is a flowchart which shows the content of color gamut evaluation processing subroutine (1) shown in FIG. It is a flowchart which shows the content of the illumination environment switching subroutine shown in FIG. It is a schematic block diagram of the imaging system according to the 2nd Embodiment of this invention. It is a figure for demonstrating the outline of the image synthetic | combination processing according to 2nd Embodiment of this invention. It is a figure for demonstrating the data structure stored in the flag table shown in FIG. It is a figure which shows another form of the flag table shown in FIG. It is a flow chart which shows the whole processing procedure in the imaging system according to a 2nd embodiment of this invention. It is a flowchart which shows the content of the color gamut evaluation processing subroutine (2) shown in FIG. It is a schematic block diagram of the computer which implement | achieves the image processing apparatus according to the modification of embodiment of this invention.

Explanation of sign

Reference Signs List 1 imaging system, 100, 100A, 100 # image processing apparatus, 102 linear correction unit, 102a, 122a look-up table (LUT), 104 first color conversion unit, 106 second color conversion unit, 104a, 106a, 130 spectral reflection Ratio estimation unit 104b 106b color reproduction unit 110 120 coordinate conversion unit 112 evaluation unit 114 evaluation table 116 combination unit 118 flag table 122 gamma correction unit 132 primary storage unit 134 color conversion unit 150 Computer main unit, 152 monitor, 154 keyboard, 156 mouse, 162 memory, 164 fixed disk, 166 FD drive, 168 CD-ROM drive, 170 communication interface, 200 imaging device, 300 spectral radiance meter, 400 Illumination device, 402 filter wheel, 404 wavelength filters, 406 motor, 408 a light source, OBJ subject.

Embodiments of the present invention will be described in detail with reference to the drawings. The same or corresponding portions in the drawings have the same reference characters allotted and description thereof will not be repeated.

First Embodiment
<Overall configuration>
FIG. 1 is a schematic configuration diagram of an imaging system 1 according to a first embodiment of the present invention.

Referring to FIG. 1, imaging system 1 captures an object OBJ, performs predetermined image processing (mainly, color reproduction processing) on the captured image data, and then refers to image data (hereinafter referred to as "color reproduction data"). .) Is output. More specifically, the imaging system 1 includes an image processing device 100, an imaging device 200, a spectral radiance meter 300, and an illumination device 400. The color reproduction data output from the image processing apparatus 100 is typically output to an output device (not shown) such as a display device (display) or a printing device (printer). Alternatively, it may be configured to be stored in a storage device (not shown) or the like.

The imaging device 200 images a subject OBJ, and outputs imaging data according to the subject OBJ. Typically, the imaging device 200 is configured by an imaging element such as a CCD or a CMOS image, and outputs imaging data corresponding to the position of each pixel formed in a matrix. More specifically, a plurality of imaging devices having different spectral sensitivities are arranged in association with each pixel, and imaging including detection values (brightness values) of the plurality of imaging devices associated with each pixel Data is output. The number of imaging elements having different spectral sensitivities that are associated with each pixel in this manner is also referred to as the number of bands. In the present embodiment, the case of using the imaging apparatus 200 of three bands in which the respective spectral sensitivities are mainly R (red), G (green), and B (blue) will be representatively described. In principle, in order to capture a color image, three or more bands are sufficient, and in order to enlarge a color gamut to be described later, it is preferable that the number of bands be large.

As described above, the imaging data output from the imaging device 200 is three-dimensional color information of each of the R, G, and B luminance values (typically, 12 bits each: 0 to 4095 gradations). That is, the imaging data output from the imaging device 200 is defined as an RGB colorimetric system. Hereinafter, the image sensor number (band number) corresponding to each pixel in the imaging device 200 is i (i = 1, 2, 3), the two-dimensional coordinates of the pixels are (m, n), and output from the imaging device 200 representing the image data to be _g i (m, n) and. That is,
Brightness value detected by the image sensor of R at g ₁ (m, n) = coordinate (m, n) g ₂ (m, n) = brightness value detected by the image sensor of G at coordinate (m, n) g ₃ (m, n) = a luminance value detected by the image sensor of B at coordinates (m, n). In the following description, the imaging data imaged by the imaging device 200 will also be collectively referred to as “imaging data g”.

The imaging device 200 may be a moving image camera or a still image camera. That is, when the imaging device 200 is a moving image camera, the imaging of the object OBJ and the output of the imaging data g are repeated in a predetermined cycle (typically 1/30 seconds), and the imaging device 200 is a still image camera. In this case, the imaging and the output of the imaging data g are performed as an event according to the operation of the user or the like. Further, the imaging device 200 may be a single-plate camera or a three-plate camera.

The illumination device 400 emits illumination light to the object OBJ when the imaging device 200 captures an image of the object OBJ. In particular, illumination apparatus 400 according to the present embodiment is configured to be able to selectively emit a plurality of illumination lights having different principal wavelengths toward object OBJ in accordance with an instruction from image processing apparatus 100. That is, as described later, as a result of imaging the object OBJ in a predetermined illumination environment, when it is evaluated that the imaging data of the object OBJ exists outside the color gamut that can be imaged by the imaging device 200, the imaging data of the object OBJ is The light emitted from the illumination device 400 to the object OBJ is switched so as to be within the color range that can be captured by the imaging device 200.

Specifically, the lighting device 400 includes a filter wheel 402, a motor 406 that rotationally drives the filter wheel 402, and a light source 408. The filter wheel 402 is provided with a plurality of wavelength filters (color filters) 404 for transmitting predetermined wavelength components. Then, the specific wavelength filter 404 is set on the optical axis of the light source 408 as the motor 406 rotationally drives the filter wheel 402 in accordance with a command from the image processing apparatus 100. As the light source 408, typically, a color rendering light source (standard light source) such as a halogen lamp or a xenon lamp is used. As a result, of the wavelength components emitted from the light source 408, light of the main wavelength corresponding to the selected wavelength filter 404 is emitted to the object OBJ.

As described later, when the imaging data of the object OBJ is out of the color gamut that can be imaged by the imaging device 200, the image processing device 100 evaluates at which position on the color space it is out of the color gamut, The light to be emitted from the illumination device 400 to the object OBJ is switched based on the evaluation result (coordinate position). Here, since the color gamut that can be imaged by the imaging device 200 is determined according to the spectral sensitivity of the imaging element that constitutes the imaging device 200, the characteristics of the wavelength filter (color filter) 404 mounted on the filter wheel 402 It is preferable to determine according to the spectral sensitivity of the element. As an example, illumination apparatus 400 according to the present embodiment includes three wavelength filters 404 of R (red), G (green) and B (blue) as main wavelengths. That is, the lighting device 400 is configured to be able to switch and emit reddish illumination light, greenish illumination light, and bluish illumination light. Note that the illumination device 400 may emit light generated by the light source 408 as standard illumination light without passing through the wavelength filter 404 as described above.

Further, a plurality of illumination lights having different principal wavelengths can be selectively irradiated by using a color temperature conversion filter instead of, or together with the wavelength filter 404.

Furthermore, instead of using the wavelength filter 404 as described above, a plurality of light sources whose main emission peaks are R (red), G (green) and B (blue) are disposed, and these light sources are selectively selected. It may be turned on.

The spectral radiance meter 300 measures the spectral radiance (or spectral irradiance) of the illumination light emitted to the object OBJ, and outputs the measurement result to the image processing apparatus 100. This is a result of the imaging data of the object OBJ imaged by the imaging device 200 as a result of the illumination light irradiated to the object OBJ being reflected according to the spectral reflectance of the object OBJ, and the spectral reflection of the object OBJ This is because the spectral radiance of the illumination light emitted to the object OBJ is required to estimate the rate and perform color management.

In addition, spectral radiance means the energy amount for every wavelength of illumination light per unit projected area and per unit solid angle. Alternatively, spectral irradiance, which is the amount of energy per wavelength of illumination light per unit area, may be used.

In imaging system 1 according to the present embodiment, object OBJ is imaged using imaging device 200. Therefore, when the spectral radiance of the illumination light is actually measured, the arrangement position of spectral radiance meter 300, etc. are devised I need it. That is, the spectral radiance meter 300 measures spectral radiance by receiving, at the light receiving unit, illumination light substantially the same as the illumination light irradiated to the object OBJ. Here, since it is assumed that it is not preferable in many cases that the spectral radiance meter 300 itself is imaged, the spectral radiance meter 300 is outside the imaging range of the imaging device 200 and is the same as the illumination environment of the object OBJ. It is preferable to arrange in the position which can be regarded as the lighting environment of Such an arrangement can be realized more easily by widening the irradiation range of the illumination light of the illumination device 400 as compared with the imaging range of the imaging device 200.

When the spectral radiance meter 300 can not be arranged at a position close to the object OBJ, a standard white plate having a known spectral reflectance at a position where the illumination light substantially the same as the illumination light irradiated to the object OBJ is irradiated The light reflected by the standard white plate may be received by the light receiving unit of the spectral radiance meter 300.

The spectral radiance meter 300 includes, as its light receiving section, a diffraction grating, a photodiode array, and the like, and the photodiode array outputs a signal indicating the radiance for each wavelength. Furthermore, by mounting a milky white diffusion plate (also called a cosine collector, a cosine diffuser, or a cosine receptor) whose spectral transmittance is known to the light receiving portion, spatial brightness of light incident on the diffraction grating is obtained. As a result, it becomes possible to measure the spectral irradiance while reducing the unevenness (light and shade) of the light.

In this embodiment, discrete values sampled at a predetermined wavelength width (typically, 1 nanometer width) are used as spectral radiances. Therefore, when the spectral radiance meter 300 has a visible light range (380 to 780 nanometers) as a detection range, the spectral radiance E output from the spectral radiance meter 300 has wavelengths λ = 380, 381, ... · It becomes 401-dimensional data including a total of 401 luminance values of 780.

<Functional Configuration of Image Processing Device>
Next, the functional configuration of the image processing apparatus 100 will be described. The image processing apparatus 100 estimates the spectral reflectance of the object OBJ using the spectral radiance measured by the spectral radiance meter 300, and uses the estimated spectral reflectance to specify an image defined on the RGB color space. Data g _i (m, n) is converted into image data g ′ _XYZ (m, n) of the _XYZ color system, and each image data g ′ _XYZ (m, n) is in a color range that can be captured by the imaging device 200 Determine if it exists. Then, if any image data g ′ _XYZ (m, n) exists outside the color gamut that can be imaged by the imaging device 200, a command is given to the illumination device 400 to switch the illumination environment for the object OBJ. Then, color reproduction data of the RGB colorimetric system is generated and output based on the imaging data g captured under an appropriate illumination environment.

Specifically, the image processing apparatus 100 includes a linear correction unit 102, a spectral reflectance estimation unit 130, a color conversion unit 134, coordinate

conversion units

110 and 120, a gamma correction unit 122, and an evaluation unit 112. An evaluation table 114 and a primary storage unit 132 are included.

The linear correction unit 102 is a part for linearizing the image data g output from the imaging device 200. Generally, in a display device, there is a non-linear relationship between an input signal level and an actually displayed luminance level, and such non-linearity is referred to as a gamma characteristic. An image that has an inverse nonlinearity to the gamma characteristic of the display device from the imaging device 200 so that an image adapted to human vision is displayed on the display device by canceling the non-linearity in such a display device Data g is often output. As described above, when the signal level of the image data g and the luminance level detected by each imaging device are non-linear, the image processing described later can not be accurately performed, and thus the linear correction unit 102 This image data g is linearized.

Since generally the gamma characteristics and inverse gamma characteristic can be expressed as a power function, linear correction unit 102, when γc inverse gamma value in the image pickup apparatus 200, according to the following such calculation formula, image data g _i Convert (m, n) into linearized linear image data gc _i (m, n).

gc _i (m, n) = g _i (m, n) ^{1 / γ c}
Here, the inverse gamma value γc is set to a value corresponding to the reciprocal of the gamma value γd of the display device. As an example, when the gamma value γd of the display device is 2.2, the inverse gamma value γc of the imaging device 200 is 1 / γd = 1 / 2.2 ≒ 0.45.

The image data g output from the imaging device 200 by one imaging operation is the same as the number of pixels of the imaging device 200. For example, when the pixels of the imaging device 200 are 1920 × 1080, The number is 2073600. Therefore, if the conversion equation as described above is executed as it is, the amount of calculation becomes enormous. If the processing capacity of the image processing apparatus 100 is sufficiently high, the calculation may be directly executed, but if there is a limit to the processing capacity, a Look-Up Table (LUT) is used. Is effective.

The linear correction unit 102 according to the present embodiment stores the look-up table 102 a in advance, and performs linearization processing by referring to the look-up table 102 a. The look-up table 102a is a data table in which the result of the above-described conversion equation is stored in advance in association with each of all possible signal levels of the input image data g. In the actual linearization operation, the conversion amount corresponding to the signal level of the input image data g may be acquired with reference to the look-up table 102a, so the amount of operation can be significantly reduced.

Since the contents of the look-up table 102a differ depending on the inverse gamma value γc, when the inverse gamma value γc is changed, it is also necessary to change the contents of the look-up table 102a.

When the image data g output from the imaging device 200 is linear, that is, the inverse gamma value γc = 1, it is not necessary to provide the linear correction unit 102.

<Color reproduction processing>
Next, the contents of the color reproduction process executed by the first color conversion unit 104 and the second color conversion unit 106 will be described. In the following, the linear image data gc ⁽¹⁾ _i (m, n) and gc ⁽²⁾ _i (m, n) are also collectively referred to as “linear image data gc”.

The light (spectrum) from the object OBJ incident on the pixels of the two-dimensional coordinates (m, n) of the imaging device 200 is the spectral radiance E (λ) of the illumination light irradiated to the object OBJ and the pixels of the object OBJ Corresponds to the product of the spectral reflectance f (m, n; λ) at the position corresponding to The linear image data gc output from the corresponding imaging element according to the light incident on each pixel is further multiplied by S _i (λ), which is the spectral sensitivity of each imaging element, and then, over the wavelength region It corresponds to the integration of light energy. Such a relationship can be expressed as a relational expression shown in equation (1).

Here, n _i (m, n) is additive noise generated by white noise or the like appearing in each imaging device, and is a value depending on the characteristics of the imaging device or lens of the imaging device 200 and the surrounding environment.

As described above, in this embodiment, since the spectral radiance E (λ) sampled at a predetermined wavelength width (typically, 1 nanometer width) is used, the integral equation of the first term of the right side of equation (1) Is calculated by product-sum operation of discrete values obtained by sampling with a predetermined wavelength width. That is, the integral expression of the first term on the right side of the equation (1) is a matrix S indicating the spectral sensitivity at each wavelength of each imaging device, a matrix E indicating the spectral radiance at each wavelength, and a spectral reflectance at each wavelength It is realized by matrix operation with a matrix f (m, n) shown.

Typically, when the visible light region (380 to 780 nanometers) is sampled at 1 nanometer width, the matrix S is a matrix of 401 rows × 3 columns, and the matrix E is a matrix of 401 rows × 1 columns The matrix f (m, n) is a matrix of 1 row × 401 columns.

Here, since the additive noise n _i (m, n) is generally a sufficiently small value, the spectral reflectance f (m, n) of the object OBJ is calculated after being ignored from the equation (1). Think about what to do. Specifically, using the linear image data gc, the spectral reflectance f (m, n) of the object OBJ is calculated according to the following equation (2).

f (m, n) = W g c (m, n) (2)
In equation (2), W is a spectral reflectance estimation matrix. The spectral reflectance estimation matrix W is calculated by the technique of Wiener estimation described below. Specifically, the spectral reflectance estimation matrix W is derived as the equation (4) by defining the system matrix H as the equation (3) shown below and modifying the equation (1).

H = S · E (3)
W = A · H ^t · (H · A · H ^t ) ^-1 (4)
Where H ^t is the transpose of the system matrix H.

In equation (4), A is an autocorrelation matrix, which should be a reference for estimating the spectral reflectance of the object OBJ. The autocorrelation matrix A can be determined in advance using statistical data of spectral reflectance that is considered to be equivalent to the object OBJ. As an example, autocorrelation matrix A can be determined with reference to SOCS (Standard Object Color Sample), which is a database of spectral reflectances standardized in International Organization for Standardization (ISO). Alternatively, if the material of the object OBJ is known in advance, the spectral reflectance of the object OBJ itself may be measured in advance by another method, and the autocorrelation matrix A may be determined. As described above, when the visible light region (380 to 780 nanometers) is sampled at 1 nanometer width, the autocorrelation matrix A is a matrix of 401 rows × 1 column. Further, for details of the winner estimation, refer to “Mr. Yoichi Miyake,“ Introduction to spectral image processing ”, The University of Tokyo Press, February 24, 2006” above, for details.

Note that, instead of the above-described winner estimation method, a principal component analysis method may be used.
As described above, after the spectral reflectance estimation matrix W is calculated according to the equations (3) and (4), the matrix f representing the spectral reflectance of the object OBJ from the linear image data gc according to the equation (2) m, n) can be calculated. The matrix f (m, n) is the essence of the color of the object OBJ, and by using this matrix f (m, n), the object OBJ can be observed under any illumination environment. Color reproduction can be performed.

Next, image processing apparatus 100 according to the present embodiment can generate color reproduction data gd of object OBJ when observed in an arbitrary illumination environment. More specifically, the tristimulus values X, Y, of the XYZ color system when an object of spectral reflectance f (m, n; λ) is observed under the condition of arbitrary spectral radiance E (λ) Z becomes like (5) Formula shown below.

In equation (5), h _i (λ) (i = 1, 2, 3) is a color matching function, which is a function corresponding to human visual sensitivity characteristics. The color matching function h _i (λ) is defined by the International Commission on Illumination (CIE). Hereinafter, three-dimensional data including tristimulus values X, Y and Z calculated by the equation (5) will be expressed as image data g ′ _XYZ (m, n).

Similar to the calculation process of the spectral reflectance described above, calculation is performed by product-sum operation of discrete values obtained by sampling at a predetermined wavelength width (typically, 1 nanometer width). That is, each integral formula of equation (5) is represented by a matrix h representing values of color matching functions at each wavelength, a matrix E representing spectral radiance at each wavelength, and a matrix f (m (m) representing spectral reflectance at each wavelength , N) (= W · gc (m, n)). Typically, when the visible light region (380 to 780 nanometers) is sampled at 1 nanometer width, the matrix h is a matrix of 401 rows × 3 columns, and the matrix E is a matrix of 401 rows × 1 columns The matrix f (m, n) is a matrix of 1 row × 401 columns.

Therefore, image data g ′ _XYZ (m, n) indicating the color of the object OBJ observed in an arbitrary illumination environment (spectral radiance E) is expressed by equation (6).

g ' _XYZ (m, n) = h. E. f (m, n) = h. E. W. gc (m, n) (6)
Here, the matrix E indicating the spectral radiance used in the equation (6) can be set to any value. This means that the color of the subject OBJ to be observed when it is set to a different arbitrary illumination environment can be reproduced based on the image data g obtained by imaging the object OBJ under a certain illumination environment. Therefore, in order to reproduce the color of the object OBJ that would be observed under a specific illumination environment, it is not necessary to image the object OBJ under the specific illumination environment, but it is necessary to perform imaging under another illumination environment. It means that the processed image data g may be used. In other words, as long as the object OBJ can be imaged so as to be within the imageable color range in the imaging device 200, colors that would be observed under any illumination environment can be accurately reproduced. Therefore, imaging system 1 according to the present embodiment has an object of imaging object OBJ so as to be within a color range that can be imaged in imaging device 200.

Again referring to FIG. 1, the spectral reflectance estimation unit 130 indicates the spectral radiance E measured by the spectral radiance meter 300 and the spectral sensitivity at each wavelength of the imaging device of the imaging device 200 stored in advance. Using the matrix S and the autocorrelation matrix A stored in advance, the matrix calculation of the equations (3) and (4) is executed to calculate the spectral reflectance estimation matrix W. Then, the spectral reflectance estimation unit 130 outputs the calculated spectral reflectance estimation matrix W to the color conversion unit 134.

Although it is desirable that the cycle in which the spectral reflectance estimation unit 130 calculates the spectral reflectance estimation matrix W is the same as the imaging cycle in the imaging device 200, the measurement cycle of the spectral radiance by the spectral radiance meter 300 is imaged If it is later than the period, the spectral reflectance estimation matrix W may be calculated at the measurement period of the spectral radiance. Alternatively, when the illumination environment of the object OBJ hardly changes, the spectral reflectance estimation matrix W may be calculated only when the spectral radiance output from the spectral radiance meter 300 changes.

The color conversion unit 134 includes the linear image data gc output from the linear correction unit 102, the spectral reflectance estimation matrix W output from the spectral reflectance estimation unit 130, and the spectral radiance measured by the spectral radiance meter 300. The matrix operation of the above equation (6) is executed using E and a matrix h indicating values of color matching functions at each wavelength stored in advance to calculate image data g ′ _XYZ .

A primary storage unit 132 is disposed between the spectral radiance meter 300 and the color conversion unit 134. The primary storage unit 132 temporarily stores the spectral radiance before switching when the evaluation unit 112 switches the illumination light from the illumination device 400 as described later, and the spectral radiance before switching is Function as a buffer for outputting to the color conversion unit 134. The detailed operation of the primary storage unit 132 will be described later.

Coordinate conversion unit 120 converts 'the _XYZ image data g of the RGB color system' image data g of the XYZ color system generated by the color conversion unit 134 into _RGB, and outputs it to the gamma correction unit 122. That is, the coordinate conversion unit 120 converts the color space of the image data g ′ _RGB and outputs it.

The gamma correction unit 122 maintains the linearity of the _RGB color system image data g ' _RGB converted by the coordinate conversion unit 120. Therefore, when outputting to a display device or the like, the gamma characteristic of the display device In order to take account of this, it is necessary to perform correction in advance to cancel the gamma characteristic of the display device. That is, the gamma correction unit 122 performs conversion processing opposite to the conversion processing in the linear correction unit 102. More specifically, the gamma correction unit 122 performs color reproduction data gd _{RGB on} the image data g ′ _RGB (m, n) of the _RGB colorimetric system according to the following arithmetic expression using the gamma value γd of the display device: Convert to (m, n).

gd _RGB (m, n) = g ' _RGB (m, n) ^{γ d}
Here, the gamma value γd of the display device is set to “2.2” or the like as an example.

Similar to the linear correction unit 102 described above, the gamma correction unit 122 according to the present embodiment performs arithmetic processing using a look-up table (LUT). More specifically, the gamma correction unit 122 stores the look-up table 122a in advance, and performs gamma correction processing by referring to the look-up table 122a. The look-up table 122a is a data table in which the result of the above-mentioned conversion equation is stored in advance in association with each of all possible signal levels of the input image data g ′ _RGB . In the actual linearization operation, the conversion amount corresponding to the input image data g ′ _RGB signal level may be acquired with reference to the look-up table 122 a, so the amount of operation can be significantly reduced.

Since the contents of the look-up table 122a differ depending on the gamma value γd, when the gamma value γd is changed, the contents of the look-up table 122a also need to be changed.

<Color gamut evaluation processing>
The evaluation unit 112 evaluates whether or not the image data g captured by the imaging device 200 is captured within the imageable color range of the imaging device 200 in the process of generating the color reproduction data gd as described above. Do. Then, when the evaluation unit 112 evaluates that the image data g includes a color outside the imageable color range, the lighting device may be configured such that the color emitted by the object OBJ falls within the imageable color range of the imaging device 200. A specific illumination light is emitted from 400. Since it is necessary to evaluate whether or not it is in such a color range in the color space of the XYZ color system, the evaluation unit 112 evaluates the image data g ′ _XYZ output from the color conversion unit 134. Do.

FIG. 2 is a diagram for explaining an outline of the color gamut evaluation process in the evaluation unit 112 shown in FIG.

As shown in FIG. 2, tristimulus values X, Y and Z of the XYZ color system are defined by two variables x and y obtained by the conversion equations as shown in equations (7) and (8). It can be expressed as two-dimensional coordinates (hereinafter also referred to as "xy coordinates").

x = X / (X + Y + Z) (7)
y = Y / (X + Y + Z) (8)
Further, FIG. 2 shows a chromaticity diagram in the XYZ color system. In this chromaticity diagram, it means that the center is white, and as it goes to the outer periphery, it is a color whose "purity" is higher. Also, B (blue), G (green) and R (red) are mainly located at each vertex of the substantially triangular shape.

As shown in FIG. 2A, the imageable color gamut of the imaging device 200 including three imaging elements of R, G, and B has colors (purity) detectable by each imaging element in xy coordinates. It will be a triangle to be a vertex. When the number of imaging elements (the number of bands) is increased, the number of polygons is the same as the number of bands.

Moreover, in FIG. 2A, the color imaged by each pixel of the imaging device 200 under the first illumination environment is illustrated as a sample point. Each sample point is obtained by plotting the image data g ′ ⁽¹⁾ _XYZ of each pixel calculated by the above-described color reproduction process on xy coordinates.

Of the sample points shown in FIG. 2A, consider the case where the original colors of two sample points exist outside the imageable color gamut. In this case, the color of the sample point outside the color gamut is falsely detected (clipped) as a color within the color gamut closest to the imageable color gamut. That is, a color present outside the imageable color gamut is detected as another color whose purity is lowered. Specifically, the colors of the two sample points shown in FIG. 2A are originally relatively high in purity yellow, but are detected as more reddish.

As described above, if the color information of the imaging data g is erroneously detected in the imaging device 200, the spectral reflectance can not be accurately calculated in the above-described color reproduction process. As a result, color reproduction can not be performed accurately.

Here, the light (color) emitted from the object OBJ changes according to the illumination environment, so by switching the illumination environment appropriately, sample points existing outside the imageable color gamut can be captured within the imageable color range It can be paid.

FIG. 2B is a diagram showing sample points in the case where an object OBJ similar to that of FIG. 2A is imaged in a more bluish illumination environment. As shown in FIG. 2B, when the bluish illumination light is irradiated by the object OBJ, each sample point moves in the blue direction (lower left direction on the xy coordinates) in xy coordinates as a whole. That is, by changing the illumination environment, sample points existing outside the color gamut in FIG. 2A (that is, points having a relatively high purity yellow) emit a yellow having a lower purity. As a result, the image can be captured within the imageable color range of the imaging device 200. As a result, since the entire object OBJ can be imaged within the color range which can be imaged by the imaging device 200, the spectral reflectance of the object OBJ can be accurately estimated.

Furthermore, as shown in the above-mentioned equation (6), since color reproduction data gd when observed in an arbitrary illumination environment can be generated, it should be imaged under the original illumination environment, not the illumination environment after this switching. It is possible to generate the color reproduction data gd that was.

FIG. 2 (C) generated color reproduction data gd which should have been generated under the illumination environment of FIG. 2 (A) based on the imaging data imaged under the illumination environment of FIG. 2 (B) Indicates the case. By performing such color reproduction processing, even if the color is out of the color gamut that can be imaged by the imaging device 200, color reproduction can be performed accurately.

In order to appropriately switch the illumination light from the illumination device 400, it is necessary to specify a color outside the imageable color gamut of the imaging device 200. Therefore, evaluation unit 112 according to the present embodiment evaluates at which position on the xy coordinates the color is out of the color gamut, and selects the illumination light according to this position.

Note that it is not necessary to evaluate all the pixels of the captured image data g, and among color information of pixels included in the image data g, it is possible to evaluate a predetermined number of pixels representing the image data g. Good.

FIG. 3 is a view for explaining an outline of switching processing of illumination light in the evaluation unit 112 shown in FIG. Referring to FIG. 3, on the xy coordinates, the coordinates corresponding to the red of the maximum purity that can be imaged by the R imaging device is (x1, y1), and corresponds to the green of the maximum purity that can be imaged by the G imaging device Let (x2, y2) be the coordinates to be taken, and let (x3, y3) be the coordinates corresponding to the blue of the maximum purity that can be imaged by the B image sensor.

Then, if there is a sample point on the linear region (a) connecting the coordinates (x1, y1) and the coordinates (x2, y2), it is judged that the purity of Y (yellow) is high, and this Y (yellow) It switches to the illumination light which contains many components of B (blue) which is a complementary color of.

Similarly, if there is a sample point on the linear region (b) connecting the coordinates (x2, y2) and the coordinates (x3, y3), it is judged that the purity of C (cyan) is high, It switches to the illumination light which contains many components of R (red) which is a complementary color of cyan).

Similarly, if there is a sample point on the straight line area (c) connecting the coordinates (x3, y3) and the coordinates (x1, y1), it is judged that the purity of M (magenta) is high, It switches to the illumination light which contains many components of G (green) which is a complementary color of magenta).

As described above, the evaluation unit 112 evaluates at which position on the xy coordinate the color included in the image data g ′ ⁽¹⁾ is out of the color gamut, and selects the illumination light according to the position.

Again referring to FIG. 1, coordinate conversion unit 110 converts the respective components of image data g ′ sequentially output from color conversion unit 134 into the conversion equations shown in the above-mentioned equations (7) and (8). Perform coordinate transformation according to. Then, the coordinate conversion unit 110 sequentially outputs the converted coordinate values (x, y) to the evaluation unit 112.

The evaluation unit 112 stores coordinate values (x, y) sequentially output from the coordinate conversion unit 110 by the number corresponding to one imaging by the imaging device 200, and the imaging device 200 defined in advance on xy coordinates. It is determined whether or not there is something outside the color gamut that can be imaged. Note that the color gamut that can be imaged by the imaging device 200, which is defined in advance on the xy coordinates, is stored in advance in the evaluation table 114. Further, since the color gamut that can be imaged by the imaging device 200 depends on the characteristics of the imaging elements that constitute the imaging device 200, the type or individual of the imaging device 200 is acquired in advance by experiment or simulation.

Furthermore, in the case where there is a coordinate value (x, y) output from the coordinate conversion unit 110, the evaluation unit 112 detects that the coordinate value (x, y) exists outside the color gamut that can be captured by the imaging device 200. y) Evaluate at which position the color gamut is out. Then, the evaluation unit 112 outputs a selection command of the illumination light to the lighting device 400 according to the position. As described above, specifically, the evaluation unit 112 determines whether the coordinate value (x, y) exists in any linear region connecting the respective coordinates corresponding to R, G, B on the xy coordinates. Judging by

When the selection command of the illumination light is output to the illumination device 400, the evaluation unit 112 maintains the output of the spectral radiance at the relevant time point (before switching of the illumination light) to the primary storage unit 132. Give a command to This is to allow the color conversion unit 134 to output the image data g ′ that should have been captured under the illumination environment before switching (that is, under the original illumination environment).

That is, when the illumination environment (spectral radiance E ⁽¹⁾ ) to be applied at the time of imaging the object OBJ is switched to a different illumination environment (spectral radiance E ⁽²⁾ ), the evaluation unit 112 The primary storage unit 132 is controlled to calculate the image data g ′.

H = S · E ⁽¹⁾
W = A · H ^t · (H · A · H ^t ) ^-1
g ' _XYZ (m, n) = h · E ⁽²⁾ · W · gc (m, n)
Further, the evaluation unit 112 performs coordinate conversion in the coordinate conversion unit 120 when none of the coordinate values (x, y) output from the coordinate conversion unit 110 exists outside the color gamut that can be captured by the imaging device 200. While giving a command for validating the process, if there is one that exists outside the color gamut that can be captured by the imaging device 200, a command for invalidating the coordinate conversion processing in the coordinate conversion unit 120 is given. This is to prevent output of low-reliability color reproduction data gd generated based on image data including colors outside the color gamut that can be imaged by the imaging device 200.

<Processing procedure>
The processing procedure in the imaging system 1 according to the present embodiment is summarized as follows.

FIG. 4 is a flowchart showing the overall processing procedure in the imaging system 1 according to the first embodiment of the present invention. FIG. 5 is a flowchart showing the contents of the color gamut evaluation processing subroutine (1) shown in FIG. FIG. 6 is a flowchart showing the contents of the lighting environment switching subroutine shown in FIG. In the following description, in order to clarify the processing content, data etc. obtained by the first imaging are indicated by superscript ⁽¹⁾ and obtained by the second imaging Data and the like are indicated with superscript ⁽²⁾ .

Referring to FIGS. 1 and 4, first, imaging device 200 captures an object OBJ, and outputs imaging data g ⁽¹⁾ according to object OBJ (step S2). The imaging device 200 performs an imaging operation in response to an operation by a user or the like. Subsequently, the spectral reflectance estimation unit 130 acquires the spectral radiance E ⁽¹⁾ measured by the spectral radiance meter 300 (step S4), and calculates the spectral reflectance estimation matrix W ⁽¹⁾ (step S6). ).

Subsequently, the linear correction unit 102 linearizes the image data g ⁽¹⁾ _i (m, n) corresponding to one pixel output from the imaging device 200 to generate linear image data gc ⁽¹⁾ _i (m, n ^). ) Is generated (step S8). Then, the color conversion unit 134 outputs the spectral radiance E ⁽¹⁾ acquired in step S4, the spectral reflectance estimation matrix W ⁽¹⁾ calculated in step S6, and the linear image data gc generated in step S8. ⁽¹⁾ Image data g ' ⁽¹⁾ _XYZ (m, n) is calculated using _i (m, n) (step S10). Furthermore, it is determined whether the processing for all the pixels included in the imaging data g ⁽¹⁾ is completed (step S12), and the processing for all the pixels is not completed (NO in step S12), The processes after step S8 are executed again.

On the other hand, if processing for all the pixels is completed (YES in step S12), image data g ' ⁽¹⁾ _XYZ (m, n) exists in a color range that can be captured by imaging device 200. In order to evaluate whether or not the color gamut evaluation processing subroutine (1) is executed (step S14). As described later, when the evaluation result of the color gamut evaluation processing subroutine (1) in step S14 is output as a "flag" to a memory area provided in the evaluation unit 112 or the like, and the flag is set, This means that the image data g ′ ⁽¹⁾ _XYZ (m, n) exist outside the color gamut that can be imaged by the imaging device 200.

After execution of the color gamut evaluation processing subroutine (1), the evaluation unit 112 determines whether any flag is set, that is, the image data g ′ ⁽¹⁾ _XYZ (outside the color gamut that can be imaged by the imaging device 200 ^). It is determined whether m, n) exists (step S16). When the flag is set (in the case of YES in step S16), the evaluation unit 112 stores the current spectral radiance E ⁽¹⁾ in the primary storage unit 132 (step S18) and from the lighting device 400 In order to switch the illumination light, the illumination environment switching subroutine is executed (step S20). After execution of the illumination environment switching subroutine, the imaging device 200 captures an image of the object OBJ again, and outputs imaging data g ⁽²⁾ according to the object OBJ (step S22). Subsequently, the spectral reflectance estimation unit 130 acquires the spectral radiance E ⁽²⁾ measured by the spectral radiance meter 300 (step S24), and calculates the spectral reflectance estimation matrix W ⁽²⁾ (step S26). ).

Subsequently, the linear correction unit 102 linearizes the image data g ⁽²⁾ _i (m, n) corresponding to one pixel output from the imaging device 200 to generate linear image data gc ⁽²⁾ _i (m, n ^). ) Is generated (step S30). Then, the color conversion unit 134 generates the spectral radiance E ⁽¹⁾ stored in the primary storage unit 132 in step S18, the spectral reflectance estimation matrix W ⁽²⁾ calculated in step S26, and the signal generated in step S30. Image data g ' ⁽²⁾ _XYZ (m, n) is calculated using linear image data gc ⁽²⁾ _i (m, n) (step S32). Further, it is determined whether or not processing for all pixels included in imaging data g ⁽²⁾ is completed (step S34), and processing for all pixels is not completed (NO in step S34) The process after step S30 is executed again.

On the other hand, if the process for all the pixels is completed (YES in step S34), coordinate conversion unit 120 converts image data g ′ ⁽²⁾ _XYZ (m, n) calculated in step S32 RGB color system image data g ′ _RGB (m, n) is converted (step S 54), and then the gamma correction unit 122 converts the image data g ′ _RGB (m, n) into color reproduction data gd _RGB (m , N) (step S56) and externally output.

On the other hand, when the flag is not set (in the case of NO in step S16), the coordinate conversion unit 120 converts the image data g ′ ⁽¹⁾ _XYZ (m, n) calculated in step S10 into the RGB color system The image data g ′ _RGB (m, n) is converted into the image data g ′ _RGB (m, n) (step S 58). Subsequently, the gamma correction unit 122 converts the image data g ′ _RGB (m, n) into color reproduction data gd _RGB Convert (step S60), and output it to the outside.

Next, the contents of the color gamut evaluation processing subroutine (1) of step S14 will be described. Referring to FIG. 5, first, coordinate conversion unit 110 converts image data g ′ ⁽¹⁾ _XYZ (m, n) into coordinate values (x, y) on xy coordinates (step S100). Then, the evaluation unit 112 reads the color gamut on the imageable xy coordinates of the imaging device 200 from the evaluation table 114 (step S102).

Then, the evaluation unit 112 corresponds to the coordinates (x1, y1) and G (green) corresponding to R (red) as shown in FIG. 3 as the coordinate values (x, y) calculated in step S100 It is determined whether it exists on a straight line area (a) connecting x2, y2) (step S104). If the coordinate value (x, y) exists on the linear area (a) (YES in step S104), the evaluation unit 112 sets a flag B (blue) (step S106).

When the coordinate value (x, y) does not exist on the linear region (a) (in the case of NO in step S104), the evaluation unit 112 calculates the coordinate value (x, y) calculated in step S100 as shown in FIG. It is determined whether or not it exists on a linear region (b) connecting the coordinates (x2, y2) corresponding to G (green) and (x3, y3) corresponding to B (blue) as shown in FIG. Step S108). If the coordinate value (x, y) exists on the linear region (b) (YES in step S108), the evaluation unit 112 sets a flag R (red) (step S110).

When the coordinate value (x, y) does not exist on the linear region (c) (in the case of NO in step S108), the evaluation unit 112 calculates the coordinate value (x, y) calculated in step S100 as shown in FIG. It is determined whether it exists on a straight line area (c) connecting the coordinates (x3, y3) corresponding to B (blue) and (x1, y1) corresponding to R (red) as shown in FIG. Step S112). If the coordinate value (x, y) exists on the linear region (c) (YES in step S112), the evaluation unit 112 sets a flag G (green) (step S114).

Further, it is determined whether or not processing for all pixels included in imaging data g ⁽¹⁾ is completed (step S116), and processing for all pixels is not completed (NO in step S116) The process after step S100 is executed again.

On the other hand, if the process for all the pixels is completed (YES in step S116), the process proceeds to step S16 in FIG.

Next, the contents of the lighting environment switching subroutine of step S20 will be described. Referring to FIG. 6, evaluation unit 112 first determines whether flag B (blue) is set (step S200). When flag B (blue) is set (YES in step S200), evaluation unit 112 causes illumination light including a large amount of B (blue) component to be emitted from illumination device 400 toward object OBJ. (Step S202).

If flag B (blue) is not set (NO in step S200), evaluation unit 112 determines whether flag R (red) is set (step S204). When the flag R (red) is set (YES in step S204), illumination light including a large amount of R (red) component is emitted from the illumination device 400 toward the object OBJ (step S206).

If the flag R (red) is not set (NO in step S204), the evaluation unit 112 determines whether the flag G (green) is set (step S208). If the flag G (green) is set (YES in step S208), illumination light including a large amount of G (green) components is emitted from the illumination device 400 toward the object OBJ (step S210).

After one of the processes in steps S202, S206, and S210 is performed, the process proceeds to step S22 in FIG.

<Operation and effect according to the present embodiment>
According to the first embodiment of the present invention, it is possible to determine the illumination environment at the time of imaging the subject in consideration of the color gamut that can be imaged by the imaging device. Therefore, since the color of the subject can be correctly captured using the imaging device, the spectral reflectance of the subject can be accurately estimated from the captured imaging data. Therefore, even if the color of the subject exists outside the imageable color gamut of the imaging device in the illumination environment where the subject should be imaged, the illumination environment where the imaging should be performed after the spectral reflectance is accurately estimated. The color that will be imaged (observed) can be properly reproduced.

As a result, in the case where the displayable color gamut of the display device is wider than the imageable color gamut of the imaging device, the display device can also display a color of purity exceeding the imaging capability of the imaging device. It becomes.

Second Embodiment
FIG. 7 is a schematic block diagram of an imaging system 2 according to the second embodiment of the present invention.

7, imaging system 2 images subject OBJ as in imaging system 1 according to the first embodiment, and performs predetermined image processing (mainly, color reproduction processing) on the imaged image data. After the output, image data (color reproduction data) is output. More specifically, in the imaging system 1 according to the first embodiment, the imaging system 2 employs the image processing apparatus 100A in place of the image processing apparatus 100, and the imaging system 1 is used for the other parts. And the detailed description will not be repeated.

The illumination device 400 emits illumination light to the object OBJ when the imaging device 200 captures an image of the object OBJ. In particular, illumination apparatus 400 according to the present embodiment is configured to be able to selectively emit a plurality of illumination lights having different principal wavelengths toward object OBJ in accordance with an instruction from image processing apparatus 100A. That is, as described later, as a result of imaging the object OBJ in a predetermined illumination environment (hereinafter also referred to as “first illumination environment”), the imaging data includes pixels having colors outside the color range that can be imaged by the imaging device 200. When it is determined that the illumination light is emitted, the illumination device 400 emits another illumination light to the object OBJ so that the color of the pixel falls within the color range that can be captured by the imaging device 200. The illumination light from the illumination device 400 forms an illumination environment (hereinafter also referred to as a “second illumination environment”) different from the first illumination environment.

As described later, in the image processing apparatus 100A, when a pixel having a color outside the color gamut that can be imaged by the imaging device 200 is included in the imaging data, the pixel is out of the color gamut at any position on the color space. It is evaluated whether it has become, and the light emitted from the illumination device 400 to the object OBJ is switched based on the evaluation result (coordinate position). Here, since the color gamut that can be imaged by the imaging device 200 is determined according to the spectral sensitivity of the imaging element that constitutes the imaging device 200, the characteristics of the wavelength filter (color filter) 404 mounted on the filter wheel 402 It is preferable to determine according to the spectral sensitivity of the element. As an example, illumination apparatus 400 according to the present embodiment includes three wavelength filters 404 of R (red), G (green) and B (blue) as main wavelengths. That is, the lighting device 400 is configured to be able to switch and emit reddish illumination light, greenish illumination light, and bluish illumination light. In order to form the first illumination environment, the light generated by the light source 408 may be irradiated as it is without passing through such a wavelength filter 404.

<Functional Configuration of Image Processing Device>
Next, the functional configuration of the image processing apparatus 100A will be described. In the following description, in order to clarify the processing content, data etc. obtained by imaging under the first illumination environment is indicated by superscript ⁽¹⁾ , and the second illumination environment The data obtained by the lower imaging and the like are indicated by superscript ⁽²⁾ .

The image processing apparatus 100A uses the imaging data g ^{(1) obtained} by imaging the object OBJ under the first illumination environment and the spectral radiance E ⁽¹⁾ measured under the first illumination environment to generate the object OBJ in each pixel. After the spectral reflectance is estimated, imaging data g ⁽¹⁾ _i (m, n) defined on the RGB color space is XYZ using the estimated spectral reflectance and the spectral radiance E ^(1). Image data of color system g ′ ⁽¹⁾ Convert to _XYZ (m, n).

Then, the image processing apparatus 100A determines whether the image data g ′ ⁽¹⁾ _XYZ (m, n) exist in a color range that can be captured by the imaging device 200. If any image data g ′ ⁽¹⁾ _XYZ (m, n) exists outside the color gamut that can be imaged by the imaging device 200, the second illumination environment is irradiated by irradiating a specific illumination light from the illumination device 400. Form. Furthermore, the image processing apparatus 100A uses the imaging data g ^{(2) obtained} by imaging the object OBJ under the second illumination environment and the spectral radiance E ⁽²⁾ measured under the second illumination environment, and the object at each pixel Image data g ⁽²⁾ _i (m, n) defined on the RGB color space by using the estimated spectral reflectance and spectral radiance E ⁽¹⁾ after estimating the spectral reflectance of OBJ Is converted to image data g ′ of XYZ color system ⁽²⁾ _XYZ (m, n).

Furthermore, the image processing apparatus 100A uses the corresponding image data g ′ ⁽¹⁾ _XYZ (m, n) for the pixels existing in the color range that can be imaged by the imaging device 200, and the remaining pixels, ie, the imaging device For pixels existing outside the color gamut that can be imaged at 200, composite image data G _XYZ (m, n) is synthesized using the corresponding image data g ' ⁽²⁾ _XYZ (m, n). Then, based on the composite image data G _XYZ (m, n), color reproduction data of the RGB color system is generated and output. That is, of the imaging data g ⁽¹⁾ _i (m, n) calculated from the spectral radiance E ⁽¹⁾ measured under the first illumination environment, the image processing apparatus 100A can capture an image with the imaging device 200. By substituting the corresponding image data g ′ ⁽²⁾ _XYZ (m, n) for pixels existing outside the color gamut, composite image data G _XYZ (m, n) is generated.

Specifically, the image processing apparatus 100A includes a linear correction unit 102A, a first color conversion unit 104, a second color conversion unit 106, coordinate

conversion units

110 and 120, an evaluation unit 112A, an evaluation table 114, and the like. , A combining unit 116, a flag table 118, and a gamma correction unit 122.

The linear correction unit 102A is a unit for linearizing the imaging data g output from the imaging device 200. Generally, in a display device, there is a non-linear relationship between an input signal level and an actually displayed luminance level, and such non-linearity is referred to as a gamma characteristic. From the imaging device 200, an image having an inverse nonlinearity to the gamma characteristic of the display device so that an image adapted to human vision is displayed on the display device by canceling the non-linearity in such a display device Data g is often output. As described above, when the signal level of the imaging data g and the luminance level detected by each imaging device are non-linear, the image processing to be described later can not be accurately performed, and thus the linear correction unit 102A This imaging data g is linearized.

Generally, the gamma characteristic and the inverse gamma characteristic can be expressed as a function of a power, and therefore, assuming that the inverse gamma value in the imaging device 200 is γc, the linear correction unit 102A sets imaging data g ^{( ₁₎} i (m, _n) or ^{_{g (2) i (m,}} n) the linear image data ^gc which is respectively linearized _{(1) i (m, n} ) or ^{_{gc (2) i (m,}} n) Convert to

gc ⁽¹⁾ _i (m, n) = g ⁽¹⁾ _i (m, n) ^{1 / γ c}
gc ⁽²⁾ _i (m, n) = g ⁽²⁾ _i (m, n) ^{1 / γ c}
Here, the inverse gamma value γc is set to a value corresponding to the reciprocal of the gamma value γd of the display device. As an example, when the gamma value γd of the display device is 2.2, the inverse gamma value γc of the imaging device 200 is 1 / γd = 1 / 2.2 ≒ 0.45.

The imaging data g ⁽¹⁾ _i (m, n) and g ⁽²⁾ _i (m, n) output from the imaging device 200 by one imaging exist for the number of pixels of the imaging device 200. For example, in the case where the pixel of the imaging device 200 is 1920 × 1080, the number of imaging data g ⁽¹⁾ _i (m, n) and g ⁽²⁾ _i (m, n) is all 2073600. . Therefore, if the conversion equation as described above is executed as it is, the amount of calculation becomes enormous. If the arithmetic processing capability of the image processing apparatus 100A is sufficiently high, the arithmetic may be directly executed, but it is effective to use a look-up table (LUT) when the arithmetic processing capability is limited. .

The linear correction unit 102A according to the present embodiment stores the look-up table 102a in advance, and performs linearization processing by referring to the look-up table 102a. The look-up table 102a is a data table in which the result of the above-described conversion equation is stored in advance in association with each of all the signal levels that can be acquired by the input imaging data g. In the actual linearization operation, the conversion amount corresponding to the signal level of the input imaging data g may be acquired with reference to the look-up table 102a, so the amount of operation can be significantly reduced.

When the imaging data g output from the imaging device 200 is linear, that is, when the inverse gamma value γc = 1, it is not necessary to provide the linear correction unit 102A.

<Color reproduction processing>
The main contents of the color reproduction process performed by the color conversion unit 104 and the spectral reflectance estimation unit 110 are the color conversion unit 134 and the spectral reflectance estimation unit 130 in the image forming apparatus 100 according to the first embodiment described above. As this is similar to the color reproduction process to be performed, the detailed description will not be repeated.

The matrix E indicating the spectral radiance used in the above equation (6) can be set to an arbitrary value. This is observed under the first illumination environment (spectral radiance E ⁽¹⁾ ) based on imaging data g ⁽²⁾ obtained by imaging the object OBJ under the second illumination environment (spectral radiance E ⁽²⁾ ) It means that the color of the object OBJ can be reproduced. Therefore, in order to reproduce the color of the object OBJ to be observed under the first illumination environment, it is not necessary to image the object OBJ under the first illumination environment, but it is necessary to perform imaging under another second illumination environment. It means that the captured image data g ⁽²⁾ may be used. In other words, as long as the object OBJ can be imaged so as to be within the imageable color range in the imaging device 200, colors that would be observed under any illumination environment can be accurately reproduced. Therefore, imaging system 2 according to the present embodiment has an object of imaging object OBJ so as to be within the imageable color range in imaging device 200.

Again referring to FIG. 7, the first color conversion unit 104 is a part for generating the image data g ′ ⁽¹⁾ from the imaging data gc ⁽¹⁾ imaged under the first illumination environment, and It includes a reflectance estimation unit 104a and a color reproduction unit 104b.

The spectral reflectance estimation unit 104a indicates the spectral radiance E ⁽¹⁾ measured by the spectral radiance meter 300 under the first illumination environment and the spectral sensitivity at each wavelength of the imaging device of the imaging device 200 stored in advance. Using the matrix S and the autocorrelation matrix A stored in advance, the matrix operation of the above equations (3) and (4) is executed to calculate a spectral reflectance estimation matrix W ⁽¹⁾ . Then, the spectral reflectance estimation unit 104a outputs the calculated spectral reflectance estimation matrix W ⁽¹⁾ to the color reproduction unit 104b.

Although it is desirable that the cycle in which the spectral reflectance estimation unit 104 a calculates the spectral reflectance estimation matrix W is the same as the imaging cycle in the imaging device 200, the measurement cycle of the spectral radiance by the spectral radiance meter 300 is imaged If it is later than the period, the spectral reflectance estimation matrix W may be calculated at the measurement period of the spectral radiance. Alternatively, when the illumination environment of the object OBJ hardly changes, the spectral reflectance estimation matrix W may be calculated only when the spectral radiance output from the spectral radiance meter 300 changes.

The color reproduction unit 104 b includes the linear image data gc ⁽¹⁾ output from the linear correction unit 102 A, the spectral reflectance estimation matrix W ⁽¹⁾ output from the spectral reflectance estimation unit 104 a, and the spectral radiance meter 300. Using the measured spectral radiance E ⁽¹⁾ and the matrix h indicating the value of the color matching function at each wavelength stored in advance, the matrix operation of the above equation (6) is executed, and the image data g ′ ^{( 1)} Calculate _XYZ .

As described later, when the second illumination environment is changed from the first illumination environment to the second illumination environment as described later, the second color conversion unit 106 generates an image from the imaging data gc ⁽²⁾ imaged under the second illumination environment A portion for generating data g ′ ⁽²⁾ , which includes a spectral reflectance estimation unit 106 a and a color reproduction unit 106 b.

Similar to the spectral reflectance estimation unit 104a, the spectral reflectance estimation unit 106a includes the spectral radiance E ⁽²⁾ measured by the spectral radiance meter 300 under the second illumination environment, and the imaging device 200 stored in advance. The spectral reflectance estimation matrix W ⁽²⁾ is calculated using the matrix S indicating the spectral sensitivity at each wavelength of the imaging device and the autocorrelation matrix A stored in advance. Then, the spectral reflectance estimation unit 106a outputs the calculated spectral reflectance estimation matrix W ⁽²⁾ to the color reproduction unit 106b.

The color reproduction unit 106 b includes the linear image data gc ⁽²⁾ output from the linear correction unit 102 A, the spectral reflectance estimation matrix W ⁽²⁾ output from the spectral reflectance estimation unit 106 a, and the first illumination environment. Image data g ′ ⁽²⁾ _XYZ is calculated using the spectral radiance E ⁽¹⁾ measured by the spectral radiance meter 300 and the matrix h indicating the value of the color matching function at each wavelength stored in advance. .

The combining unit 116 selects a specific one of the pixel data included in the image data g ′ ⁽¹⁾ _XYZ and the pixel data included in the image data g ′ ⁽²⁾ _XYZ according to the map data of the flag table 118 described later. _Are combined with a specific one of them to generate composite image data _G.sub.XYZ .

Coordinate conversion unit 120, the synthetic image data _{G XYZ} of the XYZ color system generated by the synthesis unit 116 converts the synthesized image data _{G RGB} in the RGB color system, and outputs it to the gamma correction unit 122. That is, the coordinate conversion unit 120 converts the color space of the composite image data G _RGB and outputs the result.

Since the synthesized image data G _RGB of the _RGB color system converted by the coordinate conversion unit 120 maintains linearity, when outputting to a display device or the like, the gamma characteristics of the display device are taken into consideration, and the synthesized image data G _RGB is previously It is necessary to make corrections to cancel out the gamma characteristics of the display. Therefore, the gamma correction unit 122 performs conversion processing reverse to the conversion processing in the linear correction unit 102A. More specifically, the gamma correction unit 122 performs the color reproduction data gd _{RGB on} the composite image data G _RGB (m, n) of the _RGB color system according to the following arithmetic expression using the gamma value γd of the display device. Convert to (m, n).

gd _RGB (m, n) = G _RGB (m, n) ^{γ d}
Here, the gamma value γd of the display device is set to “2.2” or the like as an example.

Similar to the above-described linear correction unit 102A, the gamma correction unit 122 according to the present embodiment performs arithmetic processing using a look-up table (LUT). More specifically, the gamma correction unit 122 stores the look-up table 122a in advance, and performs gamma correction processing by referring to the look-up table 122a. The look-up table 122a is a data table in which the result of the above-described conversion equation is stored in advance in association with each of all possible signal levels of the input composite image data G _RGB . In the actual linearization operation, the conversion amount corresponding to the signal level of the input composite image data G _RGB may be acquired with reference to the look-up table 122a, so the amount of operation can be significantly reduced.

<Color gamut evaluation processing>
In the process of generating the color reproduction data gd as described above, the evaluation unit 112A determines whether the imaging data g ⁽¹⁾ imaged under the first illumination environment is within a color range that can be imaged by the imaging device 200. evaluate. Then, when the evaluation unit 112A evaluates that the imaging data g ⁽¹⁾ includes a color outside the color gamut that can be imaged by the imaging device 200, the color emitted by the object OBJ is within the color range that can be imaged by the imaging device 200. The illumination device 400 emits specific illumination light to form a second illumination environment. Since it is necessary to evaluate the presence or absence in such a color gamut in the color space of the XYZ color system, the evaluation unit 112A determines that the image data g ′ ^{(1 )} Evaluate _XYZ .

The color gamut evaluation process in evaluation unit 112A is the same as the color gamut evaluation process in evaluation unit 112 of image forming apparatus 100 according to the first embodiment described above, and therefore detailed description will not be repeated.

As shown in FIG. 2B described above, when a bluish illumination light is irradiated by the object OBJ, each sample point moves in the blue direction (lower left direction on the xy coordinates) in xy coordinates as a whole. . That is, by changing the illumination environment, the sample point (hereinafter also referred to as “correction target”) existing outside the color gamut in FIG. 2A emits yellow having a lower purity. It falls within the imageable color range of the device 200. As a result, the color of the portion corresponding to the correction target of the object OBJ can be captured within the imageable color range of the imaging device 200.

Image data corresponding to imaging data g ⁽¹⁾ which should have been imaged under the first illumination environment from imaging data g ⁽²⁾ imaged under the second illumination environment as shown in the above-mentioned equation (6) g ' ⁽²⁾ can generate _XYZ . Therefore, while using the image data g ' ⁽²⁾ _XYZ generated from the imaging data g ⁽²⁾ captured under the second illumination environment for the pixels to be corrected, the first illumination environment is used for the remaining pixels. The color reproduction data gd can be generated by using the image data g ′ ⁽¹⁾ _XYZ generated from the imaging data g ⁽¹⁾ picked up below.

FIG. 2C should be generated under the first illumination environment shown in FIG. 2A based on the imaging data g ⁽²⁾ imaged under the second illumination environment shown in FIG. 2B. The case where the image data g ' ⁽²⁾ _XYZ which had been generated is generated is shown. By performing such color reproduction processing, even if the color is out of the color gamut that can be imaged by the imaging device 200, color reproduction can be performed accurately.

FIG. 8 is a diagram for describing an outline of the image combining process according to the present embodiment.
Referring to FIG. 8, among the imaging data g ⁽¹⁾ imaged under the first illumination environment, a portion (correction target) having a color that is out of the color gamut that can be imaged by imaging device 200 is specified. Then, the image pickup data g ⁽¹⁾ image data g of a portion excluding the corrected generated from ^'(1), the correction object generated from the captured imaged data g ⁽²⁾ in the second illumination environment The image data g ′ ^{(2) of the} corresponding part is synthesized and generated as synthesized image data G.

By the way, in order to appropriately irradiate specific illumination light from the illumination device 400, it is necessary to specify a color that is outside the imageable color gamut of the imaging device 200. Therefore, evaluation unit 112A according to the present embodiment evaluates at which position on the xy coordinates the color is out of the color gamut, and selects the illumination light according to this position.

The switching process of the illumination light in evaluation unit 112A is the same as the switching process of the illumination light in evaluation unit 112 of image forming apparatus 100 according to the first embodiment described above, and therefore detailed description will not be repeated.

The evaluation unit 112A stores coordinate values (x, y) sequentially output from the coordinate conversion unit 110 by the number corresponding to one imaging by the imaging device 200, and the imaging device 200 defined in advance on xy coordinates. It is determined whether or not there is something outside the color gamut that can be imaged. Then, when there is a pixel having a color existing outside the color gamut that can be captured by the imaging device 200, the evaluation unit 112A stores information (flag) for specifying the position of the pixel in the flag table 118. . That is, the evaluation unit 112A classifies pixels having colors present in a color gamut that can be imaged by the imaging device 200 and pixels that have colors present outside the color gamut that can be imaged by the imaging device 200.

Furthermore, when there is a pixel having a color that exists outside the color gamut that can be captured by the imaging device 200, the evaluation unit 112A determines at which position the coordinate value (x, y) falls outside the color gamut. evaluate. Then, the evaluation unit 112A outputs a selection command of the illumination light to the illumination device 400 according to the position. As described above, specifically, the evaluation unit 112A determines whether the coordinate value (x, y) exists in any linear region connecting the respective coordinates corresponding to R, G, B on the xy coordinates. Judging by

Note that the color gamut that can be imaged by the imaging device 200, which is defined in advance on the xy coordinates, is stored in advance in the evaluation table 114. Further, since the color gamut that can be imaged by the imaging device 200 depends on the characteristics of the imaging elements that constitute the imaging device 200, the type or individual of the imaging device 200 is acquired in advance by experiment or simulation.

FIG. 9 is a diagram for explaining the data structure stored in flag table 118 shown in FIG. The flag table 118 according to the present embodiment has a data structure corresponding to the pixel arrangement (for example, 1920 × 1080 pixels) in the imaging device 200. Here, it is assumed that the initial values of the flag table 118 are all set to "0".

As shown in FIG. 9A, the evaluation unit 112A selects pixels having colors that exist outside the color gamut that can be imaged by the imaging device 200 among the imaging data g ′ ⁽¹⁾ imaged in the first illumination environment. When identified, as shown in FIG. 9B, the flag of the position corresponding to the pixel in the flag table 118 is set to “1”. The evaluation unit 112A performs the above-described process on all the pixels included in the imaging data g ′ ⁽¹⁾ . As a result, in the flag table 118, a map is formed which shows, among the pixels captured by the imaging device 200, those whose colors exist outside the color gamut that can be captured by the imaging device 200.

Again referring to FIG. 7, the synthesizing unit 116 specifies the image data g ′ ⁽²⁾ necessary for generating the synthesized image data G with reference to the map formed in the flag table 118 as described above. The second color conversion unit 106 may calculate the image data g ′ ⁽²⁾ corresponding to all the pixels based on the imaging data g ⁽²⁾ captured under the second illumination environment. However, since the calculation processing of the image data g ′ ⁽²⁾ has a relatively large amount of operation, referring to the flag table 118 via the combining unit 116, only the pixels necessary for generating the combined image data G It is preferable to generate data g ′ ⁽²⁾ .

In the above-described flag table 118, the color is specified in addition to information for specifying a pixel having a color that is out of the color gamut that can be captured by the imaging device 200 among the imaging data g ′ ^(1). You may store information for

FIG. 10 is a view showing another form of the flag table 118 shown in FIG. Referring to FIG. 10, in the flag table 118 of this embodiment, “R”, “G”, and “B” for specifying the color at the position corresponding to the pixel having the color existing outside the imageable color gamut. Values such as are stored. By adopting such a flag table, it is possible to specify the pixels of the image data g ′ ⁽²⁾ necessary for generating the composite image data G, and specify the color of the illumination light to be irradiated from the illumination device 400. You can also

<Processing procedure>
The processing procedure in the imaging system 2 according to the present embodiment is summarized as follows.

FIG. 11 is a flowchart showing the overall processing procedure in the imaging system 2 according to the second embodiment of the present invention. FIG. 12 is a flowchart showing the contents of the color gamut evaluation processing subroutine (2) shown in FIG.

Referring to FIGS. 7 and 11, first, imaging device 200 captures an object OBJ in the first illumination environment, and outputs imaging data g ⁽¹⁾ according to object OBJ (step S2). The imaging device 200 performs an imaging operation in response to an operation by a user or the like. Subsequently, the spectral reflectance estimation unit 104a acquires the spectral radiance E ⁽¹⁾ measured by the spectral radiance meter 300 (step S4), and calculates the spectral reflectance estimation matrix W ⁽¹⁾ (step S6). ).

Subsequently, the linear correction unit 102A linearizes the imaging data g ⁽¹⁾ _i (m, n) corresponding to one pixel output from the imaging device 200 to generate linear image data gc ⁽¹⁾ _i (m, n ^). ) Is generated (step S8). Then, the color reproduction unit 104b uses the spectral radiance E ⁽¹⁾ acquired in step S4, the spectral reflectance estimation matrix W ⁽¹⁾ calculated in step S6, and the linear image data gc generated in step S8. ⁽¹⁾ Image data g ' ⁽¹⁾ _XYZ (m, n) is calculated using _i (m, n) (step S10). Furthermore, it is determined whether the processing for all the pixels included in the imaging data g ⁽¹⁾ is completed (step S12), and the processing for all the pixels is not completed (NO in step S12), The processes after step S8 are executed again.

On the other hand, if processing for all the pixels is completed (YES in step S12), image data g ' ⁽¹⁾ _XYZ (m, n) exists in a color range that can be captured by imaging device 200. In order to evaluate whether or not the color gamut evaluation processing subroutine (2) is executed (step S14). The evaluation result of the color gamut evaluation processing subroutine (2) is stored in the flag table 118.

After execution of the color gamut evaluation processing subroutine (2), the evaluation unit 112A determines whether the flag is set in the flag table 118, that is, the image data g ′ ⁽¹⁾ _XYZ (outside the color gamut that can be imaged by the imaging device 200 ^). It is determined whether m, n) exists (step S16). When the flag is set (YES in step S16), the color reproduction unit 106b stores the spectral radiance E ⁽¹⁾ at the present time (step S18). Then, the evaluation unit 112A executes a lighting environment switching subroutine in order to switch the illumination light from the lighting device 400 (step S20). After execution of the illumination environment switching subroutine, the imaging device 200 images the object OBJ again under the second illumination environment, and outputs imaging data g ⁽²⁾ according to the object OBJ (step S22). Subsequently, the spectral reflectance estimation unit 106a acquires the spectral radiance E ⁽²⁾ measured by the spectral radiance meter 300 (step S24), and calculates the spectral reflectance estimation matrix W ⁽²⁾ (step S26). ).

Subsequently, the evaluation unit 112A refers to the flag table 118 to specify pixels necessary for generating the composite image data G (step S28). The linear correction unit 102A linearizes g ⁽²⁾ _i (m, n) corresponding to the pixels necessary for the composite image data G out of the imaging data g ⁽²⁾ output from the imaging device 200 and generates linear image data. gc ⁽²⁾ _i (m, n) is generated (step S30). Then, the color reproduction unit 106b stores the spectral radiance E ⁽¹⁾ stored in advance in step S18, the spectral reflectance estimation matrix W ⁽²⁾ calculated in step S26, and the linear image data gc generated in step S30. ⁽²⁾ Image data g ' ⁽²⁾ _XYZ (m, n) is calculated using _i (m, n) (step S32). Furthermore, it is judged whether or not processing for all pixels necessary for composite image data G is completed (step S34), and processing for all pixels necessary for composite image data G is not completed (step S34) The process of step S28 and subsequent steps is executed again.

On the other hand, if the processing for all the pixels necessary for composite image data G has been completed (YES in step S34), combining unit 116 calculates the image data g ′ calculated in step S32 ⁽²⁾ _XYZ (M, n) and data excluding the pixel corresponding to the image data g ′ ⁽²⁾ _XYZ (m, n) among the image data g ′ ⁽¹⁾ _XYZ (m, n) calculated in step S 10 The composite image data G _XYZ are generated from (step S36). Furthermore, the coordinate conversion unit 120 converts the synthesized image data _{G XYZ} calculated in step S36 in the composite image data _{G RGB} in the RGB color system (step S38), followed by gamma correction unit 122, the synthetic image data G _RGB is converted to color reproduction data gd _RGB (step S40), and output to the outside.

On the other hand, when the flag is not set (in the case of NO in step S16), the combining unit 116 combines the image data g ′ ⁽¹⁾ _XYZ (m, n) calculated in step S10 with the combined image data G _XYZ Output (step S42). Furthermore, the coordinate conversion unit 120 converts the synthesized image data _{G XYZ} output in step S40 in the composite image data _{G RGB} in the RGB color system (step S44), followed by gamma correction unit 122, the synthetic image data G _RGB is converted to color reproduction data gd _RGB (step S46), and output to the outside.

Next, the contents of the color gamut evaluation processing subroutine (2) of step S14 will be described. Referring to FIG. 12, first, coordinate conversion unit 110 converts image data g ′ ⁽¹⁾ _XYZ (m, n) into coordinate values (x, y) on xy coordinates (step S100). Then, the evaluation unit 112A reads the color gamut on the imageable xy coordinates of the imaging device 200 from the evaluation table 114 (step S102).

Then, in the evaluation unit 112A, the coordinate values (x, y) calculated in step S100 correspond to the coordinates (x1, y1) and G (green) corresponding to R (red) as shown in FIG. It is determined whether it exists on a straight line area (a) connecting x2, y2) (step S104). If the coordinate value (x, y) exists on the linear area (a) (YES in step S104), the evaluation unit 112A sets the flag B (blue) (step S106).

When the coordinate value (x, y) does not exist on the linear region (a) (in the case of NO in step S104), the evaluation unit 112A calculates the coordinate value (x, y) calculated in step S100 as shown in FIG. It is determined whether or not it exists on a linear region (b) connecting the coordinates (x2, y2) corresponding to G (green) and (x3, y3) corresponding to B (blue) as shown in FIG. Step S108). If the coordinate value (x, y) exists on the linear area (b) (YES in step S108), the evaluation unit 112A sets a flag R (red) (step S110).

If the coordinate value (x, y) does not exist on the linear region (c) (in the case of NO in step S108), the evaluation unit 112A determines that the coordinate value (x, y) calculated in step S100 It is determined whether it exists on a straight line area (c) connecting the coordinates (x3, y3) corresponding to B (blue) and (x1, y1) corresponding to R (red) as shown in FIG. Step S112). If the coordinate value (x, y) exists on the linear region (c) (YES in step S112), the evaluation unit 112A sets a flag G (green) (step S114).

After execution of steps S106, S110, and S114, the evaluation unit 112A sets a flag "1" at the position corresponding to the target pixel in the flag table 118 (step S116).

Furthermore, it is determined whether the processing for all the pixels included in the imaging data g ⁽¹⁾ is completed (step S118), and the processing for all the pixels is not completed (NO in step S118), The processes after step S104 are performed again.

The contents of the illumination environment switching subroutine of step S20 are the same as the contents of the flowchart shown in FIG. 6 described in the first embodiment described above, and therefore detailed description will not be repeated.

In the above-described processing procedure, the case where a pixel having a color outside the color gamut that can be imaged by the imaging device 200 is one of R (red), G (green), and B (blue) has been described. Is out of the color gamut for a plurality of colors, that is, when a plurality of flags among the flag R (red), the flag G (green) and the flag B (blue) are set, The process under the illumination environment may be performed multiple times. In this case, the second illumination environment is formed multiple times according to the set flags, and imaging data is acquired by imaging the object OBJ under each second illumination environment, as described above. According to the procedure of the above, composite image data may be generated using data of necessary pixels among the image data calculated from the respective imaging data.

<Operation and effect according to the present embodiment>
According to the second embodiment of the present invention, it is possible to determine the illumination environment at the time of imaging the subject in consideration of the color gamut that can be imaged by the imaging device. Therefore, since the color of the subject can be correctly captured using the imaging device, the spectral reflectance of the subject can be accurately estimated from the captured imaging data. Therefore, even if the color of the subject exists outside the imageable color gamut of the imaging device in the illumination environment where the subject should be imaged, the illumination environment where the imaging should be performed after the spectral reflectance is accurately estimated. The color that will be imaged (observed) can be properly reproduced.

Further, according to the second embodiment of the present invention, it is possible to generate composite image data by estimating spectral reflectance only for pixels having colors outside the color gamut that can be imaged by the imaging device. The increase can be suppressed.

<Modification>
In the first and second embodiments described above, the configuration using the

image processing apparatuses

100 and 100A mainly configured by hardware is illustrated, but all or part of them may be realized by software. That is, a computer may be used to implement the processing in the

image processing apparatuses

100 and 100A.

FIG. 13 is a schematic configuration diagram of a computer that realizes image processing apparatus 100 # according to the modification of the embodiment of the present invention.

Referring to FIG. 13, the computer includes a computer main body 150 equipped with a flexible disk (FD) drive 166 and a compact disk-read only memory (CD-ROM) drive 168, a monitor 152, a keyboard 154, and a mouse. And 156.

The computer main body 150 further includes a central processing unit (CPU) 160 which is an arithmetic device, a memory 162, a fixed disk 164 which is a storage device, and a communication interface 170 which are mutually connected by a bus.

The FD 166 a is attached to the FD driving device 166, and the CD-ROM 168 a is attached to the CD-ROM driving device 168. Image processing apparatus 100 # according to the modification of the present embodiment can be realized by CPU 160 executing software using computer hardware such as memory 162. Generally, such software is stored in a recording medium such as the FD 166a or the CD-ROM 168a, or distributed via a network or the like. Then, such software is read from the recording medium by the FD drive device 166, the CD-ROM drive device 168, or the like, or received by the communication interface 170 and stored in the fixed disk 164. Furthermore, it is read from the fixed disk 164 to the memory 162 and executed by the CPU 160.

The monitor 152 is a display unit for displaying information output by the CPU 160, and is configured of, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT). The mouse 156 receives an instruction from a user corresponding to an operation such as a click or a slide. The keyboard 154 receives an instruction from the user corresponding to the key input. The CPU 160 is an operation processing unit that executes various operations by sequentially executing programmed instructions. The memory 162 stores various types of information in accordance with program execution of the CPU 160. The communication interface 170 is a device for establishing communication between the computer and the imaging device 200, the spectral radiation luminance meter 300, the lighting device 400 (FIG. 1) and the like, and the information output by the CPU 160 is converted to, for example, an electrical signal. While converting and sending to another device, the electric signal is received from the other device and converted into information usable by the CPU 160. The fixed disk 164 is a non-volatile storage device that stores programs executed by the CPU 160, predetermined data, and the like. In addition, another output device such as a printer may be connected to the computer as necessary.

Furthermore, the program according to the present embodiment is to execute a process by calling a required module in a predetermined arrangement at a predetermined timing among program modules provided as a part of an operating system (OS) of a computer. It may be. In that case, the program itself does not include the above module, and the processing is executed in cooperation with the OS. A program not including such a module may also be included in the program according to the present invention.

Also, the program according to the present embodiment may be provided by being incorporated into a part of another program. Also in this case, the program itself does not include a module included in the other program, and the process is executed in cooperation with the other program.

<Other forms>
In the first and second embodiments described above, the configuration in which the spectral radiance meter 300 directly measures the illumination environment (spectral radiance) of the object OBJ is exemplified, but the illumination light irradiated to the object OBJ is mainly When depending on the illumination light from the illumination device 400, etc., the spectral radiance may be measured in advance instead of directly measuring the spectral radiance. That is, spectral radiance is measured in advance in association with each illumination light selectively emitted from illumination device 400, and illumination light emitted from illumination device 400 among the previously measured spectral radiances. Depending on the selection of, the corresponding spectral radiance may be read out.

The illumination light emitted from the illumination device 400 may be emitted only when the imaging data of the object OBJ is outside the color gamut that can be imaged by the imaging device 200. That is, at the time of normal imaging, the object OBJ is measured under a desired illumination environment, and under this illumination environment, the illumination device is the first time when the imaging data of the object OBJ is outside the color gamut that can be imaged by the imaging device 200. It is good also as operation which irradiates specific illumination light from 400.

It should be understood that the embodiments disclosed herein are illustrative and non-restrictive in every respect. The scope of the present invention is shown not by the above description but by the scope of claims, and is intended to include all modifications within the scope and meaning equivalent to the scope of claims.

Claims

Acquiring first imaging data by imaging an object (OBJ) to be irradiated with illumination light having a specific wavelength characteristic using an imaging device;
Generating color information in which the color of the subject is estimated using the wavelength characteristic of the illumination light with respect to the first imaging data (S6, S10, S14, S20, S22, S24);
And D. regenerating data obtained by imaging an object based on the color information (S32, S54, S56).
The step of generating the color information comprises
Acquiring the spectral radiance of the illumination light as a first spectral radiance (S6);
A first color indicating the color of the subject using the spectral reflectance and the first spectral radiance after estimating the spectral reflectance of the subject using the first spectral radiance and the first imaging data Generating a first color data in space (S10);
Evaluating whether the first color data is present in a color range that can be captured by the imaging device (S14);
Irradiating the subject with illumination light having a specific dominant wavelength when it is evaluated that the first color data does not exist within the imageable color range (S20);
Capturing the subject again during illumination of the illumination light and acquiring second imaging data (S22);
Acquiring the spectral radiance of the light irradiated to the subject during the illumination of the illumination light as a second spectral radiance (S24).
The regenerating step is
A first color indicating the color of the subject using the spectral reflectance and the first spectral radiance after estimating the spectral reflectance of the subject using the second spectral radiance and the second imaging data The imaging method according to claim 1, further comprising the step (S32) of regenerating the first color data in space.
The step of evaluating is
Determining the illumination light such that the first color data falls within the imageable color range, based on where on the first color space the first color data is out of the color gamut The imaging method according to claim 2, comprising (S200, S204, S208).
The imaging method according to claim 2, further comprising the step (S54) of converting the first color data into second color data in a second color space and outputting the converted data as image data.
The imaging method according to claim 4, wherein the first color space is an XYZ color system, and the second color space is an RGB color system.
An imaging device (200) for outputting imaging data consisting of output values of the imaging element;
An illumination device (400) capable of selectively illuminating a plurality of illumination lights having different dominant wavelengths toward the subject;
A spectral radiance acquisition unit (300) for acquiring the spectral radiance of light irradiated to the subject;
A first color data on a first color space indicating a color of the subject is generated using the spectral reflectance after estimating the spectral reflectance of the subject using the spectral radiance and the imaging data. One color conversion unit (130, 134),
An evaluation unit (112) for evaluating whether or not the first color data is present in a color range that can be captured by the imaging device;
The imaging system according to claim 1, wherein the evaluation unit irradiates a specific illumination light from the illumination device toward the subject when it is evaluated that the first color data is not present in the imageable color range.
The evaluation unit evaluates at which position on the first color space the first color data is out of the color gamut, and according to the position, the first color data falls within the imageable color range 7. The imaging system of claim 6, wherein the particular illumination light is selected to fit.
The first color conversion unit uses the spectral radiance acquired in the second illumination environment when the first illumination environment to be applied when capturing the subject is switched to the second illumination environment by the evaluation unit. The imaging system according to claim 6, further comprising: estimating the spectral reflectance; and generating first color data using the spectral reflectance and the spectral radiance acquired in the first illumination environment.
The second color converter (120) for converting the first color data after conversion by the first color converter into second color data in a second color space and outputting the second color data as image data An imaging system according to claim 6.
10. The imaging system according to claim 9, wherein the first color space is an XYZ color system, and the second color space is an RGB color system.
Acquiring first imaging data by imaging an object (OBJ) under a first illumination environment using an imaging device (200);
Acquiring a first spectral radiance that is a spectral radiance of light irradiated to the subject under the first illumination environment (S4);
The spectral reflectance of the subject is estimated using the first imaging data and the first spectral radiance, and then the color of the subject is indicated using the spectral reflectance and the first spectral radiance. Generating first color data in one color space (S6, S8, S10);
Acquiring a second imaging data by imaging the subject under a second illumination environment using the imaging device (S22);
Acquiring a second spectral radiance that is a spectral radiance of light irradiated to the subject under the second illumination environment (S24);
The spectral reflectance of the subject is estimated using the second imaging data and the first spectral radiance, and then the color of the subject is indicated using the spectral reflectance and the first spectral radiance. Generating second color data on the first color space (S26, S30, S32);
Generating composite image data based on the first color data corresponding to the first imaging data and the second color data corresponding to the remaining pixels excluding the first color data (S36) Imaging method.
12. The imaging method according to claim 11, wherein the step of generating the second color data includes the step of generating the second color data only for the remaining pixels (S36).
The method further includes a step (S14, S16, S18, S20) of evaluating whether or not the first color data for each pixel is within a color range that can be captured by the imaging device,
The step of evaluating includes classifying pixels corresponding to first color data present in a color range that can be imaged by the imaging device into pixels of a first group, and not existing in a color range that can be imaged by the imaging device 12. The imaging method according to claim 11, further comprising the step (S14) of classifying pixels corresponding to one-color data into a second group of pixels.
In the evaluation step, when there is one of the first color data that exists outside a color gamut that can be captured by the imaging device, a plurality of illumination lights having different principal wavelengths are selectively emitted toward the subject. The imaging method according to claim 13, comprising the step (S202, S206, S210) of forming the second illumination environment by irradiating specific illumination light toward the subject from a possible illumination device. .
In the evaluating, the specific illumination light is set so that the color of a pixel corresponding to the first color data present outside the color gamut that can be captured by the imaging device falls within the color gamut that can be captured by the imaging device. The imaging method according to claim 14, comprising the steps of selecting (S200, S204, S208).
The step of selecting the illumination light is performed by evaluating at which position on the first color space the first color data existing outside the color gamut that can be captured by the imaging device is out of the color gamut. The imaging method according to claim 15, including the step (S104, S108, S112A) of selecting the specific illumination light according to the position.
12. The imaging method according to claim 11, further comprising the step (S38) of converting the composite image data in the first color space into image data in a second color space.
The imaging method according to claim 17, wherein the first color space is an XYZ color system, and the second color space is an RGB color system.
Acquiring the first imaging data by imaging the subject using the imaging device (S2);
Evaluating whether first color data included in the first shooting data is color data within a color range that can be captured by the imaging device (S14);
Illuminating the illumination light having a specific wavelength toward the subject when it is evaluated that the first color data is not color data within the imageable color range (S20);
Capturing the subject again during illumination of the illumination light and acquiring second imaging data (S24);
The second color data of the second shooting data corresponding to the color data evaluated not as color data within the color range of the first shooting data, using the specific wavelength characteristic of the illumination light, for the second color data of the second shooting data Generating third color data by inferring (S32);
Step of generating composite image data by replacing the first color data evaluated not to be color data within the imageable color range included in the first shooting data with the corresponding third color data (S36) And an imaging method.
An imaging device (200) for outputting imaging data composed of output values of the plurality of imaging elements at respective pixels corresponding to an object (OBJ);
A spectral radiance acquisition unit (300) for acquiring the spectral radiance of light irradiated to the subject;
The spectral reflection factor of the subject at each pixel is estimated using first imaging data obtained by imaging the subject under a first illumination environment and the first spectral radiance under the first illumination environment, and then the spectral reflection is performed. A first color converter (104) for generating, for each pixel, first color data on a first color space representing the color of the subject using a ratio and the first spectral radiance;
The spectral reflectance of the subject at each pixel is estimated using the second imaging data obtained by imaging the subject under the second illumination environment and the second spectral radiance under the second illumination environment, and then the spectral reflection of the subject is performed. A second color converter (106) for generating second color data on a first color space representing the color of the subject using a ratio and the first spectral radiance;
First color data corresponding to a first group of pixels among a plurality of pixels included in the imaging device, and second color data corresponding to a second group of remaining pixels excluding the first group of pixels An imaging system comprising: a synthesizing unit (116) for generating synthetic image data based on the imaging system.
21. The imaging system according to claim 20, wherein said second color conversion unit generates said second color data only for the pixels classified into said second group.
It further comprises an evaluation unit (112A) for evaluating whether or not the first color data for each pixel is present in a color range that can be captured by the imaging device,
The evaluation unit classifies pixels corresponding to first color data present in a color range that can be captured by the imaging device into pixels of the first group, and does not exist in a color range that can be captured by the imaging device. 21. The imaging system according to claim 20, wherein pixels corresponding to one-color data are classified into the second group of pixels.
It further comprises an illumination device (400) capable of selectively illuminating a plurality of illumination lights having different dominant wavelengths toward the subject,
The evaluation unit causes the illumination device to emit specific illumination light toward the subject when the first color data exists outside the color gamut that can be captured by the imaging device, and the second color data is generated. The imaging system according to claim 22, forming a lighting environment.
The evaluation unit selects the specific illumination light such that the color of the pixel corresponding to the first color data present outside the color gamut that can be captured by the imaging device falls within the color gamut that can be captured by the imaging device. The imaging system according to claim 23, wherein.
The evaluation unit evaluates at which position on the first color space the first color data existing out of the color gamut that can be captured by the imaging device is out of the color gamut, and according to the position The imaging system according to claim 24, wherein a specific illumination light is selected.
The imaging system according to claim 20, further comprising a third color conversion unit (120) that converts the composite image data in the first color space into image data in a second color space.
The imaging system according to claim 26, wherein the first color space is an XYZ color system, and the second color space is an RGB color system.