JP2017203682A - Imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device with which it is possible to synthesize a plurality of light sources differing in wavelength band in an active illumination method and reduce a color difference of necessary wavelength between the light sources, thereby reducing the occurrence of a flicker.SOLUTION: The imaging device comprises: a plurality of light sources 17 that emit light in different wavelengths; a light source selection unit 22 for selecting a light source 17, out of the light sources 17, that emits light in a specific wavelength specified on the basis o the characteristic of an imaging object 27, in plurality as a selected light source; a light source extraction unit 24 for extracting a light source 17 having a wavelength at which a color difference between one selected light source out of a selected plurality of selected light sources and other selected light sources decreases as they are synthesized into the one selected light source, from among unselected light sources other than the selected light sources; a light source 17 for irradiating the imaging object 27 with a light source in which the selected light source and the extracted unselected light source are synthesized while successively switching the light sources for each selected light source; and an imaging element 11 for imaging the imaging object 27 with the irradiated light source.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an imaging apparatus that switches and images a light source having a plurality of different wavelength bands.

  Humans perceive light that reaches the eye either directly from a light source or reflected from an object by three types of cone cells. This is why there are so-called three primary colors of light. The same applies to a general camera, and a color is expressed by a combination of three RGB components. In other words, humans and general cameras measure light as three components of RGB. However, actual light in the natural world exists as a continuous spectrum instead of such RGB composition. In contrast to a general camera that can measure only three channels of RGB, a camera that can measure ten to several tens of spectral components is called a hyperspectral camera, and image measurement using it is called hyperspectral imaging.

  Hyperspectral imaging can extract or visualize features that cannot be identified by humans by using more spectral information than a general RGB camera. Because of these advantages, hyperspectral imaging has been applied in various fields such as remote sensing, food, agriculture, and space. Hyperspectral imaging is also used in the medical field. In the medical field, it is very important to distinguish between normal and abnormal parts of body tissue, for example to identify abnormal parts during surgery and minimize the burden on the patient by performing treatment only on those parts. Can do. Furthermore, if much information obtained by hyperspectral imaging is used, more detailed identification is possible and the success rate of the operation can be increased.

  There are three elements required for a hyperspectral camera: spatial resolution, spectral resolution, and imaging speed. Since the clearer image can be acquired as the spatial resolution is higher, more detailed information about the object can be acquired. Also, the higher the spectral resolution, the more spectral information can be used, so that the object identification accuracy can be improved. Spectral information can be used in real time if the shooting speed is high, and the application destinations are expanded. Therefore, although it is most desirable to obtain a high spatial resolution and high spectral resolution spectral image at high speed, these elements are generally in a trade-off relationship. Improvement in high resolution and imaging speed is a very important factor in hyperspectral imaging, and in particular, improvement in technology relating to acquisition of spectral images with high spatial resolution is strongly desired.

  Various imaging methods have been proposed for hyperspectral imaging. The line spectroscopy method (see, for example, Non-Patent Document 1) is a technique for obtaining a hyperspectral image by dispersing light that has passed through a slit into a line shape with an optical element such as a prism or a grating. When this method is used, spatial information and spectral information can be acquired simultaneously in the horizontal direction and the spectral resolution is high. However, if a two-dimensional image of the target is to be obtained, the entire target must be scanned. It takes time, and the spatial resolution also deteriorates. There is also a problem that the optical element to be used becomes expensive.

  As described above, this line spectroscopic method has a high spectral resolution, but when the object is observed as a two-dimensional image, the spatial resolution is low and a long photographing time is required. When the target reflected light is simply spectrally separated by an optical element and acquired by an image sensor, the spectrally separated light interferes on the sensor. In the line spectroscopy method, this interference is prevented by passing through a slit, and this is why spatial information can be acquired only one dimension at a time. Non-patent Documents 2 and 3 disclose techniques for acquiring a two-dimensional spectral image at high speed using the line spectroscopic method. The technique shown in Non-Patent Document 2 is to acquire a two-dimensional spectroscopic image at high speed using an optical element using a hole array mask, and the technique shown in Non-Patent Document 3 has a spatial resolution in a hyperspectral camera. A two-dimensional spectral image with a high spatial resolution and a high spectral resolution is acquired by combining a high camera.

  However, the technique shown in Non-Patent Document 2 has a problem that the spatial resolution is low. In addition, the technique shown in Non-Patent Document 3 requires a beam splitter, a hole array mask, etc. in addition to two cameras, which increases the size of the apparatus and complicates the setting of the position and angle of the spectroscopic element, the camera, etc. There is a problem that it ends up.

  On the other hand, in the filter method (see, for example, Non-Patent Documents 4 and 5), a spectral image is obtained by installing a filter that passes only light in a specific wavelength region in front of a CCD camera or the like. Although a spectral image with a high spatial resolution can be obtained by using a CCD camera with a high resolution, the spectral resolution is generally not high due to the characteristics of the filter. In this method, it is possible to acquire a hyperspectral image by repeating photographing while switching filters, and a filter wheel or a liquid crystal tunable filter is used as the filter. In the case of a filter wheel, since the filter is physically rotated, there is a problem in switching speed, and there is a problem that the apparatus is also increased in size. In the case of a liquid crystal tunable filter, since the wavelength range to transmit is electrically controlled, the switching speed is faster than that of the filter wheel, but the production cost is high.

  Further, as a technique for acquiring a spectral image with a high spatial resolution by a filter method, a technique shown in Non-Patent Document 6 is disclosed. The technique shown in Non-Patent Document 6 acquires a spectral image with a high spatial resolution by using matrix factorization of an RGB image with a high spatial resolution and a spectral image. However, this apparatus requires a mechanical movable part, and it takes time for imaging compared with the case of only electrical control. In addition, there is a problem that it takes time for matrix calculation.

  As an inexpensive method, there is an active illumination method (see, for example, Non-Patent Document 7) used in the present application for obtaining a hyperspectral image by photographing while switching illuminations having different spectra. Since switching of illumination is electrically controlled in the same manner as the liquid crystal tunable filter, a target two-dimensional spectral image can be acquired at high speed. However, unlike the line spectroscopic method or the filter method, it is affected by environmental light such as other illumination or sunlight, so it cannot be used outdoors. Further, since a plurality of illuminations having different colors are switched, color flicker occurs depending on the photographing speed. In general, in order to obtain spectral information, it is necessary to switch between illuminations having different spectra, which causes a change in color and flicker.

  Further, as a technique related to the active illumination method, for example, a technique shown in Non-Patent Document 8 is disclosed. The technique shown in Non-Patent Document 8 is a technique that realizes high-speed acquisition of a two-dimensional spectral image using an inexpensive and practical hyperspectral imaging apparatus using an LED.

Ye, X. and Sakai, K .: Application of Hyperspectral Imaging in Agriculture, The Institute of Image Information and Television Engineers, Vol. 69, No. 5, ITE, pp. 464-469 (2015). Du, H., Tong, X., Cao, X. and Lin, S .: A prism-based system for multispectral video acquisition, IEEE 12th International Conference on Computer Vision, IEEE, pp. 175-182 (2009). Cao, X., Tong, X., Dai, Q. and Lin, S .: High resolution multispectral video capture with a hybrid camera system, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 297-304 (2011). Brauers, J., Schulte, N. and Aach, T .: Multispectral filterwheel cameras: Geometric distortion model and compensation algorithms, IEEE Transactions on Image Processing, Vol. 17, No. 12, pp. 2368-2380 (2008). Slawson, RW, Ninkov, Z. and Horch, EP: Hyperspectral Imaging: Wide-Area Spectrophotometry Using a Liquid-Crystal Tunable Filter, Publications of the Astronomical Society of the Pacific, Vol. 111, No. 759, pp. 621-626 (1999). Kawakami, R., Wright, J., Tai, Y.-W., Matsushita, Y., Ben-Ezra, M. and Ikeuchi, K .: High-resolution hyperspectral imaging via matrix factorization, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 2329-2366 (2011). Park, J.-I., Lee, M.-H., Grossberg, M. and Nayar, SK: Multispectral imaging using multiplexed illumination, IEEE 11th International Conference on Computer Vision (ICCV), IEEE, pp. 1-8 ( 2007). Goel, M., Whitmire, E., Mariakakis, A., T Saponas, S., Joshi, N., Morris, D., Guenter, B., Gavriliu, M., Borriello, G. and Patel, S. : HyperCam: hyperspectral imaging for ubiquitous computing applications, The 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM (2015)

  As described above, in the medical field, hyperspectral imaging is mainly used to distinguish between normal and abnormal parts of body tissue. Acquisition of a target two-dimensional spectral image is the center, and when using it in surgery, it is important to acquire a spectral image at high speed. However, as described above, the active illumination technique can obtain a two-dimensional spectroscopic image at a high speed and at a low cost, but the occurrence of flicker hinders surgery, and in the medical field. There is a problem that usability deteriorates.

  The technique shown in Non-Patent Document 8 realizes high-speed acquisition of a two-dimensional spectral image using an active illumination method, but does not consider the color of illumination to be switched. That is, when the illumination colors are different, there is a problem that flicker occurs when the illumination is switched.

  The present invention provides an imaging apparatus capable of reducing the occurrence of flicker by combining a plurality of light sources having different wavelength bands in an active illumination system and reducing a color difference between light sources having necessary wavelengths.

  An imaging apparatus according to the present invention includes a plurality of light sources that emit light at different wavelengths, and a selection unit that selects a plurality of light sources that emit light at a specific wavelength specified based on characteristics of an imaging target among the light sources as a selection light source. And, other than the selected light source, the light source having a wavelength that reduces the color difference between the selected light source and the other selected light source by combining the selected light sources with the selected light source. An imaging unit that images an imaging target object by sequentially switching the light source for each selected light source by a light source obtained by synthesizing the selected selected light source and the extracted non-selected light source. Are provided.

  As described above, in the imaging apparatus according to the present invention, a plurality of light sources that emit light at a specific wavelength specified based on characteristics of the imaging target are selected from a plurality of light sources that emit light at different wavelengths, and are selected when they are combined. The non-selected light source is extracted so that the color difference between the selected light sources is reduced, and the selected light source and the extracted light source are combined. By suppressing the color difference (between one selected light source and another selected light source) and switching with illumination that looks like the same color for humans, it is possible to reduce the occurrence of flicker. That is, by reducing the occurrence of flicker, for example, it is possible to accurately identify an abnormal part during surgery, for example, and to perform the operation only on that part, thereby minimizing the burden on the patient. .

  An image pickup apparatus according to the present invention includes an image removal unit that removes an image picked up by a non-selection light source extracted by the extraction unit from an image picked up by the image pickup unit, and a residual image after the removal process by the image removal unit. Display control means for displaying an image.

  As described above, in the imaging device according to the present invention, the image captured by the extracted non-selected light source is removed from the captured image, and the remaining image is displayed. There is an effect that a hyperspectral image captured at a wavelength can be displayed with high accuracy in real time.

  In the imaging apparatus according to the present invention, the extraction unit extracts a non-selected light source having a wavelength band in the vicinity of another selected light source with respect to the one selected light source as a non-selected light source to be combined with the one selected light source. It is.

  As described above, in the imaging apparatus according to the present invention, the extraction unit serves as a non-selected light source that combines a non-selected light source having a wavelength band in the vicinity of another selected light source with respect to the one selected light source. Since the color difference between one selected light source and the other selected light source is suppressed and the lighting is switched so that it looks like the same color for humans, the occurrence of flicker can be reduced. Play.

  In the imaging apparatus according to the present invention, the extraction unit extracts a non-selected light source having a wavelength band other than the vicinity of the other selected light source according to external light in an imaging environment.

  As described above, in the imaging apparatus according to the present invention, the extraction unit extracts the non-selected light source having a wavelength band other than the vicinity of the other selected light source according to the external light in the imaging environment. As a result, it is possible to display an image that is easy for a human to visually recognize.

  In the imaging apparatus according to the present invention, the extraction unit extracts the light source based on a wavelength band and light intensity of the non-selected light source.

  As described above, in the imaging apparatus according to the present invention, since the extraction unit extracts the light source based on the wavelength band and the light intensity of the non-selected light source, the color difference between one selected light source and another selected light source is calculated. There is an effect that it can be made smaller and the occurrence of flicker can be remarkably reduced.

  In the imaging apparatus according to the present invention, the extraction unit extracts the light source based on the light intensity in consideration of photosensitive characteristics.

  As described above, in the imaging apparatus according to the present invention, since the extraction unit extracts the light source based on the light intensity considering the photosensitive characteristic, the color difference between one selected light source and the other selected light source is further reduced. Thus, it is possible to perform switching with illumination that looks the same color for humans, and it is possible to significantly reduce the occurrence of flicker.

It is a hardware block diagram of the imaging device which concerns on 1st Embodiment. It is a functional block diagram of the imaging device concerning a 1st embodiment. It is a 1st image figure of the spectrum and color difference of LED illumination (before optimization). It is a 1st image figure of the spectrum and color difference of LED illumination (after optimization). It is a 2nd image figure of the spectrum and color difference of LED illumination (after optimization). 3 is a flowchart illustrating an operation of the imaging apparatus according to the first embodiment. It is a figure which shows the spectrum of 15 types of LED used in the Example. It is a figure which shows the image which imaged the vein of the human body. It is a figure which shows the created illumination used in the Example. It is a figure which shows the result of having compared the flicker at the time of switching of LED illumination. It is a figure which shows the light absorption characteristic of hemoglobin. It is a figure which shows a 653 nm spectral image and an 834 nm spectral image. It is a figure which shows the image imaged with the normal RGB camera, and the image of the difference of a spectral reflectance.

  Embodiments of the present invention will be described below. Also, the same reference numerals are given to the same elements throughout the present embodiment.

(First embodiment of the present invention)
An imaging apparatus according to this embodiment will be described with reference to FIGS. The imaging apparatus according to the present embodiment synthesizes illumination that looks the same color to the human eye even if the spectrum is different, in order to reduce flicker that occurs when switching illumination in the active illumination method of hyperspectral imaging. To do. In this embodiment, in order to suppress the color difference between different illuminations, the same color (reference: Miyazaki, D., Saneshige, T., Baba, M., Furukawa, R., Aoyama, M. and Hiura, S) .: Metamerism-based shading illusion, 14th IAPR International Conference on Machine Vision Applications (MVA), IEEE, pp. 255-258 (2015).

  First, the configuration and function of the imaging apparatus according to the present embodiment will be described. FIG. 1 is a hardware configuration diagram of the imaging apparatus according to the present embodiment. The imaging apparatus 1 includes an imaging element 11, a CPU 12, a RAM 13, a ROM 14, an input / output I / F 15, a display 16, and a light source 17. The imaging device 11 converts imaging information captured from the camera lens into an electrical signal. Various programs, data, and the like are stored in the ROM 14. Various programs are read into the RAM 13 as necessary and executed by the CPU 12. The input / output I / F 15 is an interface for receiving input from a shutter, a touch panel, various switches, etc., and outputting data to a display, an external memory, or the like. It is a monitor that displays a captured image on the display 16, and in the case of a touch panel, it also functions as an input device. The light source 17 is illumination that irradiates the imaging object, and includes, for example, a plurality of LEDs having different wavelength bands.

  The hardware configuration is merely an example, and can be changed as necessary. For example, the imaging process shown in FIG. 1 is executed until the imaging process, and a separately connected computer may be used for the image editing process after imaging.

  FIG. 2 is a functional block diagram of the imaging apparatus according to the present embodiment. In FIG. 2, the imaging apparatus 1 is configured to realize a function until the imaging object 27 is imaged, and information such as a specific wavelength specified based on the characteristics of the imaging object 27 is input information 26. Based on the information input to the input unit 21 from the input unit 21 that inputs information 26 and the light source information storage unit 23 that stores information about a plurality of light sources that emit light at different wavelengths prepared in advance. A light source selection unit 22 that selects a plurality of pieces of information about the light source as a selected light source, and information on a non-selected light source other than the selected light source selected by the light source selection unit 22 is acquired from the light source information storage unit 23 and combined with the selected light source A light source extraction unit 24 that identifies and extracts a non-selected light source that minimizes the color difference between the selected light sources, and the light source 17 selects the selected light source and the light source extraction selected by the light source selection unit 22. And a non-selected source extracted in section 24 in the synthesized light source, illuminating the imaged object 27 sequentially switches the light source to be used for each selected light source.

  Furthermore, the imaging apparatus 1 is configured to realize a function after imaging the imaging object 27, and an imaging information storage unit 28 that stores imaging information in which an image of the imaging object 27 is converted into an electrical signal by the imaging element 11; Based on the information stored in the light source information storage unit 23 relating to the non-selected light source and the non-selected light source, an image removing unit 29 that removes information related to the non-selected light source from the imaging information, and the remaining image is displayed on the display 16. Display control unit 30.

  Hereinafter, details of each processing unit will be described. In order to perform hyperspectral imaging with the active illumination method, it is necessary to prepare a plurality of light sources (illuminations) having different spectra. Here, it is assumed that illuminations with different m types of spectra are created by combining n types of narrow-band LEDs. When n = m, a spectral image at the wavelength of n types of LEDs can be restored from an image taken under mixed illumination.

  First, a composite model of illumination will be described. When discretizing i spectra, the illumination spectrum L is

  It can be expressed as When m types of illumination are created by combining n types of LEDs, the LED combination pattern can be expressed by the following m × n matrix P.

  Therefore, the composite model of illumination can be expressed by the following equation.

  Next, a method for acquiring a target spectral image by processing of the image removal unit 29 for an image captured using the selected light source selected by the light source selection unit 22 will be described. The luminance value I of an image captured using illumination having a spectrum of L ′ is

  It is represented by Here, Q is the spectrum of the spectral sensitivity characteristic of the camera, and R is the spectrum of the spectral reflectance of the subject.

  From the equations (3) and (4), the following relationship is established.

  When n = m, the LED combination pattern matrix is a square matrix. When an inverse matrix exists in P, the following equation (8) is established, so that a spectral image at the dominant wavelength of each LED can be obtained from the captured image.

  In addition, a spectral image can be acquired by an active illumination method even in a place affected by ambient light. The ambient light is regarded as a kind of LED having a specific spectrum, and illumination is created by combining n + 1 kinds of LEDs. By photographing with this illumination switched, a spectral image can be acquired as in the case where there is no influence of ambient light.

  Thus, a spectral image can be acquired by creating a plurality of illuminations by combining LEDs and switching and photographing them. That is, it is possible to acquire a spectral image captured by a selected light source having a specific wavelength by the processing of the light source selection unit 22.

Next, processing of the light source extraction unit 24 for reducing flicker will be described. Each spectral image can be acquired by switching the illumination as described above, but flicker occurs when the color difference is large between the switched illuminations. Conversely, in order to reduce flicker, the color difference between the illuminations should be as small as possible. In order to handle color differences, it is first necessary to quantitatively express colors and color differences. The color of illumination is expressed as a numerical value based on the XYZ color system defined by the CIE (International Lighting Commission). The numerical value is converted into a numerical value on the color coordinate in the L ^* U ^* V ^* color system similarly determined by CIE, and the color difference is handled as the Euclidean distance on the color coordinate. By treating the color difference as the Euclidean distance, it is possible to optimize the combination of LEDs that minimize the color difference. In order to acquire an n-band spectral image, n types of LEDs are sufficient, but by combining extra LEDs that are not directly related to spectral image acquisition, the color difference between illuminations can be reduced.

  3 and 4 are first image diagrams of the spectrum and color difference of LED illumination. FIG. 3 shows the spectrum of the selected light source, and FIG. 4 shows the spectrum of the selected light source and the non-selected light source when the color difference is reduced. As shown in FIG. 3, when the color difference between a plurality of selected light sources is large, flicker occurs by switching between them. On the other hand, as shown in FIG. 4, it is possible to reduce the color difference between the selected light sources by deliberately combining the extra non-selected light sources. In the state shown in FIG. 4, flicker can be reduced even when the selected light source is switched.

  FIG. 5 is a second image diagram of the spectrum and color difference of LED illumination. FIG. 5 shows the spectra of the selected light source and the non-selected light source when the color difference is reduced with respect to the selected light source of FIG. Here, as shown in FIG. 5, an extra non-selected light source is specified centering around the wavelength band near the selected light source, thereby reducing the color difference between the selected light sources. That is, one selected light source (selected light source of illumination 1 in FIG. 3 = light source A of illumination 1 in FIG. 5) has another selected light source (selected light source of illumination 2 in FIG. 3 = light source B of illumination 1 in FIG. 5). A non-selected light source having a wavelength band in the vicinity of (a light source in the vicinity of the light source B of the illumination 1 in FIG. 5) is specified and synthesized. Conversely, for another selected light source (selected light source of illumination 2 in FIG. 3 = light source B of illumination 2 in FIG. 5), one selected light source (selected light source of illumination 1 in FIG. 3 = light source A of illumination 2 in FIG. 5). A non-selected light source having a wavelength band in the vicinity of (a light source in the vicinity of the light source A of the illumination 2 in FIG. 5) is specified and synthesized. By combining the nearby light sources as shown in FIG. 5, the color difference between the selected light sources can be reduced, and the occurrence of flicker can be prevented.

  Next, the color coordinates shown above will be described. Humans perceive light that reaches the eye either directly from the light source or reflected from an object by three types of cone cells. The three types of cone cells show the highest response to wavelengths corresponding to red, green, and blue, respectively. This is why the three primary colors of light are red, green, and blue. The CIE (International Certification Board) has defined an RGB color system to quantify human standard color vision. CIE conducted color matching experiments using R (700 nm), G (546.1 nm), and B (435.8 nm) as primary colors, and defined human color vision as a color matching function. However, since there is a drawback that there is a wavelength range that cannot be accurately expressed in the RGB color system, an XYZ color system that can express the wavelength range has been defined. In the XYZ color system, a color is expressed by a mixed color amount of non-existing primal stimuli such as X, Y, and Z. In the wavelength range of 390 to 830 nm, the color matching functions are x￣ (λ), y￣ (λ), z￣ (λ) (“x￣”, “y￣”, “z￣” respectively, where the wavelength is λ. X, Y, and Z are represented by equations (9), (10), and (11), respectively.

  Here, s (λ) is the target spectral distribution. Formula (12) shows a case where i is a discretization of Formula (9), Formula (10), and Formula (11).

An xy chromaticity diagram using the color mixture ratio of X, Y, and Z is used for color determination, but the color space of this chromaticity diagram is not uniform, and humans feel the distance in the color space. Does not correspond well with the color difference. Therefore, CIE has determined the L ^* u ^* v ^* color system as one of the uniform color spaces. The color is expressed quantitatively by converting the Mihara stimulus X, Y, Z to color coordinates L ^* , u ^* , v ^* . A conversion formula from XYZ to L ^* u ^* v ^* is expressed by Formula (13). Here, Xn, Yn, and Zn are tristimulus values of a standard light source. L ^* is a lightness index, and the color in the chromaticity diagram is determined by u ^* and v ^* . In the reference white color, u ^* = 0 and v ^* = 0. In the L ^* u ^* v ^* color system, the color difference is defined as the Euclidean distance on the L ^* u ^* v ^* coordinates.

Among these LED combinations that satisfy the conditions necessary for acquiring a spectral image using these color coordinates, a combination that minimizes the color difference is obtained. In order to reduce the color difference, LEDs that are non-selected light sources that are not directly related to spectral image acquisition are combined, but the LEDs can be handled in the same manner as ambient light. In the following, a case is considered where n types of LEDs are prepared and spectral images at two specific wavelengths are acquired. The spectra of the narrow-band LEDs having two specific wavelengths selected by the light source selection unit 22 as the main wavelengths are L _λ1 and L _λ2 , respectively, and the spectra of the LEDs of the non-selected light sources that are not directly related to spectral image acquisition are L _ex And When L _ex is the spectrum of the remaining LED combinations, L _ex is

It is represented by At this time, the above equation (3) becomes the equation (15), and spectral images of wavelengths λ ₁ and λ ₂ can be acquired by applying to the equations (4) to (8).

When P and P ′ are each determined as one, three types of illumination spectra are determined. X, Y, Z for each illumination is obtained from the equation (12), and the color difference between the illuminations can be calculated by the equation (13). Therefore, by obtaining P and P ′ that minimize the color difference, it is possible to obtain a combination of LEDs that constitute flickerless LED illumination. Here, for example, this problem can be treated as a constrained nonlinear multivariable function optimization problem and solved using the SQP method. Further, the range that P and P ′ can take is set to 0 to 1, and is treated as a mixture ratio of each LED, and a lower limit is set to the ratio of L _λ1 and L _{λ2 in} order to suppress a decrease in the SN ratio of the spectral image Good. For example, a combination in which 50% or more of the target L _λ1 and L _λ2 LEDs are included in the illumination may be obtained. Further, the condition number of P ′ may be limited to 5 or less in order to obtain the inverse matrix of P stably.

  As described above, the light source extraction unit 24 can extract a non-selected light source that minimizes the color difference between the selected light sources when combined with the selected light source. At this time, it is possible to realize higher performance flickerless hyperspectral imaging by considering the light intensity and human photosensitive characteristics as described above. The above is the description of the configuration and functions of the imaging apparatus according to the present embodiment.

  Next, the operation of the imaging apparatus according to the present embodiment will be described. FIG. 6 is a flowchart illustrating the operation of the imaging apparatus according to the present embodiment. First, the input unit 21 inputs, as input information 26, information on a specific wavelength that is a target specified according to the imaging object 27 (S1). The light source selection unit 22 selects a light source whose main wavelength is the wavelength closest to the specific wavelength (S2). The light source extraction unit 24 optimizes the light source to be used so that the color difference between the selected selected light sources is minimized (S3). The imaging object 27 is irradiated with the optimized light source 17 and imaged by the imaging device 11 (S4). The image removing unit 32 removes the image of the wavelength added at the time of optimization from the image obtained by imaging (S5). The display control unit 33 displays the remaining image information on the display 16 (S6), and the process ends.

  As described above, in the imaging apparatus according to the present embodiment, when a plurality of light sources that emit light at a specific wavelength specified based on the characteristics of the imaging target are selected from a plurality of light sources that emit light at different wavelengths and combined. The non-selected light source is extracted so that the color difference between the selected light sources is reduced, and the selected light source and the extracted light source are combined. The occurrence of flicker can be reduced by suppressing the color difference between the illuminations (between one selected light source and another selected light source) and switching the illumination so that it appears to the same color for humans. That is, by reducing the occurrence of flicker, for example, an abnormal part can be accurately identified during surgery, and the burden on the patient can be minimized by performing treatment only on that part.

  In addition, you may make it provide the control part which controls a shutter speed for every light source to image. In other words, by adjusting and optimizing the wavelength, light intensity, photosensitive property, and / or shutter speed according to the characteristics of the object to be imaged and the characteristics of the acquired image, the target image can be obtained with higher accuracy. Can be obtained.

  The following experiment was conducted on the imaging apparatus according to the present invention.

(1) LED spectrum measurement In the following experiment, LED illumination was used as a light source. LED lighting may have different characteristics from those described in the data sheet. In order to know the correct spectral characteristics, the spectrum of the LED used for illumination was measured. Measurement was performed in a dark room, and 27 types of LEDs were measured one by one with a spectroscope. The spectrometer used was a Jaz spectrometer module from Ocean Optics. Since some of the measured LEDs had similar spectral characteristics, 15 types of LEDs were selected from 27 types of LEDs. The spectrum of 15 types of selected LEDs is shown in FIG.

(2) Acquisition of spectral image It is necessary to confirm whether a spectral image can be acquired by an active illumination method using LED illumination. For comparison purposes, a spectral image of an adult male arm was obtained using a liquid crystal tunable filter (LCTF). The liquid crystal tunable filter used is Varispe of CRi, and the measurement wavelength range is 400-720 nm. In this example, a spectral image at 653 nm was acquired. This is because in the wavelength region near the red light, the light traveling inside the human body is absorbed by the reduced hemoglobin in the vein, so that the reflected light of the vein portion becomes weak and the vein portion appears dark in the spectroscopic image. Photographing was performed in a dark room, an Apogee cooled CCD camera Alta260 was used for photographing, and an artificial solar illuminating lamp SOLAX XC-100E from Celic was used as a light source. The temperature of the CCD sensor during photographing is -20 degrees, and the exposure time is 2 seconds. Since the captured image contains noise, the noise image captured under the same conditions with the shutter closed was subtracted. The results are shown in FIG. It can be confirmed that the veins appear dark in the image.

  Next, a spectral image of the arm was acquired by a simple active illumination method. Illumination was created using a red light LED with a dominant wavelength of 653 nm and an infrared light LED with a dominant wavelength of 834 nm. Eight LEDs were arranged on a breadboard and switched using an arduino. The combination pattern matrix P of LEDs in a simple active illumination system is a unit matrix. Photographing was performed in a dark room, and a cooled CCD camera Alta260 manufactured by Apogee was used in the same manner as the filter system. The sensor temperature is -20 degrees and the exposure time is 0.04 seconds. A 653 nm spectral image acquired by a simple active illumination method is shown in FIG. It can be confirmed that the vein appears dark like the image by the filter method.

  Subsequently, flickerless LED lighting was created using the technique of the present invention. An LED having a dominant wavelength of 653 nm and an LED having a dominant wavelength of 834 nm are used as target LEDs (selected light sources), and the remaining 13 types of LEDs are used as extra LEDs (non-selected light sources) that are not directly related to spectral image acquisition. The combination of LEDs was determined by the inventive method. The obtained P and P ′ are shown in Formula (16) and Formula (17), respectively.

  Based on the calculation results, lighting was prepared by arranging five types of LEDs on a breadboard and controlled by arduino. The brightness of the LED was adjusted by PWM. The created illumination is shown in FIG. Further, FIG. 10 shows a result of comparison of flicker at the time of switching the LED illumination. FIG. 10 (A) reproduces the state of switching by arranging two types of illumination images created as simple LED illumination, and FIG. 10 (B) is created as flickerless LED illumination according to the present invention. The state of switching is reproduced by arranging three types of illumination images. As is clear from FIGS. 5A and 5B, it can be confirmed that flicker is suppressed as compared with simple LED illumination. The same cooled CCD camera was used for photographing, and photographing was performed in a dark room. The sensor temperature was set to -20 degrees and the exposure time was set to 0.04 seconds. FIG. 8C shows a 653 nm spectral image acquired by the flickerless active LED illumination method. As the value of the color matching function and the spectrum value of the LED, values in increments of 10 nm from 390 nm to 830 nm were used. As with the other images, it can be confirmed that the veins appear dark.

Next, as an experiment that assumed a situation closer to actual application, we identified congestion using the spectral characteristics of hemoglobin. Hemoglobin includes oxygenated hemoglobin HbO ₂ bonded to oxygen and reduced hemoglobin Hb not bonded to oxygen. The proportion of hemoglobin that is oxyhemoglobin is called oxygen saturation and serves as an index for knowing the oxygen concentration in the blood. The light absorption characteristics of each hemoglobin are shown in FIG. As shown in FIG. 11, there is a large difference in absorbance near red light, but there is almost no difference in absorbance near infrared light. As a result, there is almost no difference between normal and abnormal images dispersed at wavelengths near infrared light, but there is no difference between normal and abnormal images dispersed at wavelengths near red light due to differences in oxygen saturation. There is a difference in brightness. The difference between the two spectral images is obtained, and the change in the oxygen saturation can be known by comparing with the normal one. Congestion is a familiar situation where oxygen saturation is low. Fingers tied with a rubber band and letting blood congested and fingers not doing anything were photographed at the same time, and spectral differences were obtained from the acquired spectral images and compared. The photographer used simple LED lighting and flickerless LED lighting. Spectral reflectance R ￣ (bar) at ₆₅₃ nm The spectral reflectance R ￣ (bar) _{834 at 653} and ₈₃₄ nm was obtained by Equation (18) and Equation (19), respectively.

Here, I ₆₅₃ and I ₈₃₄ represent a spectral image at ₆₅₃ nm and a spectral image at ₈₃₄ nm, respectively, and IR ₆₅₃ and IR ₈₃₄ represent a spectral image at ₆₅₃ nm and a spectral image at ₈₃₄ nm of the standard white plate, respectively. The spectral reflectance of the standard white plate is constant in the measurement wavelength range. FIG. 12 shows a 653 nm spectroscopic image obtained from equation (18) and an 834 nm spectroscopic image obtained from equation (19), respectively. 12A is a 653 nm spectral image, and FIG. 12B is an 834 nm spectral image. Further, FIG. 13 shows an image photographed by a normal RGB camera and an image of the difference in spectral reflectance obtained from equation (20). FIG. 13A is an image taken by an RGB camera, and FIG. 13B is an image of a difference in spectral reflectance.

  Since the normal part contains a lot of oxyhemoglobin, the difference in spectral reflectance between the two wavelengths is large, and the image appears bright. On the other hand, when venous blood accumulates due to congestion, the proportion of deoxyhemoglobin increases, so the difference in spectral reflectance between the two wavelengths decreases and the image appears dark. As described above, from the imaging apparatus according to the present invention, it has been clarified that it is possible to suppress the generation of flicker in the active illumination method and to visually check the oxygen saturation concentration in the blood. .

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging element 12 CPU
13 RAM
14 ROM
15 Input / output I / F
DESCRIPTION OF SYMBOLS 16 Display 17 Light source 21 Input part 22 Light source selection part 23 Light source information storage part 24 Light source extraction part 26 Input information 27 Imaging target part 28 Imaging information storage part 29 Image removal part 30 Display control part

Claims (6)

Multiple light sources emitting at different wavelengths;
A selection unit that selects a plurality of the light sources that emit light at a specific wavelength specified based on characteristics of the imaging target among the light sources;
Among the plurality of selected selected light sources, the light source having a wavelength that reduces the color difference between the one selected light source and the other selected light source by being combined with one selected light source is a non-selected light source other than the selected light source. Extraction means for extracting from the selected light source;
An image pickup apparatus comprising: an image pickup unit configured to pick up an image pickup object by sequentially switching the light source for each selected light source by combining the selected selected light source and the extracted non-selected light source. The imaging device according to claim 1,
An image removing means for removing an image picked up by the non-selection light source extracted by the extracting means from the image picked up by the image pickup means;
An image pickup apparatus comprising: display control means for displaying an image remaining after removal processing by the image removal means. The imaging device according to claim 1 or 2,
The image pickup apparatus, wherein the extraction unit extracts a non-selected light source having a wavelength band in the vicinity of another selected light source with respect to the one selected light source as a non-selected light source to be combined with the one selected light source. The imaging device according to claim 3.
The image pickup apparatus, wherein the extraction unit extracts a non-selection light source having a wavelength band other than the vicinity of the other selection light source according to external light in an imaging environment. The imaging device according to any one of claims 1 to 4,
The image pickup apparatus, wherein the extraction unit extracts the light source based on a wavelength band and light intensity of the non-selected light source. The imaging apparatus according to claim 5,
The image pickup apparatus, wherein the extraction unit extracts the light source based on the light intensity in consideration of photosensitive characteristics.