JP2019128295A - Imaging device, imaging system, and imaging method - Google Patents

Abstract

To generate a pseudo color image of a sample.SOLUTION: An imaging device according to one embodiment comprises a light detection unit for acquiring a plurality of detection results corresponding to a wavelength by irradiating a sample with a plurality of infrared light beams different in wavelength from one another, and a control unit for generating a pseudo color image of the sample by allocating color tones of colors to the plurality of detection results, respectively on the basis of a spectrum in an observation object of the sample obtained from the plurality of detection results. The control unit allocates a red color tone to the plurality of detection results on the basis of the spectrum and allocates color tones other than red to the plurality of detection results with the red color tone as a reference.SELECTED DRAWING: Figure 2A

Description

  The present disclosure relates to an imaging device, an imaging system, and an imaging method.

  In the field of medicine and the like, there has been proposed a technique of imaging a tissue of a living body and utilizing the image for various diagnoses, examinations, observations and the like (see, for example, Patent Document 1). Further, under the present circumstances, it is difficult to develop an imaging device having good sensitivity in both the infrared light wavelength range and the visible light wavelength range. Therefore, in the case of simultaneously displaying the infrared light image and the visible light image on the screen of the display device, it is necessary to use two cameras, an infrared light camera and a visible light camera.

  However, if separate photographing optical systems are used for the infrared light camera and the visible light camera, it is difficult to align the images when the infrared light image and the visible light image are displayed in a superimposed manner.

JP, 2006-102360, A

  According to the present embodiment, by irradiating the sample with a plurality of infrared light of different wavelengths, a light detection unit for acquiring a plurality of detection results corresponding to the wavelength, and an observation target of the sample obtained from the plurality of detection results And a controller configured to assign different color tones to each of the plurality of detection results based on the spectrum in the spectrum to generate a pseudo color image of the sample, and the control unit An imaging device is provided, which assigns tones and assigns tones other than red to a plurality of detection results based on red tones.

  According to the present embodiment, the light detection unit irradiates the sample with a plurality of infrared light having different wavelengths to obtain a plurality of detection results corresponding to the wavelength, and the control unit generates a plurality of detection results from the plurality of detection results. The pseudo color image of the sample is assigned by assigning the red tones of the plurality of detection results based on the spectrum of the observation target of the obtained sample and assigning the tones other than red to the plurality of detection results based on the red tone. An imaging method is provided, including generating.

  In addition, according to the present embodiment, there is provided an imaging system including the imaging device of the above aspect and a display device that displays a pseudo color image of a sample.

It is a figure for demonstrating the example of an external appearance structure of the imaging system 1 which concerns on this embodiment. FIG. 2 is a diagram showing an example of a functional block configuration of an imaging system 1; It is a figure which shows the structural example of the table 1021 for RGB allocation by this embodiment. Example of a screen on which a visible light image of a sample is displayed on the monitor 1_51 and example of a screen on which a composite image in which a highlighted image of a sample is superimposed on a pseudo color image of a sample on the monitor 2_52 (example of two monitor configuration) FIG. It is a figure which shows the example of a screen (1 monitor 2 screen structure) on which the visible light image and the background image on which the emphasis image was superimposed are displayed on the display apparatus (monitor) 50 in this embodiment. Generation and display processing of a pseudo color image and a composite image in the imaging system (surgery support system) 1 of the present embodiment (a pseudo color image of the sample is generated, and a composite obtained by superimposing the enhanced image of the sample on the pseudo color image It is a flowchart for demonstrating the content of the process which displays an image. It is a figure (photograph) for demonstrating the outline | summary of the pseudo | simulation color image generation process by this embodiment. It is a graph which shows the relationship of the wavelength of a fat tissue (solid line) and stomach outer wall (broken line), and a spectral value (luminance value) by this embodiment, Comprising: About wavelength selection of the infrared light used when producing | generating a pseudo color image It is a figure for demonstrating. In the present embodiment, an example of an inappropriate pseudo color image (eg, R value is image data obtained by irradiating 1600 nm, G value is 1450 nm, and image data obtained by irradiating 1450 nm, B value is irradiated 1500 nm Are respectively assigned to the image data obtained in this manner. It is a figure for demonstrating the example (TYPE1) of wavelength selection by this embodiment. In TYPE1, the wavelength range is determined from the spectrum of the outer stomach wall. It is a figure which shows the wavelength range selection example (TYPE1-a) by this embodiment. It is a figure which shows the example of wavelength range selection (TYPE1-b) by this embodiment. It is a figure which shows the wavelength range selection example (TYPE1-c) by this embodiment. It is a figure which shows the wavelength range selection example (TYPE1-d) by this embodiment. It is a figure which shows the wavelength range selection example (TYPE1-e) by this embodiment. It is a figure which shows the wavelength range selection example (TYPE1-f) by this embodiment. It is a figure for demonstrating the example (TYPE2) of wavelength selection by this embodiment. It is a figure which shows the wavelength range selection example (TYPE2-a) by this embodiment. It is a figure which shows the wavelength range selection example (TYPE2-b) by this embodiment. It is a figure which shows the wavelength range selection example (TYPE2-c) by this embodiment. It is a figure which shows the wavelength range selection example (TYPE2-d) by this embodiment. It is a figure which shows the wavelength range selection example (TYPE2-e) by this embodiment. It is a figure which shows the wavelength range selection example (TYPE2-f) by this embodiment. It is a photograph which shows Example 1 of the false color picture generated based on the technique by this embodiment. It is a photograph which shows Example 2 of the false color picture generated based on the technique by this embodiment. It is a photograph which shows Example 3 of the false color picture generated based on the technique by this embodiment. It is a photograph which shows Example 4 of the false color image generated based on the technique by this embodiment. It is a photograph which shows Example 5 of the false color image generated based on the technique by this embodiment. It is a photograph which shows Example 6 of the false color image generated based on the technique by this embodiment. It is a photograph which shows Example 7 of the false color image generated based on the technique by this embodiment. It is a photograph which shows Example 8 of the false color image generated based on the method by this embodiment. A process of detecting a concavo-convex shape on the organ surface (as an example, the inner wall of a pig stomach) according to the present embodiment and generating a concavo-convex weighted image emphasizing the concavo-convex shape; It is a photograph showing an outline of processing to be superimposed on an image. It is a figure showing an example of composition of GUI (Graphical User Interface) used with imaging system 1 of this embodiment. It is a photograph which shows the emphasis image of the liquid leak (bile as an example) of the surface of a sample (a liver as an example), the pseudo color image of a sample, and those superimposed images by this embodiment. It is a photograph which shows the emphasis picture of a lymph node in a sample (for example, mesentery + lymph node as an example), the false color picture of a sample, and those superposition pictures by this embodiment.

  Hereinafter, the present embodiment will be described with reference to the attached drawings. In the attached drawings, functionally the same elements may be denoted by the same numbers. Although the attached drawings show embodiments and implementation examples in accordance with the principles of the present disclosure, these are for the understanding of the present disclosure and are not used for a limited interpretation of the present disclosure. is not. The descriptions in this specification are merely exemplary, and are not intended to limit the scope of the claims or the application in any way whatsoever.

  While this embodiment is described in sufficient detail to enable those skilled in the art to practice the present disclosure, other implementations are possible, without departing from the scope and spirit of the technical idea of the present disclosure. It is necessary to understand that it is possible to change the configuration / structure and replace various elements. Therefore, the following description should not be interpreted in a limited manner.

  Furthermore, as described later, the present embodiment may be implemented by software running on a general-purpose computer, or may be implemented by dedicated hardware or a combination of software and hardware.

<On the significance of generating a pseudo color image>
In recent years, when performing surgery on a specific site (surgery site) of a patient, for example, the affected site is emphasized using image data of the surgery site acquired by an infrared light camera having high sensitivity in the infrared light wavelength range. It is performed to generate an enhanced image and to superimpose it on the image of the surgical site. An image of such a surgical site is preferably a color image in consideration of the ease of confirmation by a doctor or the like. Therefore, in an imaging system (also referred to as a surgery support system), as an example, in addition to an infrared light camera, a visible light camera that generates a color image is provided, and a visible light image that is a color image and the above-described enhanced image are superimposed. Make it display together.
However, unlike the arrangement position of the visible light camera and the arrangement position of the infrared light camera in the imaging system, the optical axes of both cameras do not match, and therefore, when the visible light image and the enhanced image are superimposed, positional deviation occurs. It may happen. Therefore, for example, a method using the same optical system (coaxial optical system) may be considered in order to prevent the optical axes of both cameras from shifting, but in this case, a long distance from the visible light wavelength range to the infrared light wavelength range Chromatic aberration must be corrected over the wavelength range. With the correction of the chromatic aberration, the optical system has a complicated configuration (for example, the number of lenses increases and an optical element such as a half mirror for making light incident on each of the infrared light camera and the visible light camera is required) As a result, there is a problem that the imaging system itself is upsized.

  Therefore, in the present embodiment, the imaging system may be provided with a visible light camera, for example, image data acquired by an infrared light camera (eg, rough irradiation of biological tissue with infrared light of a specific infrared wavelength band) A pseudo color image of a living tissue is generated (combined) by image processing using the luminance value (or light intensity) obtained in this way (for example, RGB (RGBA), CMY (CMYK), HSV, Data (color tone) of a color space such as HLS is assigned to generate a pseudo color image), and the pseudo color image is displayed on the display screen. In the imaging system, for example, when the enhanced image and the color image generated using the image data acquired by the infrared light camera are superimposed, the pseudo color image and the enhanced image are superimposed. As a result, since the enhanced image and the pseudo color image are generated based on the image data acquired by the same infrared camera, there is no need to align the optical axes, and the system configuration can be simplified (miniaturized). System cost can also be reduced. In the following description, an example in which a pseudo color image is generated by applying (assigning) an RGB value in an RGB color space to image data acquired by an infrared light camera as a color tone is described below. If the RGB assignment table 1021 (tone conversion table) is replaced with another color space table (tone conversion table), it is possible to generate a pseudo color image using the tones of another color space.

<Configuration of Imaging System 1>
FIG. 1 is a diagram for explaining an example of an appearance configuration of an imaging system 1 according to the present embodiment. The imaging system 1 is used, for example, for medical support such as pathological diagnosis support, clinical diagnosis support, observation support, and surgery support. As shown in FIG. 1, in the present embodiment, a surgery support system (a surgery imaging system, a medical support imaging system) will be described as an example of the imaging system 1.

  The imaging system 1 includes, for example, a control device (control unit) 10 that controls the entire imaging system 1 and infrared light that irradiates the living body 80 (hereinafter, the imaging region in the living body 80 may be referred to as a “sample”) An infrared light source unit 20 (reference numeral 20 is not shown in FIG. 1) including infrared light sources 21 and 22 emitting light, and an imaging unit (light detection unit) 30 for imaging radiation light from a living body 80; It includes an input device 40 used when an operator (for example, a doctor etc.) inputs various data and an instruction command to the control device 10, a monitor 1_51 (first display) and a monitor 2_52 (second display). A display device (display unit) 50 for displaying an image, characters, and the like captured by the unit 30 and a surgical shadow lamp 60 communicably connected to the control device 10 are provided. The imaging system 1 can also be said to be an imaging device including at least the imaging unit 30. Further, in FIG. 1, the two-monitor configuration is adopted as the display device 50, but a two-screen configuration may be adopted by one monitor. Furthermore, the infrared light from each of the infrared light sources 21 to 22 of the infrared light source unit 20 may be configured, for example, to uniformly irradiate the light amount to the sample through the diffusion plate. For example, a diffusion plate can be attached to the surface from which the light of each light source is emitted, or the diffusion plate can be disposed in the light path between the infrared light source unit 20 and the living body 80. Thereby, light is uniformly emitted to the sample, and it is possible to prevent illumination unevenness, shadows, and the like from being generated in the imaging range (for example, the range corresponding to the surgical field) by the imaging unit 30. .

  The living body 80 is, for example, a patient lying on the operating table 90. For example, an image of a surgical site of a living body (patient) 80 is captured by the imaging unit 30. The surgical site (site to be imaged: imaging site) can also be referred to as an affected area, a sample or a target. In addition, the affected area, the sample and the target each include an observation target (for example, an organ such as a site, an adipose tissue, an outer wall of a stomach (for example, including a stomach, a liver, an mesentery, a liver, or a pancreas)).

FIG. 2A is a diagram illustrating a functional block configuration example of the imaging system 1. As described above, the imaging system 1 includes a control device 10, an infrared light source unit 20, an imaging unit 30, an input device 40, a display device (display unit) 50, a surgical operating light 60, and an imaging unit. And 30, a stage 70 (not shown in FIG. 1) that slides 30 in a horizontal direction, for example. The control device 10 includes, for example, a computer, and a control unit 101 including a processor and the like, various programs, parameters, imaging results, and image data of samples acquired by an infrared light camera included in the imaging unit 30 ( For example, a storage unit 102 that stores an RGB assignment table 1021 that assigns color data to each pixel based on luminance values) is provided.
The control unit 101 reads various programs and parameters from the storage unit 102, expands the various programs read into an internal memory (not shown), and performs various types according to instructions input from the input device 40 and information processing sequences specified by the various programs. Execute program processing. The control unit 101 includes, for example, a light irradiation control unit 1011 that controls irradiation of infrared light of the infrared light source unit 20, and a highlight image generation unit 1012 that generates a highlight image of a sample from image data acquired from the imaging unit 30. An image that generates a pseudo color image of the sample based on the image data acquired from the imaging unit 30 and superimposes the enhanced image of the sample on the pseudo color image of the sample according to an instruction input from the input device 40; A generation unit 1013 and an image correction unit 1014 that performs predetermined correction processing on a pseudo color image or a composite image according to an instruction input by the input device 40. The storage unit 102 may, for example, at least a program corresponding to the light irradiation control unit 1011, the enhanced image generation unit 1012, the image generation unit 1013, and the image correction unit 1014, and image data acquired by the infrared light camera 32 (for example, And an RGB allocation table 1021 used for colorizing an image by allocating RGB to the luminance value). Note that, for example, the stage device 70 can move the imaging unit 30 relative to the sample.

  FIG. 2B is a diagram showing a configuration example of the RGB assignment table 1021 according to the present embodiment. The RGB assignment table 1021 is, for example, image data (spectral data (brightness value)) obtained by detecting reflected light (including scattered light) obtained by irradiating a sample with infrared light having a first wavelength with an infrared camera. R assignment table 10211 for assigning R value (which may also be referred to as red tone or red color information) and reflected light (including scattered light) obtained by irradiating the sample with infrared light of the second wavelength G allocation table 10212 for assigning a G value (hereinafter also referred to as green color tone or green color information) to image data (spectrum data (luminance value)) detected by an infrared camera; B value (hereinafter referred to as blue tone) or image data (spectrum data (brightness value)) obtained by detecting reflected light (including scattered light) obtained by irradiating the sample with infrared light of three wavelengths to the infrared light camera And B allocation table 10213 to assign also possible) be referred to as color information of the system, and a. As will be described in detail later, since the color of the sample as a whole is often reddish as a whole, the image data to which the R value is assigned has a luminance value more than the image data to which the G value or B value is assigned. High is desirable. Therefore, for example, it is desirable that the first wavelength has a wavelength range in which the obtained luminance value is larger than the second and third wavelengths.

  Returning to FIG. 2A, the description of the configuration of the imaging system 1 will be continued. The infrared light source unit 20 emits (emits) infrared light of at least a part of a wavelength range of, for example, 900 nm to 2500 nm (or 800 nm to 3000 nm, or 900 nm to 1650 nm, or 1000 nm to 1700 nm). And 22. Although FIG. 1 illustrates an example in which the infrared light source unit 20 includes two light sources, three or more light sources may be included. The infrared light source unit 20 divides light having a wide wavelength band emitted (radiated) from, for example, the infrared light sources 21 and 22 with an optical system, and filters each of the separated light with an optical filter disposed in the optical path. It can also be configured to generate light of a desired wavelength. When the imaging unit 30 described later includes a hyperspectral camera as an imaging device, the radiation (reflection) light from the sample is split (wavelength resolved) in the hyperspectral camera to correspond to the spectral data (each wavelength) Since the luminance value is detected, the light does not have to be split by the above optical system. When using a hyperspectral camera, it is necessary to move either the camera or the sample (subject) relative to the stage or the like. For this reason, the imaging system 1 of FIG. 1 may include, for example, moving means (stage device 70) such as a camera or a stage for relatively moving a sample.

  For example, the infrared light source 21 and the infrared light source 22 of the infrared light source unit 20 are configured to emit (emit) infrared light of different wavelengths and irradiate the sample (the living body 80) with each light source It may be switched and used. The wavelength of light emitted (radiated) by each of the infrared light sources 21 and the infrared light sources 22 included in the infrared light source unit 20 may be set by the operator. In the second embodiment, an example of GUI (Graphical User Interface) of wavelength setting is described (see FIG. 31).

  For example, the imaging unit 30 includes a first imaging device (an imaging device having high detection sensitivity to light in a visible light region, a visible light camera) 31 for capturing a visible light image of the sample, and an infrared light image of the sample And a second imaging device (imaging device having high detection sensitivity to light in the infrared light region, infrared light camera) 32 to image. However, in order to realize the imaging system 1 of the present embodiment, the first imaging device (visible light camera) 31 is not essential, and may be provided when it is desired to display a visible light image on the display device 50. The second imaging device 32 is, for example, from an infrared light sample in at least a part of a wavelength band of 900 nm to 2500 nm emitted (radiated) from an infrared light source (infrared light source 21 or infrared light source 22). The hyperspectral camera can be configured to split (wavelength-resolve) radiation (reflected) light and detect spectrum data (brightness value) corresponding to each wavelength. Further, as another form, the second imaging device 32 applies normal infrared light of a plurality of wavelengths to the sample and detects an intensity (intensity value) of light emitted from the sample to obtain an image. An external light camera (an infrared light camera that captures an image without wavelength decomposition) may be used. In this case, as the infrared light of a plurality of wavelengths to be irradiated, for example, a plurality of types of light (infrared light) are selected from wavelengths of 900 nm to 2500 nm. For example, a silicon (Si) camera can be used as the first imaging device 31. As the second imaging device 32, for example, an InGaAs camera (hyperspectral camera) using InGaAs (indium potassium arsenic) as a sensor can be used. The optical axis of the first imaging device 31 and the optical axis of the second imaging device 32 may not be the same as shown in FIG.

  The input device 40 is a device including, for example, a keyboard, a mouse, a microphone, a touch panel, etc., and used by the operator (such as a doctor) to input an instruction or parameter when causing the control device 10 to execute a predetermined process. is there. Also, for example, simply inserting a semiconductor memory such as USB into an input port (not shown) provided in the control device 10 allows the control unit 101 of the control device 10 to automatically transmit data or instructions from the semiconductor memory (previously The various programs may be executed by reading an instruction described in a determined rule.

  In the display device 50, the control unit 101 corrects an image (for example, a visible light image of a sample, a pseudo color image of a sample, an enhanced image of a sample) generated by the control unit 101 or an image (for example, pseudo color image). The obtained corrected image (corrected pseudo color image) is received from the control device 10, an image obtained by superimposing the enhanced image of the sample on the generated image (visible light image of the sample or the pseudo color image of the sample) or the pseudo color image An image in which the emphasis image is superimposed on the color image is displayed on the display screen. The display device 50 includes, for example, a monitor 1_51 and a monitor 2_52. The monitor 1_51 displays, for example, a visible light image. The monitor 2_52 displays, for example, an image obtained by superimposing the enhanced image on the visible light image and / or the infrared light image, or an image obtained by superimposing the corrected and enhanced image on the visible light image and / or the infrared light image on the display screen.

  FIG. 3A shows an example of a screen in which a visible light image of a sample is displayed on a monitor 1_51 and an example of a screen in which a composite image in which a highlighted image of a sample is superimposed on a pseudo color image of a sample is displayed on a monitor 2_52 (Example of). Here, the composite image generated by superimposing the emphasis image on the pseudo color image is displayed on the monitor 2_52, but even if the pseudo color image on which the emphasis image is not superimposed is displayed on the monitor 2_52. Good. Further, in the imaging system 1, the optical axis of the first imaging device 31 and the optical axis of the second imaging device do not have to match, so the visible light image displayed on the monitor 1_51 and the monitor 2_52 are displayed. The pseudo color image or the composite image in which the emphasis image is superimposed on the pseudo color image includes some positional deviation. In addition, the display color of the emphasized image can be appropriately designated / changed by, for example, the operator.

The surgical shadow lamp 60 is a visible light source configured of a plurality of LED light sources. The surgical shadow lamp 60 is very bright, for example, up to 160000 lux. While the surgical shadow lamp 60 is on, a visible light image can be acquired by the first imaging device 31. On the other hand, since the surgical operating lamp 60 is configured not to emit light in the infrared wavelength region, infrared light is emitted during both the lighting and extinguishing periods of the surgical operating lamp 60. An image may be acquired by the second imaging device 32.
3A shows a form in which a visible light image is displayed on one of two monitors and a pseudo color image or a composite image in which a sample emphasized image is superimposed on a pseudo color image is displayed on the other monitor. As shown in FIG. 3B, one display device (monitor) 50 has a two-screen configuration of a visible light image and a composite image (for example, both images are displayed in the same size, or at least a portion of one image It may be displayed superimposed on another image, or one image may be displayed larger than the other image). In this case, the control unit 101 controls the display position of a plurality of screen areas DA (for example, the first screen area DA1, the second screen area DA2) in the display device (monitor) 50. Further, for example, the lighting and extinguishing of the surgical shadow lamp 60 may be controlled by the control device 10.

  As described above, according to the imaging system (surgery support system) 1 of the present embodiment, for example, a living body part (for example, a surgical target part of a patient) in a surgical field detected by one imaging device (infrared light camera): Based on the image data of the sample), an image (a pseudo color image) of the living body site and an enhanced image of the living body site are generated and displayed superimposed on each other, so that alignment of the images becomes unnecessary. In addition, it is not necessary to provide an optical system for aligning the optical axis of the visible light camera with the optical axis of the infrared light camera, and the imaging optical system of the sample is simplified. Therefore, the imaging system 1 can be miniaturized, and the cost can be suppressed. Hereinafter, further details of the operation and configuration of the imaging system (surgery support system) 1 according to the present embodiment will be described.

<Contents of generation / display processing of pseudo color image and composite image>
FIG. 4 is a process for generating and displaying a pseudo color image and a composite image in the imaging system (surgery support system) 1 of this embodiment (a pseudo color image of a sample is generated, and a sample emphasized image is superimposed on the pseudo color image). It is a flowchart for demonstrating the content of the process which displays the synthesized image obtained by this. Each step will be described below. In each step, each processing unit (light irradiation control unit 1011, enhanced image generation unit 1012, image generation unit 1013) is described to execute the processing of each step, but each processing unit is included in control unit 101. The control unit 101 may be an operation subject because it is a function to be performed.

(I) Step 401
The light irradiation control unit 1011 controls the infrared light source unit 20 in response to an instruction input (input signal) of the operator (doctor or the like) using the input device 40, and the above-mentioned first wavelength (RGB R value) Assign the wavelength of infrared light for detecting the luminance value to be assigned, the second wavelength (the wavelength of infrared light for detecting the luminance value for assigning the RGB G value), and assign the third wavelength (B value of RGB) The sample is irradiated with infrared light of the wavelength of infrared light for detecting the luminance value. Although details will be described later, for example, the first wavelength may be 1070 nm, the second wavelength may be 1000 nm, and the third wavelength may be 970 nm. Here, the case where the infrared light source unit 20 irradiates infrared light of the first to third wavelengths (that is, infrared light of three single wavelengths) to the sample is taken as an example, but the specific wavelength band Infrared light (for example, infrared light in at least a part of the wavelength band from 900 nm to 2500 nm) may be irradiated to the sample. In this case, the above-described hyperspectral camera can be employed as the second imaging device 32. Further, as the second imaging device 32, it is also possible to use a light receiving sensor capable of acquiring a plurality of spectrum data in one shooting of one pixel of an image.

(Ii) Step 402
The image generation unit 1013 controls the second imaging device 32 to detect reflected light (including scattered light) from the sample obtained by irradiating the sample with each of infrared light having the first to third wavelengths. And obtain spectral data (image data: luminance value) corresponding to each wavelength. The image generation unit 1013 stores, for example, the acquired image data in the storage unit 102 in association with each wavelength. The stored image data is read from the storage unit 102 when performing the subsequent steps, and is used to generate a pseudo color image or an enhanced image.
In the case where a hyperspectral camera is used as the second imaging device 32, for example, the second imaging device 32 sets radiation (reflected) light from the sample obtained by irradiating infrared light to the sample to each wavelength. While spectrally separating, spectral data (brightness value) corresponding to each wavelength is detected. Note that the luminance value data acquired by the camera is corrected by the luminance value of a reference object for correction (for example, a standard reflector, an object matched to the shape of the sample, etc.) acquired in advance, and after this correction The luminance value data of may be used for analysis. Therefore, it is also possible to use the luminance value data after correction as luminance value (corrected luminance value) data used for analysis. Further, in the case of using a light receiving sensor (multispectral sensor) capable of acquiring a plurality of spectrum data in one imaging of one pixel of an image as the second imaging device 32, for example, the second imaging device 32 has a predetermined wavelength band Spectral data obtained by irradiating the sample with light of N wavelength bands selected from (for example, a wavelength band of 900 nm to 2500 nm) is detected. For example, the second imaging device 32 can acquire three-dimensional infrared light reflection spectrum data including a sample surface position (corresponding to a pixel) and a wavelength direction. When a hyperspectral camera or the above multispectral sensor is used as the second imaging device 32, a pseudo color image may be generated using a part of the acquired spectrum data. For example, pseudo color images can be generated by assigning each value of RGB to spectrum data (brightness value of each pixel) corresponding to three specific wavelengths (first to third wavelengths).

(Iii) Step 403
The image generation unit 1013 determines and determines the ratio of the R value, the G value, and the B value based on, for example, the luminance value of each pixel of the sample obtained by irradiating the first wavelength to the third wavelength. The R, G, and B values are mixed to determine the color of each pixel. In this embodiment, with respect to the same pixel, a plurality of types of luminance values (for example, three types: luminance values obtained by irradiating the sample with infrared light of the first wavelength, irradiation of the sample with infrared light of the second wavelength) To obtain the luminance value obtained by irradiating the sample with infrared light of the third wavelength, but the ratio of these luminance values differs depending on the pixel. In the present embodiment, for example, differences in luminance values corresponding to respective wavelengths in the same pixel are regarded as differences in RGB composition ratios, and R values, G values, and B values are calculated according to the ratio of luminance values in the same pixels. It is assigned to each pixel.

(Iv) Step 404
The image generation unit 1013 mixes the R value, the G value, and the B value assigned to each pixel in step 403 to determine the color of each pixel, and generates a pseudo color image of a sample.

(V) Step 405
The emphasized image generation unit 1012 reads image data corresponding to each wavelength from the storage unit 102, and generates a emphasized image of a sample. The enhanced image can be generated, for example, by generating a difference image of infrared light images corresponding to the respective wavelengths. If the number of generated infrared light images is N (the N types of wavelengths are irradiated to obtain infrared light images), (N × (N−1)) difference images are generated. . In addition, each infrared light image is subjected to an edge filter, and teacher data of a living body part (organ) to be observed included in the sample is stored in the storage unit 102 in advance, and each infrared light image and teacher data are The emphasized image may be generated by calculating the difference of.

(Vi) Step 406
The image generation unit 1013 superimposes the enhanced image of the sample generated in step 405 on the pseudo color image of the sample generated in step 404 to generate a composite image. Since the pseudo color image and the enhanced image are generated based on the image data (spectral data) acquired by the same infrared light camera (second imaging device 32), they are superimposed without causing positional deviation. It is possible. For example, when a plurality of emphasized images of a sample are generated in step 405, an emphasized image corresponding to a wavelength designated by an operator (such as a doctor) may be superposed on a pseudo color image, or all The emphasized image may be a target to be superimposed on the pseudo color image. The generated composite image is held in the storage unit 102, for example.

(Vii) Step 407
The control unit 101 transfers the composite image generated in step 406 to the display device 50 and instructs the display device 50 to display the composite image on the display screen. For example, when there are a plurality of composite images, the control unit 101 may instruct switching display. The display device 50 receives the composite image from the control unit 101, and displays, for example, the composite image on the monitor 2_52. Further, the display device 50 may also receive a pseudo color image of the same sample from the control unit 101 and display it on the monitor 1_51, for example. The control unit 101 transfers a visible light image (an image having an imaging axis different from that of the infrared light camera) of the sample captured by the visible light camera (the first imaging device 31) to the display device 50. It may be instructed to display on a display screen together with a color image or the like.

  As described above, the imaging system 1 applies a plurality of infrared light images obtained by irradiating infrared light of a specific plurality of wavelengths according to the luminance value corresponding to the specific plurality of wavelengths in each pixel. Assign color information in RGB space to generate a sample pseudo color image. Then, since the pseudo color image of the sample and the enhanced image of the sample are generated using the spectral data acquired using the same imaging device (second imaging device 32), the sample is not displaced. The image (pseudo color image) and the enhanced image can be superimposed. Further, according to the method of the present embodiment, a realistic pseudo color image that is not inferior to a visible light image can be provided. In the present embodiment, the pseudo color image and the enhanced image are generated based on the image data acquired using the same imaging device, so the imaging optical system can be simply configured, and the imaging system 1 is provided inexpensively. You will also be able to

<Outline of pseudo color image generation processing (example)>
FIG. 5 is a diagram (photograph) for describing an outline of pseudo color image generation processing according to the present embodiment. In the example of FIG. 5, a pseudo color image is generated using 1070 nm as the first wavelength, 1000 nm as the second wavelength, and 970 nm as the third wavelength.

  First, infrared light of 1070 nm, infrared light of 1000 nm, and infrared light of 970 nm are sequentially irradiated to the sample to obtain spectral data (image data) corresponding to each wavelength.

  Next, based on the ratio of each luminance value corresponding to the three wavelengths of each pixel, the ratio of the value assigned to RGB is determined. For example, in each color (red, green, blue), colors are distributed from dark to light. If the luminance value is small, a dark color is assigned, and if the luminance value is large, a bright color is assigned. Further, for example, in a certain pixel, when the luminance value corresponding to the wavelength of 1070 nm is 100, the luminance value corresponding to the wavelength of 1000 nm is 50, and the luminance value corresponding to the wavelength of 970 nm is 30, 100: 50: 30 RGB values are assigned to the pixels at a rate of

  The RGB values assigned to each pixel are mixed to determine the color of each pixel, and a pseudo color image of the sample is generated. Although FIG. 5 shows the pseudo color image (state in which the enhanced image is not superimposed) generated as described above, the feature of the visible light image is well represented. Therefore, according to the present embodiment, it can be understood that the operator (such as a doctor) can provide a sufficiently referable image even when the pseudo color image is displayed on the display device 50 during the operation.

<Selecting the wavelength of infrared light used to generate a pseudo color image>
FIG. 6 is a graph showing the relationship between the wavelength and spectral value (brightness value: maximum value = 65535, minimum value = 0 (16 bits)) of fat tissue (solid line) and gastric outer wall (broken line) according to this embodiment, It is a figure for demonstrating wavelength selection of the infrared-light used when producing | generating a pseudo color image.

  In the case of generating a pseudo color image, it is necessary to use a wavelength range in which the luminance value (reflectance) is large to some extent so that the generated pseudo color image is not darkened. For example, it can be seen from FIG. 6 that in the relatively short wavelength range, the adipose tissue and the gastric outer wall can obtain luminance values of a certain value or more. On the other hand, in a long wavelength range, as shown in FIG. 7, a pseudo color image (eg, R value: image data obtained by irradiating 1600 nm, G value: 1450 nm, image data obtained by irradiating The values are assigned to the image data obtained by irradiating 1500 nm, respectively, and the image becomes a blackish image, which is far from the visible light image (macroscopic image). As described above, when the wavelength in the long wavelength range is used, the pseudo color image is blacked out. For this reason, it is necessary to consider which wavelength range is desirable. In this point, for example, as shown in FIG. 5, in the wavelength range shorter than 1370 nm to 1400 nm, the fat tissue and the gastric outer wall can obtain luminance values of a certain value or more. On the other hand, in the wavelength range longer than 1400 nm, especially the outer wall of the stomach has a significant decrease in luminance value due to the influence of water absorption. Referring to the graph of FIG. 6, for example, since a region near a wavelength of 1450 nm is a water absorption region, it is to be avoided to set that region as a wavelength for pseudo color image formation. As the wavelength goes to a wavelength range shorter than 1370 nm, the luminance value increases. Also, looking at the spectrum of fat tissue (solid line graph), the brightness value is a minimum (a minimum brightness value of about 17500) at a wavelength of around 1200 nm, and a horizontal line (two-dot chain line in FIG. 6) from this minimum point When the (reference line) is drawn in the long wavelength direction, it again intersects the spectrum (solid line graph) of the fat tissue, and this intersection point has a wavelength of 1370 nm. Therefore, if a wavelength band (shaded area) shorter than 1370 nm is used, a minimum luminance value of 17500 or more is ensured for fat tissue, but if an error of about 2 to 3% is taken into consideration, 1400 nm or less It can be seen that it is preferable to employ a wavelength range.

<Example of wavelength range selection: TYPE 1>
FIG. 8 is a diagram for explaining an example (TYPE 1) of wavelength selection according to the present embodiment. TYPE 1 is based on the idea of using a wavelength on the short wavelength side (wavelength range shorter than 1130 nm) for wavelength selection. In particular, the wavelength range is determined from the spectrum on the short wavelength side of the outer wall of the stomach. The application to other organs is considered based on the idea of TYPE1, which is the basic form, as described later. Since the targets for acquiring the spectrum are different, a slight difference occurs in the wavelength range between TYPE 1 (basic form) and the other wavelength range examples (TYPE 1-a to 1-f).

  For example, since the living body (sample, observation target) usually has a reddish color (see the visible light image (macroscopic image) in FIG. 7), the redness in the macroscopic image is reproduced by the infrared image For this purpose, the region with high reflectance (brightness value) (region with low absorption) is assigned to the red tone (in this case R), the green tone (in this case G) and the blue tone (in this case B) ) Is preferably assigned to an area having a reflectance (luminance value) smaller than R. Assuming this, if a horizontal line 801 (reference line) is drawn approximately halfway between the local minimum (near 970 nm) and the local maximum (near 1070 nm) in the spectrum on the short wavelength side of the gastric outer wall (broken line graph), adipose tissue and gastric outer wall In both of the above, the region where the luminance value is high is, for example, a region of 1031 ± 10 nm to 1120 ± 10 nm. Therefore, it is understood that it is better to assign this region to the R value and to assign the G value and the B value to the region of 955 ± 10 nm to 1030 ± 10 nm, which is a region where the luminance value is lower than that region. In TYPE1, for example, the wavelength range to which the R value is assigned is on the longer wavelength side than the wavelength range to which the G value and the B value are assigned. Hereinafter, various wavelength range selection based on the same idea as TYPE 1 above will be exemplified.

(I) TYPE 1-a
FIG. 9 is a view showing a wavelength range selection example (TYPE 1-a) according to the present embodiment. TYPE 1-a relates to an example of determining the wavelength range from the average spectrum of each portion (fat tissue and gastric outer wall) of the pig stomach, for example.

  In TYPE 1-a, the average spectrum of the fat tissue spectrum (solid line graph) and the stomach outer wall spectrum (broken line graph) is calculated, and in the average spectrum, a region where the luminance value is always higher than other regions is determined. , And assign R to that wavelength range. For example, as shown in FIG. 9, the horizontal line 901 (reference line) is drawn approximately halfway between the minimum point (about 980 nm) and the maximum point (about 1070 nm) in the average spectrum, and one luminance value is higher than the other luminance value. Determine two wavelength ranges that are always high. Then, for example, a wavelength range of 953 ± 10 nm to 1035 ± 10 nm and a wavelength range of 1036 ± 10 nm to 1123 ± 10 nm can be found as the RGB assigned wavelength range.

  Therefore, according to TYPE 1-a, in consideration of that it is desirable that the luminance value of the wavelength range to which the R value is assigned is higher than the luminance value of the wavelength range to which the G value and the B value are assigned, It is preferred to assign the G and B values from 953 ± 10 nm to 1035 ± 10 nm from 1036 ± 10 nm to 1123 ± 10 nm.

(Ii) TYPE 1-b
FIG. 10 is a diagram showing a wavelength range selection example (TYPE 1-b) according to the present embodiment. TYPE 1-b relates to an example of determining a wavelength range from, for example, the spectrum of porcine pancreas.

  In TYPE 1-b, in the spectrum of the pancreas, a region where the luminance value is always higher than other regions is determined, and R is assigned to the wavelength region. In this case, the condition that the wavelength range to which the R value is assigned is on the longer wavelength side than the wavelength range to which the G value and the B value are assigned is maintained. For example, as shown in FIG. 10, the horizontal line 1001 (reference line) is drawn approximately halfway between the minimum point (near 980 nm) and the maximum point (near 1070 nm) in the pancreas spectrum, and one luminance value is greater than the other luminance value. Also determine two wavelength ranges that are always high. Then, for example, a wavelength range of 940 ± 10 nm to 1030 ± 10 nm and a wavelength range of 1031 ± 10 nm to 1124 ± 10 nm can be found as assigned wavelength ranges of color tones (in this case, RGB).

  Therefore, according to TYPE1-b, considering that it is desirable that the luminance value in the wavelength region to which the R value is assigned is higher than the luminance value in the wavelength region to which the G value and the B value are assigned, for example, the R value is It is preferable to assign G value and B value from 1031 nm to 1124 nm to 940 nm to 1030 nm having different wavelength ranges from R value as a reference.

(Iii) TYPE 1-c
FIG. 11 is a diagram showing a wavelength range selection example (TYPE 1-c) according to the present embodiment. TYPE1-c relates to an example in which the wavelength range is determined from the spectrum of each part of the liver (eg, liver surface or blood vessel) of the pig. However, for convenience, only the average spectrum is shown in FIG. 11, and the spectrum of each part of the liver and the spectrum of the protein are not shown.

  In TYPE 1-c, the average spectrum (solid line graph) of each part of the liver is calculated, and on the short wavelength side of the average spectrum, a region where the luminance value is always higher than other regions is determined. Assign the color tone (in this case, R). In this case, the condition that the wavelength range to which the R value is assigned is on the longer wavelength side than the wavelength range to which the G value and the B value are assigned is maintained. For example, as shown in FIG. 11, a horizontal line 1101 (reference line) is drawn approximately halfway between the minimum point (about 970 nm) and the maximum point (about 1070 nm) in the average spectrum, and one luminance value is greater than the other luminance value. Determine two wavelength ranges that are always high. Then, for example, a wavelength range of 939 ± 10 nm to 1027 ± 10 nm and a wavelength range of 1028 ± 10 nm to 1129 ± 10 nm can be found as assigned wavelength ranges of RGB.

  Therefore, according to TYPE 1-c, in consideration of that it is desirable that the luminance value of the wavelength range to which the R value is allocated is higher than the luminance value of the wavelength range to which the G value and the B value are allocated, It is preferred to assign the G and B values from 938 ± 10 nm to 1027 nm from 1028 ± 10 nm to 1129 ± 10 nm.

(Iv) TYPE 1-d
FIG. 12 is a diagram showing a wavelength range selection example (TYPE 1-d) according to the present embodiment. TYPE1-d relates to the example which determines a wavelength range from the spectrum of each part (for example, fat tissue and a lymph node) of the mesentery of a pig, for example. However, for convenience, only the average spectrum is shown in FIG. 12, and the spectrum of adipose tissue and the spectrum of lymph nodes are not shown.

  In TYPE 1-d, an average spectrum (solid line graph) is calculated from the spectrum of fat tissue and the spectrum of lymph nodes, and in the average spectrum, a region where the luminance value is always higher than other regions is determined. Assign a red tone (in this case R) to. In this case, the condition that the wavelength range to which the R value is assigned is on the longer wavelength side than the wavelength range to which the G value and the B value are assigned is maintained. For example, as shown in FIG. 12, a horizontal line 1201 (reference line) is drawn approximately halfway between the minimum point (about 980 nm) and the maximum point (about 1080 nm) in the average spectrum, and one luminance value is higher than the other luminance value. Determine two wavelength ranges that are always high. Then, for example, a wavelength range of 953 ± 10 nm to 1048 ± 10 nm and a wavelength range of 1049 ± 10 nm to 1123 ± 10 nm can be found as an RGB assigned wavelength range.

  Therefore, according to TYPE 1-d, in consideration of that it is desirable that the luminance value of the wavelength range to which the R value is allocated is higher than the luminance value of the wavelength range to which the G value and the B value are allocated, It is preferable to assign G and B values from 953 nm to 1048 nm from 1049 nm to 1123 nm.

(V) TYPE1-e
FIG. 13 is a diagram showing a wavelength range selection example (TYPE 1-e) according to the present embodiment. TYPE1-e relates to an example in which a wavelength range is determined from an average spectrum of each part (eg, kidney surface, kidney cortex, kidney medulla, kidney cup, renal pelvis, etc.) of a pig kidney. However, for convenience, only the average spectrum is shown in FIG. 12, and the spectrum of each part of the kidney is not shown.

  In TYPE1-e, the average spectrum (solid line graph) of each part of the kidney is calculated, and in the average spectrum, a region where the luminance value is always higher than other regions is determined, and red tone (this In this case, R) is assigned. In this case, the condition that the wavelength range to which the R value is assigned is on the longer wavelength side than the wavelength range to which the G value and the B value are assigned is maintained. For example, as shown in FIG. 13, a horizontal line 1301 (reference line) is drawn approximately halfway between the minimum point (about 980 nm) and the maximum point (about 1070 nm) in the average spectrum, and one luminance value is greater than the other luminance value. Determine two wavelength ranges that are always high. Then, for example, a wavelength range of 934 ± 10 nm to 1035 ± 10 nm and a wavelength range of 1036 ± 10 nm to 1130 ± 10 nm can be found as assigned wavelength ranges of RGB.

  Therefore, in accordance with TYPE 1-e, in consideration of that it is desirable that the luminance value of the wavelength range to which the R value is assigned is higher than the luminance value of the wavelength range to which the G value and the B value are assigned, It is preferred to assign G and B values from 934 ± 10 nm to 1035 ± 10 nm from 1036 ± 10 nm to 1130 ± 10 nm.

(Vi) TYPE 1-f
FIG. 14 is a view showing a wavelength range selection example (TYPE 1-f) according to the present embodiment. TYPE 1-f relates to an example of determining a wavelength range from an average spectrum (dotted line graph) of a spectrum of water and a spectrum of oil (vegetable oil), for example.

  In TYPE 1-f, an average spectrum (dotted line graph) is calculated from the spectrum of water (solid line graph) and the spectrum of oil (broken line graph), and in the average spectrum, a region where the luminance value is always higher than other regions It is decided to assign a red tone (in this case, R) to that wavelength range. In this case, the condition that the wavelength range to which the R value is assigned is on the longer wavelength side than the wavelength range to which the G value and the B value are assigned is maintained. For example, as shown in FIG. 14, the horizontal line 1401 (reference line) is drawn approximately halfway between the minimum point (about 970 nm) and the maximum point (about 1070 nm) in the average spectrum, and one luminance value is higher than the other luminance value. Determine two wavelength ranges that are always high. Then, for example, a wavelength range of 951 ± 10 nm to 1040 ± 10 nm and a wavelength range of 1041 ± 10 nm to 1122 ± 10 nm can be found as an assigned wavelength range of RGB.

  Therefore, according to TYPE 1-f, in consideration of that it is desirable that the luminance value of the wavelength range to which the R value is assigned is higher than the luminance value of the wavelength range to which the G value and the B value are assigned, It is preferred to assign the G and B values from 951 ± 10 nm to 1040 ± 10 nm from 1041 ± 10 nm to 1122 ± 10 nm.

<Example of wavelength range selection: TYPE 2>
FIG. 15 is a diagram for explaining an example (TYPE 2) of wavelength selection according to the present embodiment. TYPE 2 is based on the idea that the wavelength on the long wavelength side (wavelength region longer than 1130 nm) is also used for wavelength selection. In particular, the wavelength range is determined from the spectrum of stomach adipose tissue. The application to other organs is considered based on the idea of TYPE2 which is a basic form, as described later. However, since the observation object from which the spectrum is acquired is different, there is a slight difference in the wavelength range between the TYPE2 (basic form) and the other (TYPE2-a to 2-f) wavelength range examples.

  For example, as in the case of TYPE 1, in order to reproduce the redness in a visible light image (eye image) as an infrared image, a region where the reflectance (brightness value) is large (a region where the absorption is small) ) Is assigned to a red color tone (in this case, R), and the green color tone (in this case, G) and the blue color tone (in this case, B) are more reflective than R (brightness value). It is desirable to allocate to a small area). The upper limit of the wavelength that can be used is 1370 nm to 1400 nm described above. Assuming this, if the horizontal lines 1501 (tangent of maximum point, reference line) and 1502 (tangent of minimum point, reference line) are drawn in the spectrum of fat tissue (solid line graph), the brightness in both fat tissue and the gastric outer wall It can be seen that the high value region is the region of 900 ± 10 nm to 1160 ± 10 nm. Therefore, it is understood that it is better to assign this region to the R value and to assign the G value and the B value to the region of 1161 ± 10 nm to 1370 ± 10 nm which is a region where the luminance value is lower than that region. In TYPE 2, for example, the wavelength range to which the R value is assigned is on the shorter wavelength side than the wavelength range to which the G value and the B value are assigned. Hereinafter, various wavelength range selection based on the same idea as TYPE 2 above will be exemplified.

(I) TYPE 2-a
FIG. 16 is a diagram showing a wavelength range selection example (TYPE 2-a) according to the present embodiment. TYPE2-a relates to the example which determines a wavelength range from the average spectrum of each part (fat tissue and stomach outer wall) of the stomach of a pig, for example.

  In TYPE2-a, the average spectrum of the spectrum of adipose tissue (solid line graph) and the spectrum of the outer stomach wall (broken line graph) is calculated, and in the average spectrum, a region where the luminance value is always higher than other regions is determined. , And assign R to that wavelength range. In this case, the condition that the wavelength range to which the R value is assigned is on the shorter wavelength side than the wavelength range to which the G value and the B value are assigned is maintained. For example, as shown in FIG. 9, horizontal lines 1601 (tangent of maximum point, reference line) and 1602 (tangent of minimum point, reference line) are drawn to the average spectrum, and the upper limit wavelength is 1370 ± 10 nm to 1400 ± 10 nm. , Determine two wavelength ranges in which one luminance value is always higher than the other luminance value. Then, for example, a wavelength range of 900 ± 10 nm to 1160 ± 10 nm and a wavelength range of 1061 ± 10 nm to 1360 ± 10 nm can be found as the RGB assigned wavelength range.

  Therefore, according to TYPE2-a, considering that it is desirable that the luminance value in the wavelength region to which the R value is assigned is higher than the luminance value in the wavelength region to which the G value and the B value are assigned, for example, the R value is It is preferable to assign G and B values from 1161 ± 10 nm to 13605 ± 10 nm from 900 ± 10 nm to 1160 ± 10 nm.

(Ii) TYPE2-b
FIG. 17 is a view showing a wavelength range selection example (TYPE 2-b) according to the present embodiment. TYPE2-b relates to the example which determines a wavelength range from the spectrum of a pig's pancreas, for example.

  In TYPE2-b, in the spectrum of the pancreas, a region where the luminance value is always higher than other regions is determined, and R is assigned to that wavelength region. In this case, the condition that the wavelength range to which the R value is assigned is on the shorter wavelength side than the wavelength range to which the G value and the B value are assigned is maintained. For example, as shown in FIG. 17, horizontal lines 1701 (tangent of maximum point, reference line) and 1702 (tangent of minimum point, reference line) are drawn on the spectrum of the pancreas, and the upper limit wavelength is 1370 nm to 1400 nm. Two wavelength regions are determined such that the luminance value is always higher than the other luminance value. Then, for example, a wavelength range of 900 ± 10 nm to 1167 ± 10 nm and a wavelength range of 1168 ± 10 nm to 1300 ± 10 nm can be found as the RGB assigned wavelength range.

  Therefore, according to TYPE 2-b, in consideration of that it is desirable that the luminance value of the wavelength range to which the R value is allocated is higher than the luminance value of the wavelength range to which the G value and the B value are allocated, It is preferable to assign G and B values from 1168 ± 10 nm to 1300 nm from 900 ± 10 nm to 1167 ± 10 nm.

(Iii) TYPE2-c
FIG. 18 is a diagram showing a wavelength range selection example (TYPE 2-c) according to the present embodiment. TYPE2-c relates to the example which determines a wavelength range from the spectrum of each part (a liver surface, a blood vessel, etc.) of a pig's liver, for example. However, for convenience, only the average spectrum is shown in FIG. 18, and the spectrum of each part of the liver is not shown.

  In TYPE2-c, a region where the luminance value is always higher than other regions in the average spectrum is determined, and R is assigned to the wavelength region. In this case, the condition that the wavelength range to which the R value is assigned is on the shorter wavelength side than the wavelength range to which the G value and the B value are assigned is maintained. For example, as shown in FIG. 18, horizontal lines 1801 (tangent of maximum point, reference line) and 1802 (tangent of minimum point, reference line) are drawn to the average spectrum, and the upper limit wavelength is, for example, 1370 ± 10 nm to 1400 nm ± 10. As, the two wavelength ranges in which one luminance value is always higher than the other luminance value are determined. Then, for example, a wavelength range of 900 ± 10 nm to 1166 ± 10 nm and a wavelength range of 1167 ± 10 nm to 1305 ± 10 nm can be found as an RGB assigned wavelength range.

  Therefore, according to TYPE2-c, in consideration of that it is desirable that the luminance value in the wavelength range to which the R value is assigned is higher than the luminance value in the wavelength range to which the G value and the B value are assigned, It is preferred to assign the G and B values from 1167 ± 10 nm to 1305 ± 10 nm from 900 ± 10 nm to 1166 ± 10 nm.

(Iv) TYPE2-d
FIG. 19 is a diagram showing a wavelength range selection example (TYPE 2-d) according to the present embodiment. TYPE2-d relates to the example which determines a wavelength range from the spectrum of each part (fat tissue and a lymph node) of the mesentery of a pig, for example. However, for convenience, only the average spectrum is shown in FIG. 19, and the spectrum of adipose tissue and the spectrum of lymph nodes are not shown.

  In TYPE2-d, a region where the luminance value is always higher than other regions in the average spectrum is determined, and R is assigned to that wavelength region. In this case, the condition that the wavelength range to which the R value is assigned is on the shorter wavelength side than the wavelength range to which the G value and the B value are assigned is maintained. For example, as shown in FIG. 19, horizontal lines 1901 (tangent of maximum point, reference line) and 1902 (tangent of minimum point, reference line) are drawn to the average spectrum, and the upper limit wavelength is, for example, 1370 ± 10 nm to 1400 ± 10 nm. As, the two wavelength ranges in which one luminance value is always higher than the other luminance value are determined. Then, for example, a wavelength range of 900 ± 10 nm to 1155 ± 10 nm and a wavelength range of 1156 ± 10 nm to 1375 ± 10 nm can be found as the RGB assigned wavelength range.

  Therefore, according to TYPE 2-d, in consideration of that it is desirable that the luminance value of the wavelength range to which the R value is assigned is higher than the luminance value of the wavelength range to which the G value and the B value are assigned, Preferably, the G and B values are assigned from 1156 ± 10 nm to 1375 ± 10 nm from 900 ± 10 nm to 1155 ± 10 nm.

(V) TYPE2-e
FIG. 20 is a diagram showing a wavelength range selection example (TYPE 2-e) according to the present embodiment. TYPE2-e relates to the example which determines a wavelength range from the average spectrum of each part (renal surface, renal cortex, renal medulla, renal cup, renal pelvis) of the kidney of a pig, for example. However, for convenience, only the average spectrum is shown in FIG. 12, and the spectrum of each part of the kidney is not shown.

  In TYPE2-e, in the average spectrum of each region of the kidney, a region where the luminance value is always higher than that of the other regions is determined, and R is assigned to that wavelength region. In this case, the condition that the wavelength range to which the R value is assigned is on the shorter wavelength side than the wavelength range to which the G value and the B value are assigned is maintained. For example, as shown in FIG. 20, horizontal lines 2001 (tangent of maximum point, reference line) and 2002 (tangent of minimum point, reference line) are drawn to the average spectrum, and the upper limit wavelength is, for example, 1370 ± 10 nm to 1400 ± 10 nm. As, the two wavelength ranges in which one luminance value is always higher than the other luminance value are determined. Then, the wavelength range of 900 ± 10 nm to 1161 ± 10 nm and the wavelength range of 1162 ± 10 nm to 133 ± 108 nm can be found as the RGB assigned wavelength range.

  Therefore, according to TYPE2-d, considering that it is desirable that the luminance value in the wavelength region to which the R value is assigned is higher than the luminance value in the wavelength region to which the G value and the B value are assigned, for example, the R value is It is preferred to assign G and B values from 1162 ± 10 nm to 1338 ± 10 nm from 900 ± 10 nm to 1161 ± 10 nm.

(Vi) TYPE2-f
FIG. 21 is a diagram showing a wavelength range selection example (TYPE2-f) according to the present embodiment. TYPE2-f relates to the example which determines a wavelength range from the average spectrum (dotted line graph) of the spectrum of water, and the spectrum of oil (vegetable oil), for example.

  In TYPE 2-f, the average spectrum (dotted line graph) is calculated from the spectrum of water (solid line graph) and the spectrum of oil (broken line graph), and in the average spectrum, the region where the luminance value is always higher than other regions Decide and assign R to that wavelength range. In this case, the condition that the wavelength range to which the R value is assigned is on the shorter wavelength side than the wavelength range to which the G value and the B value are assigned is maintained. For example, as shown in FIG. 21, horizontal lines 2101 (tangent of maximum point, reference line) and 2102 (tangent of minimum point, reference line) are drawn to the average spectrum, and the upper limit wavelength is, for example, 1370 ± 10 nm to 1400 ± 10 nm. As, the two wavelength ranges in which one luminance value is always higher than the other luminance value are determined. Then, for example, a wavelength range of 900 ± 10 nm to 1147 ± 10 nm and a wavelength range of 1148 ± 10 nm to 1393 ± 10 nm can be found as the RGB assigned wavelength range.

  Therefore, according to TYPE 1-f, in consideration of that it is desirable that the luminance value of the wavelength range to which the R value is assigned is higher than the luminance value of the wavelength range to which the G value and the B value are assigned, It is preferred to assign G and B values from 1148 ± 10 nm to 1393 ± 10 nm from 900 ± 10 nm to 1147 ± 10 nm.

<Example of pseudo color image>
Several examples of pseudocolor images generated with each wavelength falling within the wavelength range of TYPE 1 and TYPE 2 described above are shown. Below, Example 1 (FIG. 22) to Example 8 (FIG. 29) are shown, and all show examples of pseudo color images generated based on the method according to the present embodiment. 22 to 29 show visible light images (eye images) for comparison. Moreover, in Example 1 to Example 8, for example, actually generated pseudo color images are shown for each of the stomach & pancreas, kidney, mesentery & lymph node, and liver of the pig.

(I) Example 1: Example 1 of wavelength combination falling within the wavelength range of TYPE 1
FIG. 22 is a photograph showing Example 1 of a pseudo color image generated based on the method according to the present embodiment.
In the first embodiment, the wavelength used to acquire the image data to which the R value is assigned is 1070 nm, the wavelength used to acquire the image data to assign the G value is 1000 nm, and the image data to assign the B value is acquired. The wavelength used is 970 nm.
As can be seen from the pseudo color images of Example 1 (FIG. 22), it can be seen that the reproducibility as a whole is very good (the visual impression is close to the naked eye image).

(Ii) Example 2: Example 2 of wavelength combination falling within the wavelength range of TYPE 1
FIG. 23 is a photograph showing Example 3 of a pseudo color image generated based on the method according to the present embodiment.
In the second embodiment, the wavelength used to acquire the image data to which the R value is assigned is 1030 nm, the wavelength used to acquire the image data to assign the G value is 980 nm, and the image data to assign the B value is acquired. The wavelength used is 1000 nm.
Looking at each pseudo-color image of Example 2 (FIG. 23), the stomach and pancreas are darker than Example 1, and the mesentery and lymph nodes are slightly darker than the actual images. However, regarding other organs, it can be seen that the reproducibility is very good (the visual impression is close to the naked eye).

(Iii) Example 3: Example 3 of wavelength combination falling within the wavelength range of TYPE 1
FIG. 24 is a photograph showing Example 3 of a pseudo color image generated based on the method according to the present embodiment.
In the third embodiment, the wavelength used to acquire the image data to which the R value is assigned is 1120 nm, the wavelength used to acquire the image data to assign the G value is 980 nm, and the image data to assign the B value is acquired. The wavelength used is 980 nm.
Looking at each pseudo-color image of Example 3 (FIG. 24), as in Example 2, the stomach & pancreas is darker than Example 1, but the other organs are very reproducible. Good (the impression of the appearance is close to the naked eye image).

(Iv) Example 4: Example 1 of wavelength combination falling within the wavelength range of TYPE 2
FIG. 25 is a photograph showing Example 4 of a pseudo color image generated based on the method according to the present embodiment.
In the fourth embodiment, the wavelength used to acquire the image data to which the R value is assigned is 970 nm, the wavelength used to acquire the image data to assign the G value is 1160 nm, and the image data to assign the B value is acquired. The wavelength used is 1300 nm.
Looking at each pseudo color image of Example 4 (FIG. 25), the reddish image is stronger for the stomach & pancreas than Example 1, but the reproducibility is very good for the other organs (look The impression of is close to the naked eye image).

(V) Example 5: Example 2 of wavelength combination falling within the wavelength range of TYPE 2
FIG. 26 is a photograph showing Example 5 of a pseudo color image generated based on the method according to the present embodiment.
In the fifth embodiment, the wavelength used to acquire the image data to which the R value is assigned is 1070 nm, the wavelength used to acquire the image data to assign the G value is 1250 nm, and the image data to assign the B value is acquired. The wavelength used is 1300 nm.
Looking at each pseudo color image of Example 5 (FIG. 26), the image is reddish in general, but there is no particular problem in the reproducibility of each organ. In particular, with regard to the liver, it can be seen that the reproducibility is very good (the visual impression is close to the naked eye image).

(Vi) Example 6: Example 3 of wavelength combination falling within the wavelength range of TYPE 2
FIG. 27 is a photograph showing Example 6 of a pseudo color image generated based on the method according to the present embodiment.
In the sixth embodiment, the wavelength used to acquire the image data to which the R value is assigned is 1070 nm, the wavelength used to acquire the image data to assign the G value is 1300 nm, and the image data to assign the B value is acquired. The wavelength used is 1200 nm.
Looking at each pseudo color image of Example 6 (FIG. 27), the image is generally reddish, and the adipose tissue has a yellowish color stronger than those of the other examples.

(Vii) Example 7: Example 4 of wavelength combination falling within the wavelength range of TYPE 2
FIG. 28 is a photograph showing Example 7 of a pseudo color image generated based on the method according to the present embodiment.
In the seventh embodiment, the wavelength used to acquire the image data to which the R value is assigned is 1070 nm, the wavelength used to acquire the image data to assign the G value is 1160 nm, and the image data to assign the B value is acquired. The wavelength used is 1300 nm.
Looking at each pseudo color image in Example 7 (FIG. 28), the image is generally reddish, but the reproducibility of each organ is very good (the impression of the appearance is close to that of the naked eye) . In particular, regarding the liver, an image very close to the color of the visible light image is obtained.

(Viii) Example 8: Example 5 of wavelength combination falling within the wavelength range of TYPE 2
FIG. 29 is a photograph showing Example 8 of a pseudo color image generated based on the method according to the present embodiment.
In the eighth embodiment, the wavelength used to acquire the image data to which the R value is assigned is 1160 nm, the wavelength used to acquire the image data to assign the G value is 1330 nm, and the image data to assign the B value is acquired. The wavelength used is 1370 nm.
When looking at each pseudo color image of Example 8 (FIG. 29), it is an image in which the tea taste is strong as a whole.

(Ix) Summary of Examples The above examples will be compared. However, what is described below is an evaluation based on the subjectivity of a specific person (e.g., the inventor), and since preference and sensibility differ depending on the person, it is not a fixed evaluation but an evaluation as a mere example. It should be noted that.

  Example 1 (Example 1 of TYPE 1) is better than the other Examples (except Example 5 (Example 2 of TYPE 2) and Example 7 (Example 4 of TYPE 2)) when viewing a pseudo color image of the liver. Since it can be said that it is bright and closest to the visible light image (macroscopic image), it is considered that Example 1 is the best result as a whole. When the pseudo color images are compared between the examples only for the liver, Example 5 (Type 2 example 2) and Example 7 (TYPE 2 example 4) are the best results. In addition, with regard to the mesentery & lymph nodes, all examples are taken, and good results are obtained. Because the mesentery has many fat tissues, it tends to become white. Furthermore, if it compares among TYPE2, it can be judged that Example 5 (Example 2 of TYPE 2) and Example 7 (Example 4 of TYPE 2) are generally good regarding the stomach. However, when it is desired to emphasize the fat tissue in yellow, Example 6 (Example 3 of TYPE 2) can be said to be a good result.

  As described above, there is no combination of wavelengths perfectly adapted to generate pseudo color images for all organs, and the wavelength depends on the type of organ included in the surgical field or the type of organ targeted for surgery. By changing the combination of the above, it is possible to generate a pseudo color image close to the naked eye image and without a sense of incongruity. It is also possible to appropriately adjust the color of the pseudo color image by changing the combination of wavelengths according to the preference of the operator (such as a doctor).

<Concavity and convexity detection processing: Application example 1 of pseudo color image>
Heretofore, for example, in an examination such as an endoscope, a method of spraying indigo carmine has been used in order to visualize a lesion (tumor) produced in the mucous membrane of the esophagus or the stomach. This is because indigo carmine, which is a pigment, enters the depressions in the inner wall to emphasize unevenness and to make it easy to identify a lesion.

However, when indigo carmine is used, side effects such as vomiting occur and the mucous membrane is colored.
Therefore, the present embodiment further generates an asperity-emphasized image that emphasizes asperities of a lesion, without using a colorant such as indigo carmine, and superimposes the asperity-weighted image on the above-mentioned pseudo color image or single wavelength image. Disclose what to do. The outline of the concavo-convex shape detection processing and the superimposition processing on a pseudo color image or the like will be described below.

FIG. 30 shows a process of detecting a concavo-convex shape on an organ surface (an example) according to this embodiment and generating a concavo-convex weighted image for emphasizing the concavo-convex shape, and converting the generated concavo-convex weighted image into a pseudo color image or a single wavelength image. It is a photograph which shows the outline of processing to superimpose.
In the present embodiment, when detecting the uneven shape, for example, zero-order light (regular reflection light) is detected from light reflected from the sample. Usually, specular reflection light is suppressed as much as possible to generate an image using only scattered light. However, specular reflection light is often generated at a portion with unevenness, and this property is rather positive in this embodiment. Are used in In other words, it can be said that there are irregularities in the sample where regular reflection is generated, and there may be a lesion there.

When detecting the uneven | corrugated shape of a sample surface, first, the infrared light of arbitrary wavelengths is irradiated to a sample surface using the imaging system 1, for example. In addition, since regular reflection light does not have wavelength dependency, if the incident angle with respect to a polarizing plate is the same, it does not depend on the wavelength of infrared light, but the result is the same. Then, all the reflected light from the sample is transmitted by using a pair of polarizing plates having the same polarization angle (P or S, whichever is preferable but an S polarizing plate having a large reflectance is better). The image at this time (all-light transmission image data) is an image shown in FIG. On the other hand, the reflected light from the sample is obtained using a crossed Nicol polarizing plate (polarizing plate pair having a polarization angle reversal of 90 degrees: P or S, whichever is better but an S polarizing plate with a larger reflectance is better). And specularly reflected light is removed, and multiple scattered light (polarization angle is changed) is transmitted. The image at this time is an image (multiple reflection light transmission image data) shown in FIG. Then, regular reflection light image data (see FIG. 30C) is generated by subtracting the multiple reflection light transmission data from the total light transmission image data. Although the regular reflection light image data is calculated using the difference here, the regular reflection light image data may be acquired by dividing the total light transmission image data by the multiple reflection light transmission image data. In FIG. 30, regular reflection light image data (concave / convex shape) is expressed in green, but the operator (such as a doctor) appropriately changes the color of the concavo-convex shape by specifying the color using the input device 40. It may be possible to
Next, for example, in the imaging system 1, regular reflection light image data (concave / convex enhancement image data) is superimposed on the pseudo color image of the sample generated by the above-described method to generate a synthetic pseudo color image (FIG. 30 (FIG. d) see). Further, for example, in the imaging system 1, specular reflection light image data (concave and convexity enhancement image data) is superimposed on a single wavelength image of the sample obtained by irradiating the sample with infrared light of an arbitrary single wavelength, and synthetic pseudo color An image is generated (see FIG. 30 (e)).

  As described above, by superimposing the regular reflection light image (concave and convex image) of the sample on the same sample pseudo color image, the unevenness on the sample surface can be emphasized and displayed. For example, the unevenness of the lesion generated on the inner surface of the digestive tract is highlighted on a pseudo color image during endoscopic examination, or the unevenness of the fibrotic portion (hepatic cirrhosis) of the liver during the operation of the liver is displayed. It becomes possible to emphasize and display on the pseudo color image. In the present embodiment, the pseudo color image and the enhanced image (concave / convex enhanced image) of the sample are generated using only the image data acquired by the infrared light camera, so there is no need to separately prepare a visible light camera. Also, the system cost can be reduced because the correction of the misalignment is not required.

<Configuration example of GUI>
FIG. 31 is a view showing a configuration example of a GUI (Graphical User Interface) used in the imaging system 1 of the present embodiment.

  The GUI 3100 is, for example, an individual wavelength setting unit 3101 that selects wavelength setting individually for each light source, and a wavelength band of infrared light to be irradiated (however, in this case, spectral data is A continuous wavelength setting unit 3102 for selecting the setting to be acquired, an RGB assignment wavelength setting unit 3103 for setting the wavelength of infrared light, acquiring spectral data by irradiating a sample and acquiring spectral data; Concave-convex shape detection setting unit 3104 for setting presence / absence of execution of processing to detect concavo-convex shape on the surface of sample, Individual wavelength for setting the wavelength of infrared light emitted from k (k = 1 to n) light sources Setting areas 3105 to 3109, a wavelength band setting area 3110 for setting the wavelength band of the infrared light to be irradiated, and an R value assigned wavelength for setting a wavelength for assigning an R value A constant area 3111, a G value assignment wavelength setting area 3112 for setting a wavelength to which a G value is to be assigned, a B value assignment wavelength setting area 3113 for setting a wavelength to which a B value is to be assigned, A superimposed image selection unit 3114 that selects an image target on which an image is to be superimposed (in this embodiment, for example, a pseudo color image or a single wavelength image can be selected as a target to be superimposed), and a save button 3115 for saving the setting result. And an end button 3116 for closing the wavelength setting GUI. In addition to these GUI components, a parameter adjustment unit capable of adjusting “brightness / contrast” or “gamma value” may be displayed on the display device as a GUI.

  The individual wavelength setting unit 3101 and the continuous wavelength setting unit 3102 are alternatively used. For example, one of the setting units can be selected by a radio button. For example, after the operator (such as a doctor) operates the input device 40 and selects the individual wavelength setting unit 3101, the respective wavelengths of the k infrared light sources 21 to 22 included in the infrared light source unit 20 are input be able to. Alternatively, for example, when an operator (such as a doctor) operates the input device 40 and selects the individual wavelength setting unit 3101, the control unit 101 may automatically input the wavelength of each infrared light source. Further, for example, an operator (such as a doctor) selects the continuous wavelength setting unit 3102 by operating the input device 40, and then generates an enhanced image in the enhanced image generation unit 1012 out of the light emitted from the infrared light source unit 20. The value of the wavelength band of infrared light to be used can be input. Alternatively, for example, when the operator (such as a doctor) operates the input device 40 and selects the continuous wavelength setting unit 3102, the control unit 101 automatically inputs the value of the wavelength band of infrared light. Good.

  For example, when the operator (such as a doctor) selects either the individual wavelength setting unit 3101 or the continuous wavelength setting unit 3102 using the input device 40, each wavelength is input to each of the areas 3111 to 3113 of the RGB allocation wavelength setting unit 3103 You will be able to For example, when the individual wavelength setting unit 3101 is selected, the operator (such as a doctor) inputs the wavelength of any one of the light sources 1 to n into the assigned wavelength setting areas 3111 to 3113. On the other hand, for example, when the continuous wavelength setting unit 3102 is selected, the operator inputs each wavelength to each of the assigned wavelength setting regions 3111 to 3113 within the set value range of the wavelength band 3110. Also, for example, in each of the RGB value assignment wavelength setting areas 3111 to 3113 of the RGB assignment wavelength setting unit 3103, a combination of wavelengths stored in the storage unit 102 (for example, a combination of wavelengths of TYPE 1 and TYPE 2 ) May be input automatically.

  The uneven shape detection setting unit 3104 can be arbitrarily selected by an operator (such as a doctor). For example, if it is desired to execute the concavo-convex shape detection processing, the operator (such as a doctor) makes the radio button of the concavo-convex shape detection setting unit 3104 selected (input a check). Then, the superimposed image selection unit 3114 becomes effective, and, for example, it is possible to select either superimposed on a pseudo color image or superimposed on a single wavelength image.

  For example, the control unit 101 reads the wavelength value of each light source set by the GUI 3100 based on the light irradiation control unit 1011 (program), and applies a voltage to a drive unit (not shown) of the infrared light source unit 20. And the wavelength of the light emitted (radiated) by each light source is transmitted to the drive unit. The drive unit applies a voltage to the infrared light sources 21 and 22 under the control of the control unit 101 to emit (emit) light. Further, the control unit 101 transmits, for example, the timing at which the infrared light source 22 emits (radiates) light of each wavelength and the light emission (emission) time to a drive unit (not shown) of the infrared light source unit 20. The infrared light source unit 20 is controlled so that light of a plurality of wavelengths is periodically emitted (radiated) from the infrared light source 22.

  The control unit 101 samples reflected light (including scattered light) obtained by irradiating the sample with infrared light having a wavelength set with the R value assigned wavelength, and infrared light with a wavelength set with the G value assigned wavelength. To obtain the reflected light (including the scattered light) obtained by irradiating the sample and the reflected light (including the scattered light) obtained by irradiating the sample with the infrared light of the wavelength set at the B-value assignment wavelength. The second imaging device 32 (for example, an infrared light camera or a hyperspectral camera) is controlled. Then, as described above, the image generation unit 1013 assigns RGB to spectral data corresponding to each acquired wavelength, for example, generates a pseudo color image, and stores the pseudo color image in the storage unit 102. In addition, the image generation unit 1013 stores, for example, each of the acquired spectrum data corresponding to each wavelength as a single wavelength image in the storage unit 102.

  Note that, in the present embodiment, the imaging system 1 is provided with a plurality of light sources for emitting a plurality of infrared light of a single wavelength, switched to irradiate the sample, but emits, for example, infrared light of a specific wavelength band The infrared light of the bandwidth may be emitted from the light source, and the filter may selectively extract infrared light of a desired wavelength and irradiate the sample. Alternatively, a filter (a band pass filter or the like) for transmitting infrared light of a specific wavelength band may be disposed in the light path of the photographing optical system. The GUI 3100 may display the individual wavelength setting unit 3101 and the continuous wavelength setting unit 3102 at the same time, or may display the individual wavelength setting unit 3101 and the continuous wavelength setting unit 3102 separately based on the initial setting. It may be.

<Generation of Superimposed Image of Highlighted Image and Pseudo-Color Image: Application Example 2 of Pseudo-Color Image>
For example, when a body fluid flows out or adheres to the surface of a living body (a tissue surface, an organ surface, or a cross section) during a medical procedure such as surgery or treatment, the distribution or source of the body fluid (or attachment source) on the surface of the living body There is a request that you want to determine the However, for example, when a body fluid that is difficult to distinguish with visible light (such as the naked eye) flows out or adheres, it may be difficult to determine the distribution of the body fluid or the source of the body fluid. Therefore, in the present embodiment, as an application example of the pseudo color image, for example, the enhanced image of the body fluid existing on the sample surface is generated, and the distribution of the body fluid, the outflow source, etc. is generated by superimposing it on the pseudo color image of the organ. Also disclosed is a technology for making it easy to determine the Hereinafter, an outline of a process for generating a superimposed image of an emphasized image and a pseudo color image as an application example of the pseudo color image will be described.

(I) FIG. 32 shows a highlight image of liquid leakage (as an example bile) of the surface of a sample (as an example, a liver to be observed) according to the present embodiment, a pseudo color image of the sample, and a superimposed image thereof It is a photograph shown. First, in the case of generating an enhanced image of liquid leakage (bile) present on the surface of a sample (liver), infrared light of an arbitrary wavelength is irradiated onto the sample surface using, for example, the imaging system 1. In the present embodiment, the imaging system 1 includes, for example, infrared light with a wavelength of 1070 nm and infrared light with a wavelength of 1450 nm before dropping bile on the surface of the liver and after dropping bile on the surface of the liver, respectively. The sample surface is irradiated at different timings to obtain an infrared light image corresponding to infrared light of each wavelength. Then, for each pixel of the infrared light image corresponding to each wavelength on the surface of the liver after dropping bile, the imaging system 1 sets the pixel value of the infrared light image of 1070 nm to the pixel value of the infrared light image of 1450 nm. By dividing, the value of Post @ 1070nm / 1450nm is calculated. In addition, the imaging system 1 sets the pixel value of the infrared light image of 1070 nm to the pixel value of the infrared light image of 1450 nm for each pixel of the infrared light image corresponding to each wavelength of the liver surface before dropping bile. The value of Pre @ 1070nm / 1450nm is calculated by dividing. Then, the imaging system 1 calculates Post @ 1070 nm / 1450 nm-Pre @ 1070 nm / 1450 nm for each pixel, and takes each value as the value of each pixel of the enhanced image. The emphasized image obtained in this way is shown in FIG. On the other hand, a pseudo color image of the generated sample (liver) generated using the method described above (see FIG. 4) is shown in FIG. In the pseudo color image, for example, a red color tone (R in this case) is assigned to an infrared light image obtained by irradiating an infrared light of 1070 nm, and an infrared light of 1160 nm with reference to the red color tone. Is assigned a green color tone (in this case G) to the infrared light image obtained by irradiating the light, and a blue color tone (in this case B) in the infrared light image obtained by irradiating the infrared light of 1300 nm. Is an image generated by assigning Then, the imaging system 1 superimposes the liquid leakage emphasis image (FIG. 32 (a)) on the pseudo color image (FIG. 32 (b)) to generate a superimposed image (see FIG. 32 (c)). In this case, the highlighted image of the leak is generated by calculating Post @ 1070 nm / 1450 nm-Pre @ 1070 nm / 1450 nm, but as described later (see FIG. 33), the pixel of the 1070 nm infrared light image The enhanced image may be generated by dividing the value by the pixel value of the 1450 nm infrared light image.

(Ii) FIG. 33 is a photograph showing a highlight image of a lymph node in a sample (for example, the mesentery + lymph node to be observed as an example), a pseudo color image of the sample, and a superimposed image thereof according to this embodiment. It is. First, when generating an enhanced image of a lymph node (containing a large amount of water) contained in a sample (mesenteric membrane), infrared light of an arbitrary wavelength is irradiated to the sample surface using, for example, the imaging system 1. In the present embodiment, for example, the imaging system 1 irradiates the sample with infrared light with a wavelength of 1070 nm and infrared light with a wavelength of 1450 nm at different timings, and applies infrared light to each wavelength to the sample. Acquire the corresponding infrared light image. Then, the imaging system 1 generates an enhanced image by dividing the pixel value of the 1070 nm infrared light image by the pixel value of the 1450 nm infrared light image for each pixel of the infrared light image corresponding to each wavelength. To do. The enhanced image obtained in this way is shown in FIG. On the other hand, a pseudo color image of the generated sample (mesenteric + lymph node) generated using the above-mentioned method (see FIG. 4) is shown in FIG. 33 (b). The pseudo color image is, for example, assigned R to an infrared light image obtained by irradiating 1070 nm infrared light, and assigned G to an infrared light image obtained by irradiating infrared light of 1160 nm, It is an image generated by assigning B to an infrared light image obtained by irradiating infrared light of 1300 nm. Then, the imaging system 1 superimposes the enhanced image (FIG. 33 (a)) on the pseudo color image (FIG. 33 (b)) to generate a superimposed image (see FIG. 33 (c)).

(Iii) As described above, the superimposed image is generated using the pseudo color image instead of the visible light image for the sample image to be superimposed on the emphasized image. Since the pseudo color image and the enhanced image are generated using the same infrared light camera, the imaging axis (optical axis) at the time of image acquisition is different from the case where the enhancement image is superimposed on the visible light image. ) Can be the same, it is not necessary to correct the misalignment of the acquired image. Further, it becomes unnecessary to prepare a visible light camera separately from the infrared light camera. This can reduce the cost of the system.

<Other Embodiments>
(I) When a sample is imaged by an infrared camera, a characteristic image (for example, an enhanced image for identifying a lesion) can be generated based on the acquired image data. However, such a characteristic image is composed of spectrum data (luminance value), and is not a color image such as a visible light image. On the other hand, for example, when displaying a color image on a monitor, conventionally, a visible light camera is provided separately from the infrared light camera, and a characteristic image generated from data acquired by the infrared light camera is acquired by the visible light camera The image was superimposed on the visible color image. However, in this case, alignment is necessary when superimposing a visible light color image and a characteristic image, or to eliminate the need for alignment, the imaging axes of the visible light camera and the infrared light camera (light If an optical system for aligning the axes) must be provided, further improvements are required for the imaging system. Such a device leads to an increase in the manufacturing cost of the imaging system.

  Therefore, in the present embodiment, by using the image data acquired by the infrared light camera, a characteristic image of the sample and a pseudo color image of the sample are generated and superimposed to generate a composite image, The disadvantages of providing a visible light camera separately from the infrared light camera are eliminated.

  In each of the above-described embodiments, all processing from infrared light irradiation processing to superimposed image display processing is performed in one imaging system (surgery support system) 1. For example, at least one of the functions of the imaging system 1 is performed. The unit may be configured to be executed by another device (imaging device). For example, a plurality of infrared light beams having different wavelength values are irradiated to the sample, and different color information is allocated to each of the plurality of measurement results and a light detection unit that acquires a plurality of measurement results corresponding to the wavelength values. The controller configured to generate a pseudo color image of a sample can be configured as an imaging device. By adopting such a configuration, a color image of a sample (for example, a portion exposed as a surgical field in a living body) can be artificially generated from spectral data, and a characteristic image of the sample (for example, It becomes possible to superimpose and display an enhanced image generated from spectral data on the screen of the display device. Therefore, the imaging apparatus can be configured at low cost, and an operator (for example, a doctor or the like) can perform an operation as if a visible color image is confirmed by a pseudo color image. become.

  In the present embodiment, the light detection unit (imaging unit) samples infrared light of a plurality of types of wavelengths (a plurality of types of single-wavelength infrared light having different wavelengths or an infrared light having a predetermined wavelength band). To obtain a plurality of measurement results (a plurality of detection results) obtained by irradiating the light (for example, in the case of the former, the light detection unit is constituted by a single wavelength infrared light camera; The control unit may be configured to generate a pseudo color image from a plurality of measurement results (eg, luminance values for each pixel). For example, in the case of using a hyperspectral camera, a pseudo color image is generated using spectral data corresponding to a plurality of wavelength values (for example, three wavelength values) in the wavelength band based on a plurality of measurement results. Then, in the case of generating a plurality of characteristic images (emphasized images of samples), the control unit superimposes and displays the plurality of emphasized images on the pseudo color image while switching them, or averages the plurality of emphasized images The captured image may be displayed superimposed on the pseudo color image. By doing this, it is possible to generate a pseudo color image of the sample and to generate a composite image in which the enhanced image is superimposed on the pseudo color image without alignment of the image, so the image composition processing is easy. In addition, the configuration of the optical system in the imaging apparatus can be simplified. Therefore, the manufacturing cost of the imaging device can be reduced.

  In the present embodiment, when generating a pseudo color image, the control unit samples a plurality of infrared light (for example, three or more infrared light and infrared light of first to third wavelengths) having different wavelengths. Different color information (for example, each color information in the RGB color space) is assigned to a plurality of measurement results obtained by irradiating the light source. At this time, the control unit is a relative unit among a plurality of measurement results (for example, three or more luminance values) in an observation target of the sample (for example, a specific portion of the sample (observation target) or a pixel corresponding to the portion). Color information (for example, red color information (for example, an area with a large reflectance (for example, a pixel), a luminance value by infrared light of the first wavelength) having the highest luminance value (light intensity) , R value), and color information (for example, G value or B value) other than red system is allocated to measurement results other than the first measurement result (for example, second measurement result, third measurement result), A pseudo color image of a sample is generated by synthesizing the result (for example, image, luminance value data) assigned to each position (observation target) for each position (for example, each pixel). By assigning the R value to the measurement result showing the highest luminance, natural color can be obtained, and a pseudo color image very close to the visible light image (a macroscopic image) of the sample can be generated. Become. In the color tone assignment as described above, the control unit can adjust the assigned R value (or G value or B value) within the range of the red color tone.

For example, the combination of the wavelengths of the infrared light irradiated to the sample is stored in advance in the storage unit, and the corresponding combination is appropriately read from the storage unit and used. The control unit is configured to use infrared light of the corresponding wavelength from the light source (for example, when using an imaging device (infrared light camera) for detecting infrared light of a single wavelength) or the corresponding wavelength based on the combination of the wavelengths. The light source is controlled so that infrared light (for example, when using a hyperspectral camera) having a wavelength band including. Note that the combination of wavelengths differs depending on, for example, the type of sample (the type of target organ for which a pseudo color image is to be generated).
When color information in RGB space is used as color information when generating a pseudo color image, the control unit controls the luminance value of the pixel at the corresponding position in each of the plurality of measurement results corresponding to the wavelength value. , RGB mixing ratio is determined, and the color of each pixel is determined. By doing so, the color of each pixel in the pseudo color image can be brought close to the color of each pixel in the naked eye image.

  In the present embodiment, from TYPE 1 above, the wavelength value for acquiring the measurement result to which the R value is allocated is selected from 1031 nm to 1130 nm, and the wavelength value for acquiring the measurement result to which the G value and B value are allocated. Is preferably selected from 940 nm to 1048 nm. However, it is a condition that the wavelength for obtaining the measurement result to which the R value is assigned is longer than the wavelength for obtaining the measurement result to which the G value and the B value are assigned. On the other hand, from TYPE 2, the wavelength value for acquiring the measurement result to which R value is assigned is selected from 900 nm to 1167 nm, and the wavelength value for acquiring the measurement result to assign G value and B value is 1148 nm to 1393 nm. It turns out that it is preferable to select from the inside. However, it is a condition that the wavelength for acquiring the measurement result to which the R value is assigned is shorter than the wavelength for acquiring the measurement result to which the G value and the B value are assigned.

  In this embodiment, the sample is irradiated with infrared light, and the combination of the wavelengths of infrared light detected by the imaging unit among the light (reflected light and scattered light) emitted from the sample is read out from the storage unit. However, the present invention is not limited to this form. For example, the patient's body (surgery site) may be incised and a combination of wavelengths matching the characteristics may be selected. For example, a region other than fat tissue or fat tissue is imaged with a hyperspectral camera or the like to acquire those characteristics. Then, the control unit may execute the wavelength determination operation described in TYPE 1 or TYPE 2 described above. In addition, when the patient's body is incised, for example, an operator (such as a doctor) visually checks the color of the patient's fat tissue, and a combination which can express a color close to the macroscopic image with a simulated impression. It may be made to be able to select and use among combinations prepared in advance.

(Iii) The functions of the present embodiment can also be realized by program code of software. In this case, a storage medium storing the program code is provided to the system or apparatus, and a computer (or CPU or MPU) of the system or apparatus reads the program code stored in the storage medium. In this case, the program code itself read out from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium storing the same constitute the present embodiment. As a storage medium for supplying such program code, for example, a flexible disk, CD-ROM, DVD-ROM, hard disk, optical disk, magneto-optical disk, CD-R, magnetic tape, nonvolatile memory card, ROM Etc. are used.

  Also, based on instructions of the program code, an operating system (OS) operating on the computer performs a part or all of the actual processing, and the processing of the above-described embodiment is realized by the processing. Also good. Furthermore, after the program code read out from the storage medium is written in the memory on the computer, the CPU of the computer or the like performs part or all of the actual processing based on the instruction of the program code, and the processing The functions of the embodiments described above may be realized by

  Furthermore, by distributing the program code of the software that realizes the functions of the embodiment via a network, it can be used as a storage means such as a hard disk or a memory of a system or device or a storage medium such as a CD-RW or CD-R. It may be stored, and the computer (or CPU or MPU) of the system or apparatus may read out and execute the program code stored in the storage means or the storage medium at the time of use.

  The processes and techniques described herein are not inherently related to any particular device, and can be implemented by any suitable combination of components. Furthermore, various types of general purpose devices may be used in accordance with the methods described herein. It may be beneficial to build a dedicated device to perform the method steps described herein. In addition, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components in different embodiments may be combined as appropriate.

  Other implementations of the present invention will become apparent to those skilled in the art from consideration of the specification and embodiments of the invention disclosed herein. Various aspects and / or components of the described embodiments can be used alone or in any combination.

Reference Signs List 1 imaging system 10 control device 20 infrared light source unit 21 infrared light source 22 infrared light source 30 imaging unit 31 first imaging device 32 second imaging device 40 input device 50 display device 51 monitor 1
52 Monitor 2
60 Operating shadow lamp 80 living body 101 control unit 1011 light irradiation control unit 1012 enhanced image generation unit 1013 image generation unit 1014 image correction unit 102 storage unit 1021 RGB allocation table

Claims (31)

A light detection unit that acquires a plurality of detection results corresponding to the wavelength by irradiating the sample with a plurality of infrared light having different wavelengths;
A controller configured to assign different color tones to each of the plurality of detection results based on the spectrum of the observation target of the sample obtained from the plurality of detection results, and generate a pseudo color image of the sample;
The image pickup apparatus, wherein the control unit assigns a red color tone to the plurality of detection results based on the spectrum, and assigns a color tone other than red to the plurality of detection results based on the red color tone. In claim 1,
The control unit is configured to use a luminance value corresponding to a second wavelength shorter than the first wavelength with reference to a first wavelength corresponding to the luminance value to which the red color tone is assigned in the spectrum. An imaging device that assigns to a blue color tone or a green color tone among the color tones. In claim 1 or 2,
The control unit uses a luminance value corresponding to a second wavelength of a wavelength longer than the first wavelength on the basis of a first wavelength corresponding to a luminance value to which the red color tone is assigned in the spectrum, except for the red color. An imaging device that assigns to a blue color tone or a green color tone among the color tones. In any one of claims 1 to 3,
The imaging device, wherein the observation target includes a stomach, a kidney, a mesentery, a liver, or a pancreas. In any one of claims 1 to 4,
An imaging device, wherein a wavelength of infrared light irradiated when acquiring the detection result to which the red color tone is assigned is selected from a wavelength range which varies depending on a type of the observation object. In claim 5,
The wavelength of infrared light irradiated when acquiring the detection result to which the red color tone is assigned is (i) when the observation target is the stomach, 1030 nm to 1130 nm, 900 nm to 1160 nm, (ii) the observation target is In the case of pancreas, 1030 nm to 1130 nm, or 890 nm to 1170 nm, (iii) when the observation target is liver, 1020 nm to 1130 nm, or 890 nm to 1170 nm, (iv) when the observation target is mesentery, 1040 nm to 1130 nm, Alternatively, an imaging device selected from 890 nm to 1160 nm, and (v) 1030 nm to 1140 nm, or 890 nm to 1170 nm when the observation target is a kidney. In any one of claims 1 to 6,
The control unit allocates color tones to the plurality of detection results such that the luminance value of the detection result to which the red color tone is allocated is higher than the luminance value of the detection result to which the red color tone is allocated. An imaging device that generates the pseudo color image. In any one of claims 1 to 7,
The control unit generates an enhanced image of the sample based on the plurality of detection results, and generates a composite image of the sample by superimposing the enhanced image on the pseudo color image. In any one of claims 1 to 8,
The imaging device, wherein the control unit generates and outputs a control signal for causing the display device to display the pseudo color image. In any one of claims 1 to 9,
The control unit is a combination of wavelengths for irradiating the sample with infrared light to obtain the plurality of measurement results, and reads out a combination of wavelengths different depending on the type of the sample from the storage unit, An imaging apparatus which emits infrared light of a plurality of wavelengths included in a combination and reads out a combination of the wavelengths corresponding to the type of the sample from the storage unit in response to the type of the input sample. In claim 10,
The light detection unit is an image pickup apparatus including an image pickup device having at least higher detection sensitivity with respect to light in an infrared light region than light in a visible light region. In any one of claims 1 to 11,
The color tone is composed of colors in RGB space,
The control unit determines the color mixing ratio of the RGB space according to the luminance value of the pixel at the corresponding position in each of the plurality of detection results corresponding to the wavelength, and determines the color of each pixel. An imaging device that generates the pseudo color image; In any one of claims 1 to 12,
Among the wavelength values of the infrared light, a wavelength for acquiring the detection result for assigning the red color tone is selected from 1030 nm to 1140 nm, and a wavelength for acquiring the detection result for assigning the color tone other than red is Selected from 930 nm to 1050 nm,
An imaging device, wherein a wavelength for acquiring a detection result to which the red color tone is allocated is longer than a wavelength for acquiring a detection result to which the color tone other than the red color is allocated. In any one of claims 1 to 13,
Of the wavelength values of the infrared light, a wavelength for acquiring a detection result for assigning the red color tone is selected from 890 nm to 1170 nm, and a wavelength for acquiring the detection result for assigning a color tone other than red is Selected from 1140 nm to 1400 nm,
An imaging device, wherein a wavelength for acquiring a detection result to which the red color tone is allocated is shorter than a wavelength for acquiring a detection result to which the color tone other than the red color is allocated. In any one of claims 1 to 14,
The light detection unit includes a hyperspectral camera that acquires a spectral value of an imaging target, and obtains a detection result obtained by irradiating the sample with infrared light having a predetermined wavelength band by the hyperspectral camera,
The control unit generates an enhanced image of the sample using at least a part of the detection result in the predetermined wavelength band, and detects the wavelength corresponding to the wavelength to which the color tone is assigned among the detection results in the predetermined wavelength band. An imaging apparatus that generates the pseudo color image using a result. An imaging device according to any one of claims 1 to 15,
A display device for displaying at least the pseudo color image;
An imaging system comprising: Obtaining a plurality of detection results corresponding to the wavelengths by irradiating the sample with a plurality of infrared lights having different wavelengths;
The control unit assigns red tones of the plurality of detection results based on the spectrum of the observation target of the sample obtained from the plurality of detection results, and the plurality of tones other than red with the red tone as a reference Generating a pseudo color image of the sample by assigning to the detection result of
Imaging methods, including: In claim 17,
The control unit uses a luminance value corresponding to a second wavelength shorter than the first wavelength with reference to a first wavelength corresponding to a luminance value to which the red color tone is assigned in the spectrum, and uses a luminance value other than red. An imaging method which is assigned to a blue tone or a green tone among the tones of. In claim 17 or 18,
The control unit uses a luminance value corresponding to a second wavelength of a wavelength longer than the first wavelength on the basis of a first wavelength corresponding to a luminance value to which the red color tone is assigned in the spectrum, except for the red color. An imaging method which is assigned to a blue tone or a green tone among the tones of. In any one of claims 17-19,
The imaging method, wherein the observation target includes a stomach, a kidney, a mesentery, a liver, or a pancreas. In any one of claims 17 to 20,
An imaging method, wherein a wavelength of infrared light emitted when acquiring the detection result to which the red color tone is assigned is selected from a wavelength range that varies depending on the type of the observation target. In claim 21,
The wavelength of infrared light irradiated when acquiring the detection result to which the red color tone is assigned is (i) 1030 nm to 1130 nm, 890 nm to 1170 nm when the observation target is the stomach, and (ii) the observation target is In the case of pancreas, 1030 nm to 1130 nm, or 8900 nm to 1170 nm, (iii) when the observation target is liver, 1020 nm to 1130 nm, or 890 nm to 1170 nm, (iv) when the observation target is mesentery, 1040 nm to 1130 nm, Alternatively, the imaging method is selected from 890 nm to 1160 nm, and (v) 1030 nm to 1140 nm, or 890 nm to 1170 nm, when the observation target is a kidney. In any one of claims 17 to 22,
The control unit allocates color tones to the plurality of detection results such that the luminance value of the detection result to which the red color tone is allocated is higher than the luminance value of the detection result to which the red color tone is allocated. An imaging method for generating the pseudo color image; In any one of claims 17 to 23,
The imaging method, wherein the control unit generates an enhanced image of the sample based on the plurality of detection results, and superimposes the enhanced image on the pseudo color image to generate a composite image of the sample. In any one of claims 17 to 24,
The imaging method, wherein the control unit generates and outputs a control signal for displaying the pseudo color image on a display device. In any one of claims 17 to 25,
The control unit is a combination of wavelengths for irradiating the sample with infrared light to obtain the plurality of measurement results, and reads out a combination of wavelengths different depending on the type of the sample from the storage unit, An imaging method of emitting infrared light of a plurality of wavelengths included in the combination and reading out the combination of the wavelengths corresponding to the type of the sample from the storage unit in response to the type of the input sample. In claim 26,
The imaging method includes an imaging device having at least a detection sensitivity with respect to light in an infrared light region than light in a visible light region. In any one of claims 17 to 27,
The color tone is composed of colors in RGB space,
The control unit determines the color mixing ratio of the RGB space according to the luminance value of the pixel at the corresponding position in each of the plurality of detection results corresponding to the wavelength, and determines the color of each pixel. An imaging method for generating the pseudo color image. In any one of claims 17 to 28,
Among the wavelength values of the infrared light, a wavelength for acquiring the detection result for assigning the red color tone is selected from 1030 nm to 1140 nm, and a wavelength for acquiring the detection result for assigning the color tone other than red is Selected from 930 nm to 1050 nm,
The imaging method, wherein the wavelength for acquiring the detection result to which the red color tone is allocated is longer than the wavelength for acquiring the detection result to which the color tone other than the red color is allocated. In any one of claims 17 to 28,
Of the wavelength values of the infrared light, a wavelength for acquiring a detection result for assigning the red color tone is selected from 890 nm to 1170 nm, and a wavelength for acquiring the detection result for assigning a color tone other than red is Selected from 1140 nm to 1400 nm,
An imaging method, wherein a wavelength for acquiring a detection result to which the red color tone is allocated is shorter than a wavelength for acquiring a detection result to which the color tone other than the red color is allocated. In any one of claims 17 to 30,
The light detection unit includes a hyperspectral camera that acquires a spectral value of an imaging target, and obtains a detection result obtained by irradiating the sample with infrared light having a predetermined wavelength band by the hyperspectral camera,
The control unit generates an enhanced image of the sample using at least a part of the detection result in the predetermined wavelength band, and detects the wavelength corresponding to the wavelength to which the color tone is assigned among the detection results in the predetermined wavelength band. An imaging method for generating the pseudo color image using a result.