JP7147890B2 - Information processing system, information processing method and program - Google Patents

Description

The present disclosure relates to an information processing system , an information processing method, and a program .

2. Description of the Related Art In recent years, various medical observation devices such as medical endoscopes and medical microscopes have been used in the medical field. In particular, in medical endoscopes, as disclosed in Patent Document 1 below, in addition to an observation image in a normal visible light band, light in a specific wavelength band is irradiated to irradiate an observation site. Attempts have been made to generate a spectral image in a specific wavelength band by imaging and to display the spectral image on a display screen.

JP 2012-81087 A

However, in the medical endoscope disclosed in Patent Document 1, the selection of the image to be displayed on the display screen is left to the judgment of the operator of the medical endoscope. If the operator does not notice the presence of the image, it may not be possible to improve the efficiency of the work under endoscopic observation.

In addition, the situation described above can occur not only under observation with a medical endoscope, but also when working under observation with a medical microscope.

Therefore, in view of the above circumstances, the present disclosure provides an information processing apparatus, an information processing method program, and a medical device capable of recommending a more appropriate living body observation image to a user during work using a medical observation device. We propose an observation system.

According to the present disclosure, a plurality of biological images generated by imaging the living body in a plurality of wavelength bands by a medical observation device that observes the inside of the living body, and a situation related to the situation when observing the inside of the living body a recommendation unit that determines at least one of the plurality of biometric images or at least one of images generated from the plurality of biometric images as a recommended image according to the situation, based on information; An information processing device is provided.

Further, according to the present disclosure, a plurality of biological images generated by imaging the living body in a plurality of wavelength bands by a medical observation device that observes the inside of the living body, and the situation when the inside of the living body is observed and determining at least one of the plurality of biometric images or at least one of an image generated from the plurality of biometric images as a recommended image according to the situation, based on , an information processing method is provided.

Further, according to the present disclosure, a computer capable of mutually communicating with a medical observation device that observes the inside of a living body and generates a plurality of biological images generated by imaging the living body in a plurality of wavelength bands, at least one of the plurality of biological images, or at least an image generated from the plurality of biological images, based on the plurality of biological images and situation information regarding the situation when the inside of the biological body is observed; A program is provided for realizing a recommendation function of determining any one of them as a recommended image according to the situation.

Further, according to the present disclosure, a medical observation device that observes the inside of a living body and generates a plurality of biological images by imaging the living body in a plurality of wavelength bands; at least one of the plurality of biometric images or at least an image generated from the plurality of biometric images based on the plurality of biometric images and situation information regarding the situation when observing the inside of the biometric images; and an information processing device having a recommendation unit that determines any one of the images as a recommended image according to the situation.

According to the present disclosure, a plurality of biological images generated by imaging the living body in a plurality of wavelength bands with a medical observation device for observing the inside of the living body, and status information about the situation when observing the inside of the living body. , at least one of the plurality of biometric images or at least one of the images generated from the plurality of biometric images is determined as the recommended image according to the situation.

As described above, according to the present disclosure, it is possible to recommend a more appropriate living body observation image to the user during work using a medical observation device.

In addition, the above effects are not necessarily limited, and together with the above effects or instead of the above effects, any of the effects shown in this specification, or other effects that can be grasped from this specification effect may be exhibited.

1 is an explanatory diagram schematically showing the overall configuration of a medical observation system according to an embodiment of the present disclosure; FIG. FIG. 4 is an explanatory diagram schematically showing an example of an imaging unit included in the medical observation device according to the same embodiment; FIG. 4 is an explanatory diagram schematically showing an example of an imaging unit included in the medical observation device according to the same embodiment; FIG. 4 is an explanatory diagram schematically showing an example of an imaging unit included in the medical observation device according to the same embodiment; 2 is a block diagram schematically showing an example of the configuration of the information processing apparatus according to the same embodiment; FIG. 4 is a block diagram schematically showing an example of the configuration of an image recommendation processing unit included in the information processing apparatus according to the embodiment; FIG. FIG. 4 is an explanatory diagram for explaining a situation identification database that the information processing apparatus according to the embodiment has; FIG. 2 is an explanatory diagram showing an example of absorption spectra of main substances in living tissue; FIG. 4 is an explanatory diagram for explaining a recommended image database that the information processing apparatus according to the embodiment has; FIG. 4 is an explanatory diagram schematically showing an example of a display screen in the information processing apparatus according to the same embodiment; FIG. 4 is an explanatory diagram schematically showing an example of a display screen in the information processing apparatus according to the same embodiment; It is the flowchart which showed an example of the flow of the information processing method which concerns on the same embodiment. It is a block diagram showing an example of the hardware constitutions of the information processor concerning the embodiment.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

Note that the description will be given in the following order.
1. Embodiment 1.1. Overall Configuration of Medical Observation System 1.2. Configuration of medical observation equipment 1.3. Configuration of information processing apparatus 1.4. Information processing method 1.5. Hardware configuration

(embodiment)
<Overall Configuration of Medical Observation System>
First, an overall configuration of a medical observation system 1 according to an embodiment of the present disclosure will be briefly described with reference to FIG. 1 . FIG. 1 is an explanatory diagram schematically showing the overall configuration of a medical observation system 1 according to this embodiment.

A medical observation system 1 according to this embodiment includes a medical observation device 10 for observing the inside of a living body and an information processing device 20, as shown in FIG.

The medical observation device 10 is a medical device that observes the inside of the living body S by inserting a portion thereof into the living body S or placing it near internal organs of the living body S. . Examples of such medical observation equipment 10 include various endoscope devices such as rigid endoscopes, flexible endoscopes, and arthroscopes, and various microscope devices.

The medical observation device 10 according to the present embodiment irradiates a part of the living body S to be observed with predetermined observation light, and images the part of the living body S irradiated with the observation light. A plurality of living body images including captured images of the living body in a band (a band corresponding to a wavelength range of approximately 400 nm to 800 nm) are generated.

Such biological images include, in addition to the captured images of the living body in the visible light band described above, for example, captured images of the living body in the near-infrared band (band corresponding to the wavelength range of about 800 nm to about 2.5 μm), , captured images of the living body in various wavelength bands, such as captured images of the living body in the ultraviolet band. In addition, the biological image includes, for example, an image when a living body is observed using ultrasound, an image when a living body is observed by an optical coherence tomography (OCT) method, and other imaging of a living body. May contain images.

A plurality of biological images, including captured images of the living body in the visible light band, generated by the medical observation device 10 are displayed as needed on a display device such as a display connected to the medical observation device 10 by wire or wirelessly. In addition, it is output to the information processing apparatus 20 according to the present embodiment, which will be described in detail below.

A detailed configuration of the medical observation device 10 will be described later.

The information processing apparatus 20 according to the present embodiment analyzes captured images generated by the medical observation device 10, determines an image that is more appropriate for work as a recommended image, and uses the recommended image as a recommended image for the medical observation device. 10 users (eg, doctors performing surgery, etc.). As a result, the information processing apparatus 20 according to the present embodiment grasps the situation when the user observes the inside of the living body, and provides an observation image corresponding to the situation to the user of the medical observation device 10 as a recommended image. can be recommended. As a result, the user of the medical observation device 10 can notice that there is an image that can be a more appropriate observation image in continuing the work being performed, and can use the medical observation device to It is possible to improve the efficiency of the work of

A detailed configuration of the information processing device 20 will be described later.

Note that FIG. 1 describes a case where the medical observation device 10 and the information processing device 20 according to the present embodiment exist as different devices, but the functions of the information processing device 20 described in detail below are Needless to say, it may be implemented as one function of the observation device 10 for use.

<Regarding the configuration of the medical observation device 10>
Next, a detailed configuration of the medical observation device 10 according to this embodiment will be briefly described with reference to FIGS. 1 to 2C. 2A to 2C are explanatory diagrams schematically showing an example of an imaging unit included in the medical observation device according to this embodiment.

As mentioned above, the medical observation device 10 according to the present embodiment can be partially inserted into the living body S or placed in the vicinity of internal organs of the living body S. It is a medical device for observing the inside of the body. This medical observation device 10 mainly includes a control unit 101, an observation unit 103, a light source unit 105, an imaging unit 107, and a display unit 109, as schematically shown in FIG. there is

The control unit 101 is implemented by, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The control unit 101 is a unit that controls overall functions of the medical observation device 10, and includes an observation unit 103, a light source unit 105, an imaging unit 107, and a display unit 109 that constitute the medical observation device 10. collectively control the operating status of Such a control unit 101 is, for example, a unit corresponding to a camera control unit (CCU) of a medical endoscope device or a medical microscope device. The control unit 101 also controls various information such as information on setting conditions set for each unit that configures the medical observation device 10 and information on the operating state of each unit that configures the medical observation device 10. can be output to the information processing apparatus 20 according to the present embodiment at any timing.

The observation unit 103 is a unit that is partly inserted into the living body S or arranged near internal organs of the living body S under the control of the control unit 101 . Such an observation unit 103 forms an observation image obtained by observing the inside of the living body S using the observation light emitted from the light source unit 105 on the imaging unit 107 in the subsequent stage. Such an observation unit 103 corresponds to, for example, an endoscope unit in medical endoscope equipment or a microscope unit in medical microscope equipment. The observation unit 103 according to this embodiment is not particularly limited, and a known endoscope unit, microscope unit, or the like can be applied.

The light source unit 105 is a unit that emits observation light for observing the inside of a living body under the control of the control unit 101 and is optically connected to the observation unit 103 . The light source unit 105 has at least a visible light source (such as a white light source) for obtaining a captured image of a living body in the visible light band. In addition to the visible light source, the light source unit 105 includes a near-infrared light source for obtaining a captured image of a living body in the near-infrared band, an ultraviolet light source, and various laser light sources that emit specific visible light. You may have well-known various light sources, such as. Under the control of the control unit 101, the light source unit 105 can switch the types of observation light emitted from these various light sources at arbitrary timing. Moreover, the light source unit 105 may further have a light source for realizing a specific function, such as an OCT light source and a range finding light source, in addition to the light source described above.

Under the control of the control unit 101, the imaging unit 107 captures an observation image of the inside of the living body S using observation light from the light source unit 105, and produces a plurality of living body images including a captured image of the living body in the visible light band. A unit that generates image data. Such imaging unit 107 is optically connected to the observation unit 103 . The imaging unit 107 according to the present embodiment captures a plurality of biological images by capturing various types of observation light (for example, light in various wavelength bands such as visible light, near-infrared light, ultraviolet light, etc.) while distinguishing them from each other. Generate image data.

In a general medical observation system, by using an imaging element sensitive to wavelengths in the visible light band, an image close to the situation directly observed by the human eye is captured, and after such an image is appropriately developed, , the observation result is displayed on the display device as a normal observation image. In addition, in general medical observation systems capable of special light observation, in addition to the normal observation mode that displays captured images in the visible light band, an image sensor that is sensitive to wavelengths in the near-infrared band is used to A fluorescence observation mode that observes fluorescence generated in vivo, and a narrow wavelength image (Narrow Band Imaging: NBI) observation mode that makes it easy to identify blood vessels at different depths from the skin surface by combining multiple specific narrow wavelengths. , often implements various functions.

However, the special light observation as described above is realized by combining an illumination device that selectively irradiates a specific wavelength and an optical filter that transmits or reflects the specific wavelength, so it is different from the normal observation mode of the visible light. Observation becomes difficult at the same time.

Therefore, in the medical observation device 10 according to the present embodiment, by including the imaging unit 107 as shown in FIGS. , can be obtained at the same time.

For example, as shown in FIG. 2A, the image pickup unit 107 includes various beam splitters such as a normal beam splitter or a polarizing beam splitter, two types of image pickup elements such as an image pickup element A and an image pickup element B, may have In the case of the image pickup unit 107 as shown in FIG. 2A, an observation image from the observation unit 103 is split into two optical paths by a beam splitter and formed on image pickup elements A and B, respectively. Here, for example, as the imaging device A, a visible light imaging device having sensitivity to light in the visible light band is used, and as the imaging device B, a near-infrared imaging device having sensitivity to light in the near-infrared band is used. , a captured image of the living body in the visible light band and a captured image of the living body in the near-infrared band can be generated. Further, by using other known imaging devices as the imaging device B, it is possible to generate various captured images of the living body in addition to the captured images of the living body in the visible light band.

Note that FIG. 2A illustrates the case where one beam splitter is used, but by using two or more beam splitters in combination, it is possible to branch the observation image into more optical paths, It becomes possible to simultaneously capture more types of captured images of a living body.

Further, as shown in FIG. 2B, for example, the imaging unit 107 includes a spectral prism in which a plurality of (three in FIG. 2B) optical prisms are joined together, and a plurality of types of imaging elements (in FIG. 2B, imaging elements A to 3 types of image sensor C). In the case of the imaging unit 107 as shown in FIG. 2B, the cemented surfaces of adjacent optical prisms are made to function as either a beam splitter, a polarizing beam splitter, or a wavelength selection filter, so that the observed image from the observation unit 103 is It is possible to branch into a plurality of optical paths (three optical paths in the case of FIG. 2B). An observation image from the observation unit 103 is branched into a plurality of optical paths by such a spectral prism, and formed on each imaging element provided at the end of each optical path. Here, for example, as the imaging device A, a visible light imaging device having sensitivity to light in the visible light band is used, and as the imaging device B, a near-infrared imaging device having sensitivity to light in the near-infrared band is used. , a captured image of the living body in the visible light band and a captured image of the living body in the near-infrared band can be generated. In addition, if necessary, by arranging another known imaging device at the position of the imaging device C, in addition to the captured image of the living body in the visible light band and the captured image of the living body in the near-infrared band In addition, it is possible to generate captured images of various living bodies.

In addition, the spectral prism as shown in FIG. 2B can split the observation image into more optical paths by increasing the number of optical prisms to be used, so that more types of captured images of the living body can be captured at the same time. It becomes possible to take an image.

The imaging unit 107 may also have a so-called multispectral sensor, as shown in FIG. 2C. A multispectral sensor, as schematically shown in FIG. 2C, is a two-dimensional image sensor that includes at least pixels that detect light intensities in a plurality of wavelength bands separately for each wavelength band at once. By using such a multispectral sensor, it is possible to acquire multispectral data, which is the intensity of light in a plurality of narrow wavelength bands, for each pixel of an image.

It should be noted that all the pixels constituting the multispectral sensor, as schematically shown in FIG. 2C, are preferably pixels that detect the intensity of light in a plurality of wavelength bands separately for each wavelength band at once. However, not all the pixels need to be pixels that detect the intensity of light in a plurality of wavelength bands separately for each wavelength band as described above at once. As described above, by performing various known interpolation processes using outputs from pixels that detect the intensity of light in a plurality of wavelength bands separately for each wavelength band at once, This is because the intensity of light in a plurality of wavelength bands can be calculated for each wavelength band even at the position of a pixel having no function.

By using the multispectral sensor as shown in FIG. 2C, the imaging unit 107 can be made smaller than the imaging unit 107 as shown in FIGS. It is possible to achieve further miniaturization.

2A or 2B and a multispectral sensor as shown in FIG. It goes without saying that 107 may also be configured.

For example, various biological images generated by the imaging unit 107 as shown in FIGS. A suitable image is selected and displayed on the display screen provided in the display unit 109 in the subsequent stage. Normally, the user of the medical observation device 10 often pays attention to the captured image of the living body in the visible light band as the observation image of the inside of the living body. Images of living organisms captured in the visible light band are often displayed.

Further, the imaging unit 107 outputs a plurality of biometric images generated as described above to the information processing apparatus 20 according to the present embodiment at any time under the control of the control unit 101 .

In addition, in the imaging unit 107 according to the present embodiment, only one imaging element exists in the imaging unit 107, for example, there is only one imaging element having sensitivity in the visible light band to the near infrared band. Even if you don't, you can consider it. Even in such a case, the light source unit 105 and the imaging unit 107 cooperate with each other under the control of the control unit 101 to perform imaging while switching the wavelength band of the light imaged on the imaging element. It becomes possible to generate biological images in the wavelength band.

The display unit 109 is a unit that displays at least one biometric image out of a plurality of biometric images generated by the imaging unit 107 to the user of the medical observation device 10 under the control of the control unit 101 . The number of display screens provided in the display unit 109 is not particularly limited, and the display unit 109 may have only one display screen, or may have a plurality of display screens. good too. The display unit 109 is not particularly limited, and a known display unit 109 can be used.

Further, the display screen of the display unit 109 according to the present embodiment appropriately displays information about recommended images recommended by the information processing apparatus 20, which will be described later.

An example of the configuration of the medical observation device 10 according to the present embodiment has been briefly described above with reference to FIGS. 1 to 2C. In addition to the various units shown in FIG. 1, the medical observation device 10 according to the present embodiment includes various other units such as an ultrasonic oscillator and an ultrasonic detection unit. may

<About the configuration of the information processing device 20>
Next, the information processing apparatus 20 according to this embodiment will be described in detail with reference to FIGS. 3 to 9. FIG.
FIG. 3 is a block diagram schematically showing an example of the configuration of the information processing apparatus according to this embodiment. FIG. 4 is a block diagram schematically showing an example of the configuration of the image recommendation processing section included in the information processing apparatus according to this embodiment. FIG. 5 is an explanatory diagram for explaining the situation identification database possessed by the information processing apparatus according to the present embodiment, and FIG. 6 is an explanatory diagram showing an example of absorption spectra of main substances in living tissue. . FIG. 7 is an explanatory diagram for explaining the recommended image database possessed by the information processing apparatus according to this embodiment. 8 and 9 are explanatory diagrams schematically showing examples of display screens in the information processing apparatus according to the present embodiment.

[Regarding the overall configuration of the information processing device 20]
As mentioned above, the information processing apparatus 20 according to the present embodiment analyzes the captured images generated by the medical observation device 10, determines a more appropriate image as a recommended image for work, and makes such a recommended image. It is a device for recommending images to the user of the medical observation device 10 . The information processing apparatus 20 mainly includes an information acquisition unit 201, an image recommendation processing unit 203, an output processing unit 205, a display control unit 207, and a storage unit 209, as schematically shown in FIG. have in

The information acquisition unit 201 is implemented by, for example, a CPU, ROM, RAM, communication device, and the like. The information acquisition unit 201 acquires various types of information from the medical observation device 10 . For example, the information acquisition unit 201 acquires image data of a plurality of living body images including captured images of the living body in the visible light band generated by the medical observation device 10 at any time. Further, the information acquisition unit 201 can acquire various types of information about the medical observation device 10 from the medical observation device 10 (more specifically, the control unit 101 of the medical observation device 10, etc.). Furthermore, the information acquisition unit 201 acquires from the medical observation device 10 information on the usage status of the recommended images, such as the user's selection results for the images recommended by the information processing apparatus 20 according to the present embodiment. is also possible. In addition, the information processing apparatus 20 according to the present embodiment can acquire information regarding various user operations performed by the user of the medical observation device 10, the operator of the information processing apparatus 20, or the like.

Here, the various information related to the medical observation device 10 includes, for example, setting information related to various settings of the medical observation device 10, characteristic information related to the characteristics of the medical observation device 10, the medical observation device 10 and the medical observation device. 10, and the like regarding the operating status of various peripheral devices. The setting information related to various settings of the medical observation device 10 includes, for example, setting information of the light source unit 105 in the medical observation device 10 (for example, information related to the wavelength of observation light), and what kind of peripheral device is the medical observation device. 10, and the like. In addition, the characteristic information related to the characteristics of the medical observation device 10 can include, for example, setting information related to the sensor spectral characteristics of the imaging unit 107 in the medical observation device 10 . In addition, as operation information related to the operation status of the medical observation device 10 and various peripheral devices linked with the medical observation device 10, for example, what kind of light source is operating as the light source unit 105, an energy device such as an electric scalpel, and so on. and the operating status of a smoke exhaust device for expelling mist and smoke present in body cavities to the outside of the body.

When the information acquisition unit 201 acquires image data related to a plurality of biological images from the medical observation device 10, the information acquisition unit 201 outputs the acquired image data to the image recommendation processing unit 203, which will be described later. Further, the information acquisition unit 201 may associate the acquired image data with time information related to the date and time when the image data was acquired, and record the image data as history information in the storage unit 209, which will be described later. good. Furthermore, the information acquisition unit 201 may record the usage history of the recommended images in the storage unit 209 to be described later, based on the information on the usage status of the recommended images acquired from the medical observation device 10 .

Further, when the information acquisition unit 201 acquires various kinds of information about the medical observation device 10 as described above from the medical observation device 10, the information acquisition unit 201 outputs the various acquired information to the image recommendation processing unit 203, which will be described later. Various types of information about the medical observation device 10 may be notified from the control unit 101 of the medical observation device 10 each time image data of a plurality of biological images are output to the information processing device 20 . In addition, the control unit 101 of the medical observation device 10 notifies various information about the medical observation device 10 each time image data is output, but instead notifies the control unit 101 of the medical observation device 10 at the timing when the content of any information is changed. may be notified by .

The image recommendation processing unit 203 is implemented by, for example, a CPU, ROM, RAM, and the like. The image recommendation processing unit 203 uses the image data and various information acquired by the information acquisition unit 201 to determine a recommended image according to the situation at the time when the inside of the living body is being observed, and determines the recommended image. to the user of the medical observation device 10. The image recommendation processing unit 203 generates at least one of a plurality of biometric images or a and a recommending unit that determines at least one of the images as a recommended image according to the situation.

A detailed configuration of the image recommendation processing unit 203 will be described later.

When the image recommendation processing unit 203 uses a plurality of biological images to determine a recommended image corresponding to the situation at the time when the inside of the biological body is being observed, the image recommendation processing unit 203 outputs information about the determined recommended image to the output processing unit 205 in the subsequent stage. output to In addition, the image recommendation processing unit 203 associates the processing result of the determined recommended image with time information related to the date and time when the processing result was obtained, and stores the processing result as history information in a subsequent storage unit. 209 may be stored.

The output processing unit 205 is implemented by, for example, a CPU, ROM, RAM, output device, communication device, and the like. The output processing unit 205 outputs the processing result regarding the determined recommended image output from the image recommendation processing unit 203 to the medical observation device 10 . The control unit 101 of the medical observation device 10 that has acquired the processing result displays the acquired processing result on the display unit 109, thereby recommending an observation image of the living body according to the situation to the user of the medical observation device 10. It becomes possible to In addition, the output processing unit 205 may directly output the processing result to the display unit 109 of the medical observation device 10 while cooperating with the display control unit 207 in the subsequent stage.

Note that the output processing unit 205 can output the processing result regarding the recommended image output from the image recommendation processing unit 203 as a printed matter or as data to an external information processing apparatus, server, or the like. be.

The display control unit 207 is implemented by, for example, a CPU, ROM, RAM, output device, communication device, and the like. The display control unit 207 outputs various results output from the output processing unit 205 to an output device such as a display provided in the information processing device 20 or an output device provided outside the arithmetic processing device 200 (for example, a medical observation device). It performs display control when displaying on the display unit 109) of the device 10 or the like. This allows the user of the medical observation device 10 to grasp various results on the spot.

The storage unit 209 is an example of a storage device included in the information processing device 20, and is implemented by a RAM, a storage device, or the like included in the information processing device 20. FIG. The storage unit 209 stores various databases used by the information processing apparatus 20 according to the present embodiment when performing image recommendation processing, which will be described in detail below. In addition, the storage unit 209 may record various types of history information related to image recommendation processing, which will be described in detail below. Further, in the storage unit 209, various parameters and progress of processing that need to be stored when the information processing apparatus 20 according to the present embodiment performs some processing, or various databases, programs, etc. Recorded as appropriate. The storage unit 209 allows the information acquisition unit 201, the image recommendation processing unit 203, the output processing unit 205, the display control unit 207, and the like to freely perform read/write processing.

[Regarding the configuration of the image recommendation processing unit 203]
Next, the configuration of the image recommendation processing unit 203 included in the information processing apparatus 20 according to this embodiment will be described in detail with reference to FIGS. 4 to 7. FIG.
As mentioned above, the image recommendation processing unit 203 according to the present embodiment utilizes the image data and various information acquired by the information acquisition unit 201 to determine the conditions at the time the inside of the living body is being observed. In addition to determining the recommended image, an image recommendation process for recommending the recommended image to the user of the medical observation device 10 is performed. The image recommendation processing unit 203 includes a situation identification unit 211, a recommendation unit 213, and a recommended image generation unit 215, as schematically shown in FIG.

The situation identification unit 211 is realized by, for example, CPU, ROM, RAM, and the like. The situation identification unit 211 is a processing unit that identifies the situation when the inside of a living body is observed based on a plurality of biological images output from the medical observation device 10 . More specifically, the situation identifying unit 211 acquires the setting information related to the settings of the medical observation device 10, the characteristic information related to the characteristics of the medical observation device 10, the medical observation device 10 and the medical condition acquired by the information acquisition unit 201. Based on at least one of the operating information about the operating state of the peripheral device linked with the observation device 10, the biological image, or the secondary information generated based on the biological image, the situation when the inside of the living body is being observed Identify.

The secondary information generated based on the biometric image is not particularly limited, and may be information that is secondarily calculated by performing various image processing or information processing on the biometric image. Any one can be used. Such secondary information includes, for example, information on the spectral reflectance or absorption coefficient of organs and biological substances present inside the living body, which can be calculated by focusing on the captured image of the living body in a certain wavelength band, and , information on blood oxygen saturation, information on recognition results of image recognition processing on biological images using image information on internal organs and medical instruments used in medical practice, etc. can be mentioned.

More specifically, the situation identification unit 211 collects setting information regarding the setting of the medical observation device 10, characteristic information regarding the characteristics of the medical observation device 10, the medical observation device 10, and peripheral devices that cooperate with the medical observation device 10. at least one of the operating information related to the operating state of the biometric image, the biometric image, or the secondary information generated based on the biometric image, and the situation when the biometric image is generated are associated with each other in advance with reference to a situation identification database It is preferable to identify the situation when the inside of the living body is being observed.

An example of such a situation identification database is schematically shown in FIG.
In the situation identification database stored in the storage unit 209 or the like according to the present embodiment, as schematically shown in FIG. stored. The data format of such situation-associated data is not particularly limited, but examples include those in which the various types of information and situations described above are recorded in a data format such as a lookup table. can be done. By referring to the situation association data using the acquired various information as described above, the situation identification unit 211 can identify the situation corresponding to the acquired various information.

For example, if there is information that an energy device such as an electric scalpel is in operation, or information that a smoke exhaust device is in operation, by referring to the various types of information described above, The situation identification unit 211 can refer to the situation association data by focusing on the information as described above, and identify that the visibility of the field of view is degraded due to smoke or mist caused by the energy device. can.

In addition, the situation identification database may store information on the absorption coefficients or spectral reflectances of organs and biological substances existing inside the living body. For example, as shown in FIG. 6, biomaterials present inside living organisms have absorption coefficients unique to each biomaterial. Therefore, by paying attention to the captured images of the living body in a specific wavelength band included in the plurality of living body images, it is possible to detect the biological substance present in the obtained living body images. In addition, organs and biological substances existing inside a living body have their own spectral reflectances, like absorption coefficients. Here, by focusing on captured images of a living body in a specific wavelength band included in a plurality of living body images, the light source characteristics of the light source unit 105 of the medical observation device 10, the sensor characteristics of the imaging unit 107, and the like, It is possible to calculate the spectral reflectance in the captured biological image. Therefore, the situation identifying unit 211 can identify the situation when the inside of the living body is being observed by comparing the obtained information with pre-recorded information about the spectral reflectance and the extinction coefficient. can.

For example, focusing on the above information (in particular, the biometric image itself, secondary information obtained from the biometric image, etc.), the spectral reflectance or absorption coefficient corresponding to blood is specified from the value of the spectral reflectance or absorption coefficient. If so, the situation identifying unit 211 can identify that there is a blood vessel and that bleeding may occur. Focusing on the above information (in particular, the biometric image itself, secondary information obtained from the biometric image, etc.), when data corresponding to water is detected, the situation identifying unit 211 determines whether the energy device is used. It becomes possible to presume that mist is generated along with this.

Further, the situation identification database may store image information related to organs existing inside a living body, medical instruments used in medical practice, and the like. By performing image recognition on the biological image obtained from the medical observation device 10 using this image information, various organs present in the biological image, blood flow blockage medical equipment, blood vessel anastomosis medical equipment, and the like can be recognized. medical instruments, etc. can be detected. This enables the situation identifying unit 211 to estimate the surgical site, the surgical situation, and the like. Note that the image recognition processing performed by the situation identification unit 211 is not particularly limited, and various known image recognition processing such as pattern recognition and machine learning recognition can be appropriately used.

In addition to the various types of data described above, the situation identification database may appropriately store various types of situation estimation data for estimating situations.

When the situation identifying unit 211 identifies the situation when the inside of the living body is being observed while appropriately using the situation identifying database or the like as described above, the situation identifying unit 211 sends the situation information about the identified situation to the recommending unit 213 in the latter stage. output to

It should be noted that the situation identification database as shown in FIG. 5 can be appropriately updated via various networks and external memory devices. Moreover, the situation identification unit 211 may automatically update the situation identification database by a known machine learning process or the like.

The recommendation unit 213 is implemented by, for example, a CPU, ROM, RAM, and the like. The recommending unit 213 recommends a plurality of or at least one of images generated from a plurality of biometric images is determined as a recommended image according to the situation. Also, the recommendation unit 213 recommends the determined recommended image to the user of the medical observation device 10 .

More specifically, the recommendation unit 213 refers to a recommended image database in which situations in which the inside of a living body is observed and types of recommended images that can be presented in each situation are associated in advance with each other, It is preferable to recommend a recommended image according to the situation to the user.

An example of such a recommended image database is schematically shown in FIG.
In the recommended image database stored in the storage unit 209 or the like according to the present embodiment, as schematically shown in FIG. possible recommended image types are recorded in association with each other. If the situation information output from the situation identification unit 211 describes that a certain situation (situation A) is occurring, the recommendation unit 213 refers to the item related to situation A in the recommended image database, As the recommended image, it is possible to decide to recommend, for example, a near-infrared image.

Also, in the recommended image database, when a plurality of recommended images are associated with a certain situation, the recommending unit 213 does not recommend one optimal image from among the plurality of recommended images. It is preferable to recommend all associated recommended images. Further, when the situation information output from the situation identification unit 211 lists a plurality of possible situations, the recommendation unit 213 selects a recommended image for one of the plurality of situations. , it is preferable to recommend recommended images corresponding to all situations described in the situation information. This is because it is preferable to have the user of the medical observation device 10 (that is, the doctor performing the operation, etc.) make the decision as to which recommended image to use.

After determining the recommended image to be recommended to the user of the medical observation device 10 , the recommendation unit 213 outputs information about the determined recommended image to the output processing unit 205 . Information related to such recommended images is not particularly limited, but may include, for example, at least one of text information indicating the type of the recommended image and a thumbnail image of the recommended image. The output processing unit 205 causes the user of the medical observation device 10 or the operator of the information processing device 20 to output the information about the recommended image. This allows the user of the medical observation device 10 or the operator of the information processing device 20 to decide on the spot whether or not to use the presented recommended image. In addition, the output processing unit 205 presents information about the recommended image to the user of the medical observation device 10 or the operator of the information processing device 20 using means other than text and images, such as sound and vibration. may

After determining the type of recommended image to be recommended to the user of the medical observation device 10, the recommendation unit 213 requests the subsequent recommended image generation unit 215 to generate the determined recommended image, if necessary.

When the user of the medical observation device 10 selects to refer to the recommended image, the recommendation unit 213 appropriately presents the selected recommended image to the user of the medical observation device 10 via the output processing unit 205. present it. In addition, the recommendation unit 213 appropriately records the history of the selection result by the user or the like of the medical observation device 10 in the storage unit 209 or the like. The selection of the user of the observation device 10 or the like may be assisted.

Note that the recommended image database as shown in FIG. 7 can be appropriately updated via various networks and external memory devices. Also, the recommendation unit 213 may automatically update the recommended image database by a known machine learning process or the like.

The recommended image generation unit 215 is implemented by, for example, a CPU, ROM, RAM, and the like. The recommended image generating unit 215 generates at least one of an image of at least a part of the plurality of biological images and an image obtained by visualizing secondary information generated by analyzing at least one of the plurality of biological images. is used to generate a recommended image recommended by the recommendation unit 213 .

For example, when the recommendation unit 213 recommends an observation image in a specific wavelength band (for example, a near-infrared band) as a recommended image, the recommended image generation unit 215 generates Using the picked-up image of the living body, observation images in the corresponding wavelength band, thumbnail images thereof, and the like can be generated as recommended images. In addition, for example, as shown in the database of FIG. 7, when oxygen saturation visualization information that visualizes the distribution of oxygen saturation that can be specified as secondary information of the biological image is the recommended image, such secondary information is used. Then, corresponding recommended images, thumbnail images thereof, and the like can be generated. In addition to these images, various known images such as blood vessel-enhanced images emphasizing the presence of blood vessels and images showing the state of blood flow using known methods such as the principle of NBI, etc. A thumbnail image or the like thereof can be generated as appropriate.

In addition, when the plurality of biological images include a captured image of the living body in the visible light band and a captured image of the living body in the near-infrared band (for example, a fluorescence image in the near-infrared band). Focus on Here, in the fluorescence image in the near-infrared band, portions corresponding to fluorescence are displayed in a predetermined color tone, and portions not corresponding to fluorescence (portions with a brightness value of zero or close to zero) are displayed in black. often displayed. Even if such a fluorescence image is presented as a recommended image, there is a high possibility that it will be difficult to understand the positional relationship between organs and the like. Therefore, the recommended image generation unit 215 first converts the color tone of a region (for example, a portion corresponding to fluorescence) where light in the near-infrared band is detected to a predetermined color tone ( For example, the color tone is changed to green, which is a color tone that does not exist in vivo, and the luminance value of the region where light in the near-infrared band is not detected is changed to zero. After that, the recommended image generating unit 215 generates a superimposed image by superimposing the changed captured image of the living body in the near-infrared band on the captured image of the living body in the visible light band, and generates a superimposed image as a recommended image. good too.

An example of the functions of the information processing apparatus 20 according to the present embodiment has been described above. Each component described above may be configured using general-purpose members and circuits, or may be configured by hardware specialized for the function of each component. Also, the functions of each component may be performed entirely by a CPU or the like. Therefore, it is possible to appropriately change the configuration to be used according to the technical level at which the present embodiment is implemented.

Note that it is possible to create a computer program for realizing each function of the information processing apparatus according to the present embodiment as described above and implement it in a personal computer or the like. A computer-readable recording medium storing such a computer program can also be provided. Recording media include, for example, magnetic disks, optical disks, magneto-optical disks, and flash memories. Also, the above computer program may be distributed, for example, via a network without using a recording medium.

[Examples of recommended images]
Next, a method of presenting recommended images determined by the recommendation unit 213 will be briefly described with reference to FIGS. 8 and 9. FIG.
8 is viewed by the user of the medical observation device 10 or the operator of the information processing device 20, such as the display unit 109 of the medical observation device 10 and the display device (not shown) of the information processing device 20. FIG. An example of a display screen is shown typically.

Normally, when various medical procedures such as surgery are being performed, the control unit 101 of the medical observation device 10 or the display control unit 207 of the information processing device 20 operates as shown schematically in the upper left diagram of FIG. Thus, as a normal display, it is considered that a captured image of a living body in the visible light band (visible light image) is displayed on the display screen. Here, when the recommending unit 213 proposes a recommended image according to the situation, the recommending unit 213, via the output processing unit 205, generates a recommended image, for example, as schematically shown in the upper right diagram of FIG. Text information indicating the type of image may be superimposed on the visible light image and displayed. In addition to the text information, the recommendation unit 213 also displays a text image indicating the type of the recommended image and a thumbnail image of the recommended image, as schematically shown in the lower left diagram of FIG. 8, for example. Alternatively, thumbnail images of recommended images may be displayed independently in a so-called picture-in-picture display format. Further, the recommendation unit 213 causes a display area for displaying information about the recommended image to be provided around the visible light image, as schematically shown in the lower right diagram of FIG. Text information indicating the type of recommended image, thumbnail images of the recommended images, and text information describing the recommended images may be displayed together.

By generating the display screen as shown in FIG. 8, it is possible for the user of the medical observation device 10 to select whether or not to display the recommended image on the display screen.

Moreover, as schematically shown in FIG. 9, instead of within one display screen as shown in FIG. The recommended image as described above and information related to the recommended image may be displayed on the sub-monitor.

It should be noted that the display of information on the recommended images as shown in FIGS. 8 and 9 may disappear each time the user of the medical observation device 10 does not select to use the recommended images. . Also, the display of information on recommended images as shown in FIGS. 8 and 9 may automatically disappear after a certain period of time has passed. In addition, the display of information related to recommended images as shown in FIGS. 8 and 9 may be recorded by various methods so that playback confirmation can be performed during or after surgery.

On the other hand, when the user of the medical observation device 10 selects the use of the recommended image as a result of displaying the information about the recommended image as shown in FIGS. 8 and 9, the selected recommended image is displayed on the display screen. will be displayed. Further, the timing of re-switching from the recommended image to the normal visible light image is not particularly limited, and is appropriately performed according to the user's operation of the medical observation device 10 or the like.

The method of presenting the recommended image determined by the recommendation unit 213 has been briefly described above.

[Specific examples of recommended images]
Specific examples of recommended images will be briefly described below with some examples.

○Case 1
For example, when bleeding occurs during an operation under the medical observation device 10, it is important for the user of the medical observation device 10 to quickly find the bleeding point and perform hemostatic treatment. However, if there is a large amount of bleeding, the surrounding area may become bloody and difficult to find the bleeding point.

At this time, it is assumed that blood is recognized and a bleeding situation is specified by situation specifying processing (for example, analysis processing of a captured image in a specific wavelength band) by the situation specifying unit 211 . On the other hand, in the near-infrared band, blood has a low absorption coefficient as shown in FIG. 6, and light in the near-infrared band passes through blood. Therefore, the recommendation unit 213 refers to a recommended image database or the like, and selects an observation image in a wavelength band in which the absorption coefficient of blood is relatively low as a recommended image. Using such a recommended image makes it easier for the user of the medical observation device 10 to visually recognize the surface of the living body, which is difficult to see because it is buried in blood, and to find the bleeding point.

○Case 2
For example, when an energy device such as an electric scalpel is used, the inside of the body cavity may be covered with surgical smoke or mist, resulting in poor visibility.

At this time, it is assumed that the situation that surgical smoke or mist is occurring is identified by the situation identification processing (for example, analysis processing of captured images in a specific wavelength band and information on the operation status of various devices) by the situation identification unit 211. do. In this case, the recommendation unit 213 refers to a recommended image database or the like, and selects an observation image in the near-infrared band, in which the absorption coefficient of surgical smoke and mist is relatively low, as a recommended image. By using such a recommended image, the user of the medical observation device 10 can continue medical practice without interruption due to surgical smoke or mist.

In addition, after the above situation has passed for a while, if it is determined that the surgical smoke, mist, etc. has disappeared due to the normal operation of the smoke exhaust equipment, etc., the recommendation unit 213 , the display may be switched to an observation image in a normal visible light band. As a result, the user of the medical observation device 10 can continue medical practice using familiar images.

○Case 3
For example, when safely excising a blood vessel, a doctor applies a blood flow-blocking medical device such as a clip to the blood vessel to completely block the blood flow, and then excises the blood vessel with a predetermined surgical instrument. At this time, there are cases where the blood flow blocking medical device, such as a clip, is not sufficiently applied and the blood flow is not completely blocked. Unintentional bleeding occurs when the blood vessel is resected in a situation where the blood flow is not completely cut off. Similarly, it is necessary to appropriately anastomose blood vessels or tissues with each other using a blood vessel anastomosis medical instrument such as a stapler, and confirm whether or not blood flow has been resumed.

At this time, if situation information indicating that a blood flow blockage medical device or a blood vessel anastomosis medical device has been detected is generated by the situation identification processing (for example, image recognition processing for clips, staplers, etc.) by the situation identification unit 211, the recommended The unit 213 can calculate the oxygen saturation by a known method as secondary information based on the biometric image, and use the image obtained by visualizing the distribution of the oxygen saturation as the recommended image.

Also, instead of an image that visualizes the distribution of oxygen saturation, an image of the living body captured in the near-infrared band corresponding to the distribution of blood can be used as the recommended image. For example, a fluorescent reagent called ICG (indocyanine green) has the property of emitting fluorescence of 830 nm by excitation light of 760 nm by binding to plasma proteins in blood. Therefore, when grasping the situation as described above, the recommendation unit 213 first recommends the use of a fluorescent reagent to the doctor who is the user of the medical observation device 10, and generates a A near-infrared image with a wavelength of about 830 nm, which is obtained by the observation, may be recommended as an observation image in the near-infrared band corresponding to the distribution of blood.

○Case 4
For example, during surgery under the medical observation device 10, the user of the medical observation device 10 may want to know information about the course of blood vessels and the depth of the blood vessels. Images may not provide enough of this information.

At this time, when the presence of a blood vessel in the biological image is identified by the situation identification process (for example, the analysis process of the captured image in a specific wavelength band) by the situation identification unit 211, the recommendation unit 213 An image in a wavelength band suitable for visual recognition of blood vessels can be recommended as a blood vessel-enhanced image by means similar to the principle of NBI.

Specific examples of the recommended image have been briefly described above while giving some examples. Images can be used as recommended images.

<Information processing method>
Next, the flow of the information processing method according to this embodiment will be briefly described with reference to FIG. FIG. 10 is a flowchart showing an example of the flow of the information processing method according to this embodiment.

In the information processing method according to the present embodiment, first, a biological image is generated by the medical observation device 10 and output to the information processing device 20 . When the information acquisition unit 201 of the information processing apparatus 20 acquires data of a biological image output from the medical observation device 10 (step S101), the information acquisition unit 201 outputs the acquired data to the image recommendation processing unit 203 .

The situation identification unit 211 of the image recommendation processing unit 203 identifies the situation when the inside of the living body is being observed by the method described above (step S103), and sends the situation information about the identified situation to the recommendation unit. 213.

The recommendation unit 213 uses the situation information and the like output from the situation identification unit 211 to determine whether or not there is an image that can be recommended to the user or the like of the medical observation device 10 (step S105). If there is no recommendable image (step S105-NO), step S115, which will be described later, is performed. On the other hand, if there is an image that can be recommended (step S105-YES), the recommendation unit 213 notifies the user of the medical observation device 10 of information on the recommended image by the method described above ( step S107).

If the user or the like of the medical observation device 10 does not select a recommended image (step S109-NO), step S115, which will be described later, is performed. On the other hand, if the user or the like of the medical observation device 10 selects a recommended image (step S109-YES), the recommended image is generated by the recommended image generator 215 as necessary (step S111), and output processing is performed. The unit 205 and the control unit 101 of the medical observation device 10 work together to display the recommended image on the screen (step S113). After that, the process returns to step S101 to continue the information processing method according to the present embodiment.

On the other hand, if it is determined in step S105 that there is no recommendable image, or if no recommended image is selected in step S109, a captured image of the living body in the visible light band is generated (step S115). The output processing unit 205 and the control unit 101 of the medical observation device 10 cooperate to display a visible light image on the screen (step S117). After that, the process returns to step S101 to continue the information processing method according to the present embodiment.

The flow of the information processing method according to the present embodiment has been briefly described above with reference to FIG.

<About hardware configuration>
Next, the hardware configuration of the information processing device 20 according to the embodiment of the present disclosure will be described in detail with reference to FIG. 11 . FIG. 11 is a block diagram for explaining the hardware configuration of the information processing device 20 according to the embodiment of the present disclosure.

The information processing device 20 mainly includes a CPU 901 , a ROM 903 and a RAM 905 . The information processing apparatus 20 further includes a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, and a connection port 923. and a communication device 925 .

The CPU 901 functions as an arithmetic processing device and a control device, and controls all or part of operations in the information processing device 20 according to various programs recorded in the ROM 903 , RAM 905 , storage device 919 , or removable recording medium 927 . A ROM 903 stores programs, calculation parameters, and the like used by the CPU 901 . The RAM 905 temporarily stores programs used by the CPU 901, parameters that change as appropriate during execution of the programs, and the like. These are interconnected by a host bus 907 comprising an internal bus such as a CPU bus.

The host bus 907 is connected via a bridge 909 to an external bus 911 such as a PCI (Peripheral Component Interconnect/Interface) bus.

The input device 915 is operation means operated by a user, such as a mouse, keyboard, touch panel, button, switch, and lever. Further, the input device 915 may be, for example, a remote control means (so-called remote control) using infrared rays or other radio waves, or an external connection device such as a mobile phone or PDA corresponding to the operation of the information processing device 20. 929 may be used. Furthermore, the input device 915 is composed of, for example, an input control circuit that generates an input signal based on the information input by the user using the operation means and outputs the signal to the CPU 901 . By operating the input device 915, the user can input various data to the information processing apparatus 20 and instruct processing operations.

The output device 917 is configured by a device capable of visually or audibly notifying the user of the acquired information. Such devices include display devices such as CRT display devices, liquid crystal display devices, plasma display devices, EL display devices and lamps, audio output devices such as speakers and headphones, printer devices, mobile phones, and facsimiles. The output device 917 outputs, for example, results obtained by various processes performed by the information processing device 20 . Specifically, the display device displays the results obtained by various processes performed by the information processing device 20 as text or images. On the other hand, the audio output device converts an audio signal including reproduced audio data, acoustic data, etc. into an analog signal and outputs the analog signal.

The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 20 . The storage device 919 is composed of, for example, a magnetic storage device such as a HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like. The storage device 919 stores programs executed by the CPU 901, various data, and various data acquired from the outside.

The drive 921 is a recording medium reader/writer and is built in or externally attached to the information processing apparatus 20 . The drive 921 reads out information recorded on a removable recording medium 927 such as a mounted magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 905 . The drive 921 can also write records to a removable recording medium 927 such as a mounted magnetic disk, optical disk, magneto-optical disk, or semiconductor memory. The removable recording medium 927 is, for example, DVD media, HD-DVD media, Blu-ray (registered trademark) media, or the like. Also, the removable recording medium 927 may be a CompactFlash (registered trademark) (CompactFlash: CF), a flash memory, an SD memory card (Secure Digital memory card), or the like. Also, the removable recording medium 927 may be, for example, an IC card (Integrated Circuit card) equipped with a contactless IC chip, an electronic device, or the like.

The connection port 923 is a port for directly connecting a device to the information processing device 20 . Examples of the connection port 923 include a USB (Universal Serial Bus) port, IEEE1394 port, SCSI (Small Computer System Interface) port, and the like. Other examples of the connection port 923 include an RS-232C port, an optical audio terminal, an HDMI (High-Definition Multimedia Interface) port, and the like. By connecting the external connection device 929 to the connection port 923 , the information processing apparatus 20 directly acquires various data from the external connection device 929 and provides various data to the external connection device 929 .

The communication device 925 is, for example, a communication interface configured with a communication device or the like for connecting to the communication network 931 . The communication device 925 is, for example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB). Also, the communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), a modem for various types of communication, or the like. For example, the communication device 925 can transmit and receive signals and the like to and from the Internet and other communication devices according to a predetermined protocol such as TCP/IP. Also, the communication network 931 connected to the communication device 925 is configured by a network or the like connected by wire or wireless, and may be, for example, the Internet, home LAN, infrared communication, radio wave communication, satellite communication, or the like. .

An example of the hardware configuration capable of realizing the functions of the information processing device 20 according to the embodiment of the present disclosure has been described above. Each component described above may be configured using general-purpose members, or may be configured by hardware specialized for the function of each component. Therefore, it is possible to appropriately change the hardware configuration to be used according to the technical level at which the present embodiment is implemented.

Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is obvious that those who have ordinary knowledge in the technical field of the present disclosure can conceive of various modifications or modifications within the scope of the technical idea described in the claims. is naturally within the technical scope of the present disclosure.

Also, the effects described herein are merely illustrative or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification in addition to or instead of the above effects.

Note that the following configuration also belongs to the technical scope of the present disclosure.
(1)
Based on a plurality of living body images generated by imaging the living body in a plurality of wavelength bands with a medical observation device for observing the inside of the living body, and situation information regarding the situation when the inside of the living body is observed. , an information processing apparatus comprising: a recommending unit that determines at least one of the plurality of biometric images or at least one of images generated from the plurality of biometric images as a recommended image according to the situation.
(2)
the plurality of biological images include at least captured images of the biological body in a visible light band,
further comprising a display control unit that performs display control of a display screen on which the biological image is displayed;
The display control unit causes the display screen to display the captured image of the living body in the visible light band, and changes the display content of the display screen to the recommended image when the user selects display of the recommended image. The information processing apparatus according to (1), which switches to
(3)
The recommendation unit refers to a recommended image database in which situations in which the inside of a living body is to be observed and types of the recommended images that can be presented in each of the situations are associated in advance with each other. The information processing apparatus according to (1) or (2), which determines a recommended image according to the image.
(4)
The recommended image using at least one of an image of at least part of the plurality of biological images and an image visualizing secondary information generated by analyzing at least one of the plurality of biological images. The information processing apparatus according to any one of (1) to (3), further comprising a recommended image generation unit that generates a .
(5)
The plurality of biological images include at least a captured image of the biological body in a visible light band and a captured image of the biological body in a near-infrared band,
The recommended image generation unit
changing the color tone of the area in which light in the near-infrared band is detected in the captured image of the living body in the near-infrared band to a predetermined color tone, and changing the color tone of the area in which the light in the near-infrared band is not detected; After changing the luminance value to zero, the superimposed image obtained by superimposing the captured image of the living body in the near-infrared band after the change on the captured image of the living body in the visible light band is used as the recommended image. The information processing apparatus according to (4), which generates.
(6)
the plurality of biological images include at least captured images of the biological body in a visible light band,
The recommendation unit displays information about the recommended image on a display screen on which the captured image of the living body in the visible light band is displayed, and allows the user to select whether to display the recommended image, ( 1) The information processing apparatus according to any one of (5).
(7)
The information processing apparatus according to (6), wherein the information about the recommended image is at least one of text information indicating a type of the recommended image and a thumbnail image of the recommended image.
(8)
Any one of (1) to (7), wherein the recommendation unit displays the recommended image on a display screen different from the display screen on which the captured image of the living body in the visible light band is displayed. The information processing device according to .
(9)
further comprising a situation identifying unit that identifies a situation when the inside of the living body is being observed;
The situation identification unit provides setting information related to settings of the medical observation device, characteristic information related to characteristics of the medical observation device, and operation information related to operating states of the medical observation device and peripheral devices linked with the medical observation device. , based on at least one of the biometric image or secondary information generated based on the biometric image, specifying a situation when the interior of the biometric organism is being observed, any of (1) to (8) The information processing device according to claim 1.
(10)
The situation identification unit provides setting information related to settings of the medical observation device, characteristic information related to characteristics of the medical observation device, and operation information related to operating states of the medical observation device and peripheral devices linked with the medical observation device. , at least one of the biometric image or secondary information generated based on the biometric image, and a situation in which the biometric image is generated are associated with each other in advance, with reference to a situation identification database; The information processing device according to (9), which specifies the situation when the inside of the is being observed.
(11)
The situation identification database further records information on the spectral reflectance or absorption coefficient of organs and biological substances existing inside the living body,
The situation identifying unit generates information about spectral reflectance as secondary information generated based on the biometric image, or refers to the biometric image to specify an absorption coefficient in a specific wavelength band, The information processing apparatus according to (10), wherein the information processing apparatus identifies the situation when the inside of the living body is being observed by comparing with information about the spectral reflectance or the extinction coefficient of the organs and biological substances existing inside the living body. .
(12)
The situation identification database further records image information related to organs existing inside the living body and medical instruments used in medical practice,
The situation identifying unit generates, as secondary information generated based on the biometric image, information related to a recognition result obtained by performing image recognition processing on the biometric image using the image information,
The information processing apparatus according to (10) or (11), wherein the information about the recognition result is used to identify the situation when the inside of the living body is being observed.
(13)
The living body image includes at least a captured image of the living body in a visible light band and a captured image of the living body in a near-infrared band,
(1) to (12), wherein the recommendation unit determines an image of the living body captured in the near-infrared band as the recommended image when information indicating that blood has been detected is obtained as the situation information; The information processing device according to any one of the above.
(14)
The living body image includes at least a captured image of the living body in a visible light band and a captured image of the living body in a near-infrared band,
When the recommendation unit acquires at least one of information indicating that smoke or mist has been detected or information regarding the use of a medical energy device as the situation information, The information processing apparatus according to any one of (1) to (13), wherein a captured image is determined as the recommended image.
(15)
When the recommendation unit acquires at least one of information indicating that smoke or mist is no longer detected or information indicating that use of the medical energy device has ended as the situation information, the recommendation unit displays the information on the display screen. The information processing apparatus according to (14), wherein the image to be captured is switched from the recommended image to the captured image of the living body in the visible light band.
(16)
When the recommendation unit acquires, as the situation information, information indicating that a medical device for blocking blood flow or an anastomosis medical device has been detected, the oxygen saturation distribution generated as secondary information based on the biological image. The information processing according to any one of (1) to (15), wherein a visualized image of the living body or a captured image of the living body in a near-infrared band corresponding to blood distribution is determined as the recommended image. Device.
(17)
When the recommendation unit acquires information indicating that the presence of a blood vessel has been detected as the situation information, the recommendation unit determines a captured image of the living body in a wavelength band suitable for visual recognition of the blood vessel as the recommended image, ( 1) The information processing apparatus according to any one of (16).
(18)
The medical observation equipment has a multispectral sensor that includes at least pixels that detect the intensity of light in a plurality of wavelength bands separately for each wavelength band at once. The information processing apparatus according to any one of (1) to (17), wherein the biological image is generated in a wavelength band of .
(19)
(1) the medical observation equipment has a plurality of imaging elements corresponding to specific wavelength bands, and the biological images in a plurality of wavelength bands are generated from output results of the respective imaging elements; The information processing device according to any one of (17).
(20)
The medical observation device generates the biological image in a plurality of wavelength bands by performing imaging while switching wavelength bands of light formed on an imaging element of the medical observation device, (1) to The information processing apparatus according to any one of (17).
(21)
The information processing apparatus according to any one of (1) to (20), wherein the medical observation device is a medical endoscope or a medical microscope.
(22)
Based on a plurality of living body images generated by imaging the living body in a plurality of wavelength bands with a medical observation device for observing the inside of the living body, and situation information regarding the situation when the inside of the living body is observed. and determining at least one of the plurality of biometric images or at least one of an image generated from the plurality of biometric images as the recommended image according to the situation.
(23)
A computer capable of mutually communicating with a medical observation device that observes the inside of a living body and generates a plurality of biological images generated by imaging the living body in a plurality of wavelength bands,
at least one of the plurality of biological images, or at least an image generated from the plurality of biological images, based on the plurality of biological images and situation information regarding the situation when the inside of the biological body is observed; A program for realizing a recommendation function of determining any one of them as a recommended image according to the situation.
(24)
a medical observation device that observes the inside of a living body and generates a plurality of living body images by imaging the living body in a plurality of wavelength bands;
at least one of the plurality of biometric images, or the plurality of biometric images, based on the plurality of biometric images generated by the medical observation device and situation information regarding the situation when the inside of the biometric images is observed; an information processing device having a recommendation unit that determines at least one of the images generated from the biometric image of the above as a recommended image according to the situation;
A medical observation system comprising:

1 Medical Observation System 10 Medical Observation Equipment 20 Information Processing Device 101 Control Unit 103 Observation Unit 105 Light Source Unit 107 Imaging Unit 109 Display Unit 201 Information Acquisition Section 203 Image Recommendation Processing Section 205 Output Processing Section 207 Display Control Section 209 Storage Section 211 Situation identification unit 213 Recommendation unit 215 Recommended image generation unit

Claims (27)

a situation identification unit that identifies an imaging situation, which is a situation when an observation target including living tissue is being imaged by an imaging device;
At least one biological image for each wavelength band generated by the imaging device detecting light from the biological tissue for each wavelength band based on the identified imaging situation, or the wavelength a recommendation unit that determines at least one of the images generated from the biological images for each band as a recommended image according to the imaging situation;
with
The situation identifying unit identifies the imaging situation based on secondary information generated by performing image processing or information processing on the biometric image ,
In addition to the secondary information, the situation identification unit refers to a situation identification database in which image information related to organs existing inside the biological tissue and medical instruments used in medical practice are recorded, Identifying the imaging situation when the interior is being observed,
The situation identifying unit generates, as the secondary information, information related to a recognition result obtained by performing image recognition processing on the biometric image using the image information,
Identifying the imaging situation when the inside of the living tissue is being observed using information about the recognition result;
Information processing system. a situation identification unit that identifies an imaging situation, which is a situation when an observation target including living tissue is being imaged by an imaging device;
At least one biological image for each wavelength band generated by the imaging device detecting light from the biological tissue for each wavelength band based on the identified imaging situation, or the wavelength a recommendation unit that determines at least one of the images generated from the biological images for each band as a recommended image according to the imaging situation;
with
The situation identifying unit identifies the imaging situation based on secondary information generated by performing image processing or information processing on the biometric image ,
The biological image includes at least an image of the biological tissue captured in a visible light band and an image of the biological tissue captured in a near-infrared band,
When the recommendation unit obtains at least one of information indicating that smoke or mist has been detected as the imaging situation, or information regarding the use of a medical energy device, the biotissue in the near-infrared band determining the captured image of as the recommended image;
Information processing system. a situation identification unit that identifies an imaging situation, which is a situation when an observation target including living tissue is being imaged by an imaging device;
At least one biological image for each wavelength band generated by the imaging device detecting light from the biological tissue for each wavelength band based on the identified imaging situation, or the wavelength a recommendation unit that determines at least one of the images generated from the biological images for each band as a recommended image according to the imaging situation;
with
The situation identifying unit identifies the imaging situation based on secondary information generated by performing image processing or information processing on the biometric image ,
The recommendation unit visualizes the oxygen saturation distribution generated as the secondary information when information indicating that a blood flow blocking medical device or a blood vessel anastomosis medical device has been detected is acquired as the imaging situation. Alternatively, an image of the biological tissue captured in a near-infrared band corresponding to blood distribution is determined as the recommended image.
Information processing system. a situation identification unit that identifies an imaging situation, which is a situation when an observation target including living tissue is being imaged by an imaging device;
At least one biological image for each wavelength band generated by the imaging device detecting light from the biological tissue for each wavelength band based on the identified imaging situation, or the wavelength a recommendation unit that determines at least one of the images generated from the biological images for each band as a recommended image according to the imaging situation;
with
The situation identifying unit identifies the imaging situation based on secondary information generated by performing image processing or information processing on the biometric image ,
The recommendation unit determines, as the recommended image, a captured image of the biological tissue in a wavelength band suitable for visual recognition of the blood vessel when information indicating that the presence of a blood vessel has been detected is acquired as the imaging situation.
Information processing system. the biological image for each wavelength band includes at least a captured image of the biological tissue in a visible light band;
further comprising a display control unit that performs display control of a display screen on which the biological image is displayed;
The display control unit causes the display screen to display the picked-up image of the biological tissue in the visible light band, and changes the display content of the display screen to the recommended image when the user selects display of the recommended image. switch to image
The information processing system according to any one of claims 1 to 4 . The recommendation unit refers to a recommended image database in which an imaging situation when the inside of the living tissue is observed and a type of the recommended image that can be presented for each imaging situation are associated in advance with each other, and the imaging is performed. 5. The information processing system according to any one of claims 1 to 4 , wherein the recommended image is determined according to the situation. 3. The method further comprises a recommended image generation unit that generates the recommended image using at least one of an image of at least part of the biometric image for each wavelength band and an image obtained by visualizing the secondary information. 5. The information processing system according to any one of 1 to 4 . The biological image for each wavelength band includes at least a captured image of the biological tissue in a visible light band and a captured image of the biological tissue in a near-infrared band,
The recommended image generation unit
changing the color tone of a region in which light in the near-infrared band is detected in the captured image of the living tissue in the near-infrared band to a predetermined color tone, and a region in which the light in the near-infrared band is not detected; After changing the luminance value of to zero, the superimposed image obtained by superimposing the imaged image of the biological tissue in the near-infrared band after the change on the imaged image of the biological tissue in the visible light band is the above Generate as a recommended image,
The information processing system according to claim 7 . the biological image for each wavelength band includes at least a captured image of the biological tissue in a visible light band;
The recommendation unit displays information about the recommended image on a display screen on which the captured image of the biological tissue in the visible light band is displayed, and prompts the user to select whether to display the recommended image.
The information processing system according to any one of claims 1 to 4 . 10. The information processing system according to claim 9 , wherein the information about said recommended image is at least one of text information indicating a type of said recommended image and a thumbnail image of said recommended image. 5. The method according to any one of claims 1 to 4 , wherein the recommendation unit displays the recommended image on a display screen different from a display screen on which the captured image of the living tissue in the visible light band is displayed. Information processing system as described. In addition to the secondary information, the situation identification unit provides: setting information regarding settings of the imaging device; characteristic information regarding characteristics of the imaging device; and specifying the imaging situation when the inside of the biological tissue is being observed based on at least one of the biological images .
The information processing system according to any one of claims 1 to 4 . In addition to the secondary information, the situation identification unit provides: setting information regarding settings of the imaging device; characteristic information regarding characteristics of the imaging device; and, based on at least one of the biomedical images, specifying the imaging situation when the inside of the biomedical tissue is being observed,
In addition to the secondary information, the situation identification unit provides: setting information regarding settings of the imaging device; characteristic information regarding characteristics of the imaging device; and referring to the situation identification database in which at least one of the biometric images and the situation in which the biometric image is generated are further associated with each other in advance, and observing the inside of the biomedical tissue. The information processing system according to claim 1 , wherein the imaging situation is specified. In addition to the secondary information, the situation identification unit provides: setting information regarding settings of the imaging device; characteristic information regarding characteristics of the imaging device; and, based on at least one of the biomedical images, specifying the imaging situation when the inside of the biomedical tissue is being observed,
In addition to the secondary information, the situation identification unit provides: setting information regarding settings of the imaging device; characteristic information regarding characteristics of the imaging device; and referring to a situation identification database in which at least one of the biometric images and a situation when the biometric image is generated are associated with each other in advance, and the imaging situation when the inside of the biomedical tissue is being observed. identify the
The situation identification database further records information on the spectral reflectance or absorption coefficient of organs and biological substances existing inside the biological tissue,
The situation identification unit generates information about spectral reflectance as the secondary information, or identifies an absorption coefficient in a specific wavelength band by referring to the biological image, and identifies the absorption coefficient existing inside the biological tissue. Identifying the imaging situation when the inside of the biological tissue is being observed by comparing with information about the spectral reflectance or extinction coefficient of the organ and biological material;
The information processing system according to any one of claims 2 to 4 . In addition to the secondary information, the situation identification unit provides: setting information regarding settings of the imaging device; characteristic information regarding characteristics of the imaging device; and, based on at least one of the biomedical images, specifying the imaging situation when the inside of the biomedical tissue is being observed,
In addition to the secondary information, the situation identification unit provides: setting information regarding settings of the imaging device; characteristic information regarding characteristics of the imaging device; and referring to a situation identification database in which at least one of the biometric images and a situation when the biometric image is generated are associated with each other in advance, and the imaging situation when the inside of the biomedical tissue is being observed. identify the
The situation identification database further records image information related to organs existing inside the living tissue and medical instruments used in medical practice,
The situation identifying unit generates, as the secondary information, information related to a recognition result obtained by performing image recognition processing on the biometric image using the image information,
Identifying the imaging situation when the inside of the living tissue is being observed using information about the recognition result;
The information processing system according to any one of claims 2 to 4 . The biological image includes at least an image of the biological tissue captured in a visible light band and an image of the biological tissue captured in a near-infrared band,
When the recommendation unit acquires information indicating that blood has been detected as the imaging situation, the recommendation unit determines an image of the biological tissue captured in the near-infrared band as the recommended image.
The information processing system according to any one of claims 1 to 4 . The biological image includes at least an image of the biological tissue captured in a visible light band and an image of the biological tissue captured in a near-infrared band,
When the recommendation unit obtains at least one of information indicating that smoke or mist has been detected as the imaging situation, or information regarding the use of a medical energy device, the biotissue in the near-infrared band determining the captured image of as the recommended image;
The information processing system according to claim 1, 3 or 4 . When the recommendation unit acquires at least one of information indicating that smoke or mist is no longer detected or information indicating that use of the medical energy device has ended as the imaging situation, the recommendation unit displays the information on the display screen. 18. The information processing system according to claim 17 , wherein the image to be captured is switched from the recommended image to the captured image of the biological tissue in the visible light band. The recommendation unit visualizes the oxygen saturation distribution generated as the secondary information when information indicating that a blood flow blocking medical device or a blood vessel anastomosis medical device has been detected is acquired as the imaging situation. 5. The information processing system according to claim 1, 2, or 4 , wherein an image of said biological tissue captured in a near-infrared band corresponding to distribution of blood is determined as said recommended image. The recommendation unit determines, as the recommended image, a captured image of the biological tissue in a wavelength band suitable for visual recognition of the blood vessel when information indicating that the presence of a blood vessel has been detected is acquired as the imaging situation. The information processing system according to any one of claims 1 to 3 . The imaging device has a multispectral sensor that includes at least pixels that detect the intensity of light in a plurality of wavelength bands separately for each wavelength band at once, and from the output result of the multispectral sensor, the wavelength bands 5. The information processing system according to any one of claims 1 to 4 , which generates a biometric image for each individual. 5. The imaging device according to any one of claims 1 to 4, wherein the imaging device has a plurality of imaging elements corresponding to specific wavelength bands, and generates a biological image for each wavelength band from the output result of each of the imaging elements. The information processing system according to item 1 . 5. The imaging device according to any one of claims 1 to 4 , wherein the imaging device performs imaging while switching wavelength bands of light formed on an imaging device of the imaging device, thereby generating a biological image for each wavelength band. The information processing system according to . The information processing system according to any one of claims 1 to 4, wherein the imaging device is an imaging device provided in an endoscope or a microscope. The information processing system according to claim 24 , wherein said endoscope or said microscope is a medical endoscope or medical microscope. An information processing method executed in an information processing system comprising a situation identifying unit and a recommending unit,
The situation identifying unit identifies an imaging situation that is a situation when an observation target including living tissue is being imaged by an imaging device,
At least one biological image for each wavelength band generated by the imaging device detecting light from the biological tissue for each wavelength band, based on the specified imaging situation, by the recommendation unit or determining at least one of the images generated from the biological images for each wavelength band as the recommended image according to the imaging situation;
including
The situation identifying unit identifies the imaging situation based on secondary information generated by performing image processing or information processing on the biometric image ,
In addition to the secondary information, the situation identification unit refers to a situation identification database in which image information related to organs existing inside the biological tissue and medical instruments used in medical practice are recorded, Identifying the imaging situation when the interior is being observed,
The situation identifying unit generates, as the secondary information, information related to a recognition result obtained by performing image recognition processing on the biometric image using the image information,
Identifying the imaging situation when the inside of the living tissue is being observed using information about the recognition result;
Information processing methods. A program for operating a computer capable of mutually communicating with a medical observation device that observes an observation target including a biological tissue and images the biological tissue in a plurality of wavelength bands to generate a biological image for each wavelength band. and
a situation identification step of identifying an imaging situation, which is a situation when the observation target is being imaged by an imaging device;
At least one biological image for each wavelength band generated by the imaging device detecting light from the biological tissue for each wavelength band based on the identified imaging situation, or the wavelength a recommending step of determining at least one of the images generated from the biological images for each band as a recommended image according to the imaging situation;
causing the computer to execute
The situation identifying step causes the computer to identify the imaging situation based on secondary information generated by performing image processing or information processing on the biometric image ,
In the situation identification step, in addition to the secondary information, a situation identification database in which image information related to organs existing inside the biological tissue and medical instruments used for medical treatment is recorded is referred to. Identifying the imaging situation when the interior is being observed,
The situation identifying step generates, as the secondary information, information related to a recognition result obtained by performing image recognition processing on the biometric image using the image information,
Identifying the imaging situation when the inside of the living tissue is being observed using information about the recognition result;
program.

Images