JP2021112220A - Image processing system, image processing device, image processing method and program - Google Patents

Abstract

To provide an image processing system, image processing device, image processing method and program which allow a user to more correctly grasp a bleeding spot in a surgical area in the surgery.SOLUTION: An image processing system according to the present disclosure comprises: a light source device which has at least a white light source emitting white light and emits the white light to a surgical area being a portion of a living body in which the surgery is performed; a surgical camera which acquires a white light image of the surgical area to which the white light is emitted by the light source device; and an image processing device which specifies a bleeding position in the surgical area on the basis of the temporal change of the surgical area in the plurality of white light images by using the plurality of white light images acquired at the different time points.SELECTED DRAWING: Figure 8

Description

The present disclosure relates to an image processing system, an image processing apparatus, an image processing method and a program.

With the widespread use of surgical cameras that can be attached to endoscopes and microscopes, which are devices for observing the inside of a living body in the medical field, doctors can directly check the affected part inside the living body with the naked eye. Instead, the number of scenes in which the affected area is treated while being confirmed by images acquired using a surgical camera is increasing. In recent years, many techniques have been proposed to support surgery by performing image processing according to a purpose on an image acquired by a surgical camera.

As one of the above-mentioned surgical support techniques, for example, the following Patent Document 1 discloses a technique that makes it easy to obtain information on a three-dimensional structure and a positional relationship between objects by using depth information. There is. Patent Document 1 discloses that the thickness of a blood clot generated by bleeding is analyzed, and a position where the obtained thickness becomes a predetermined thickness or more is detected as a bleeding position.

Japanese Unexamined Patent Publication No. 2016-93210

However, the surface of the organ existing in the lower part of the blood clot is not flat and often has irregularities. Therefore, in the technique disclosed in Patent Document 1, there is a possibility that a portion other than the bleeding position is determined to be the bleeding position. Accurate identification of the bleeding position in the medical field is an important problem that may be life-threatening for the patient in some cases, and there is still room for improvement in the accurate identification of the bleeding position.

Therefore, in view of the above circumstances, the present disclosure proposes an image processing system, an image processing device, an image processing method, and a program capable of more accurately grasping the bleeding part of the surgical site at the time of surgery.

According to the present disclosure, a light source device that has at least a white light source that irradiates white light and irradiates the surgical site that is a part of a living body in which surgery is performed with the white light, and the white light source device. Using a surgical camera that acquires a white light image of the surgical site irradiated with light and a plurality of the white light images acquired at different time points, the time change of the surgical site between the plurality of white light images Based on this, an image processing system including an image processing device for identifying a bleeding position in the surgical site is provided.

Further, according to the present disclosure, a surgical camera that is a part of a living body under surgery, which is illuminated by the white light from a light source device having at least a white light source that irradiates white light. A plurality of white light images obtained by imaging at different time points are used, and a specific portion for identifying a bleeding position in the surgical site is provided based on a time change of the surgical site between the plurality of white light images. , An image processing apparatus is provided.

Further, according to the present disclosure, a surgical camera that is a part of a living body under surgery, which is illuminated by the white light from a light source device having at least a white light source that irradiates white light. An image including identifying a bleeding position in the surgical site based on a time change of the surgical site between the plurality of white light images using a plurality of white light images obtained by imaging at different time points. A processing method is provided.

Further, according to the present disclosure, a light source device that has at least a white light source that irradiates white light and irradiates the surgical site that is a part of a living body in which surgery is performed with the white light, and the light source device. A computer capable of communicating with both of the surgical cameras that acquire the white light image of the surgical site irradiated with the white light by the surgical camera, and a plurality of the white light images acquired by the surgical camera at different times. Is provided to provide a program for functioning as a specific part for identifying a bleeding position in the surgical part based on a time change of the surgical part between the plurality of images.

According to the present disclosure, a surgical camera acquires white light images of a surgical site at different time points, and an image processing device is based on a time change of the surgical site between a plurality of white light images acquired at different time points. , Identify the location of bleeding in the surgical site.

As described above, according to the present disclosure, it is possible to more accurately grasp the bleeding site of the surgical site at the time of surgery.

It should be noted that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with the above effects or in place of the above effects. The effect of may be achieved.

It is a figure which shows an example of the endoscopic surgery system using the technical idea which concerns on this disclosure. It is a block diagram which shows the structure of the image processing system which concerns on embodiment of this disclosure. It is explanatory drawing for demonstrating the feature point extracted from the image taken by a camera. It is a schematic diagram for demonstrating the method of predicting the appearance position of the feature point which concerns on this embodiment. It is explanatory drawing for demonstrating the method of acquiring an image by the surgical camera which estimated the position and the posture which concerns on this embodiment. It is explanatory drawing for demonstrating the method of acquiring an image by the surgical camera which estimated the position and the posture which concerns on this embodiment. It is explanatory drawing for demonstrating an example of the method of detecting blood which concerns on this embodiment. It is a schematic diagram which shows an example of the feature amount plane created for detecting the blood which concerns on the same embodiment. It is a schematic diagram which shows the image which displayed the distribution of blood which concerns on the same embodiment. It is explanatory drawing for demonstrating an example of the method of determining the bleeding which concerns on the said embodiment. It is explanatory drawing for demonstrating an example of the method of specifying the bleeding position which concerns on this embodiment. It is explanatory drawing for demonstrating an example of the method of calculating the bleeding amount which concerns on the same embodiment. It is explanatory drawing for demonstrating an example of the method of calculating the bleeding amount which concerns on the same embodiment. It is explanatory drawing for demonstrating an example of the method of calculating the bleeding amount which concerns on the same embodiment. It is explanatory drawing for demonstrating an example of the method of calculating the bleeding amount which concerns on the same embodiment. It is a schematic diagram which shows the image which displayed the distribution of blood obtained by using the white light image which concerns on the same embodiment. It is a schematic diagram which shows the image which displayed the distribution of blood obtained by using the near-infrared light image which concerns on the same embodiment. It is explanatory drawing for demonstrating an example of the output method which concerns on this embodiment. It is explanatory drawing for demonstrating an example of the output method which concerns on this embodiment. It is a figure which shows the hardware configuration example of the control system which concerns on this embodiment.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

Further, in the present specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished by adding different alphabets after the same reference numerals. However, if it is not necessary to distinguish each of the plurality of components having substantially the same functional configuration, only the same reference numerals are given.

The explanations will be given in the following order.
<1. Configuration>
<1-1. Configuration of endoscopic surgery system 1>
<1-2. Configuration of image processing system 10>
<2. Operation>
<3. Hardware configuration>
<4. Conclusion>

<1. Configuration>
<1-1. Configuration of endoscopic surgery system 1>
First, a schematic configuration of an endoscopic surgery system 1 to which the technique according to the present disclosure can be applied will be described. FIG. 1 is a diagram showing an example of a schematic configuration of an endoscopic surgery system 1 to which the technique according to the present disclosure can be applied. The endoscopic surgery system 1 is used for endoscopic surgery performed in place of open surgery in the medical field.

In endoscopic surgery, instead of cutting and opening the abdominal wall, a laparotomy device called a trocca 200a-200d is attached to the abdominal wall of the affected area 230. The surgical camera 110, the energy treatment tool 210 and the forceps 220 are inserted into the body cavity of the patient from the troccers 200a to 200d. While viewing the image of the affected area 230 captured by the surgical camera 110 in real time, a procedure such as excising the affected area 230 is performed by an energy treatment tool 210 or the like. Although not shown, the surgical camera 110, the energy treatment tool 210, and the forceps 220 are supported by an operator, an assistant, a scopist, a robot, or the like during the operation.

In the operating room where endoscopic surgery is performed, a cart 190 equipped with devices for endoscopic surgery, a bed 240 on which a patient lies, and a foot switch 250 are arranged. The cart 190 includes, for example, a camera control unit (Camera Control Unit: CCU) 100, a light source device 130, an output device 140, a treatment tool control device 150, a pneumoperitoneum device 160, a recorder 170, and a printer 180 as medical devices. It is location.

(Surgery camera 110)
The surgical camera 110 has a function of acquiring an image of the surgical site where surgery is being performed. The surgical camera 110 acquires an image according to the type of light emitted to the surgical site from the light source device 130 described later. The surgical camera 110 is, for example, a white light image which is an image of a surgical part irradiated with white light, or a near infrared image which is an image of a surgical part irradiated with near-infrared light having a wavelength belonging to the near-infrared light band. An infrared light image, a laser light image obtained by irradiating the surgical site with a laser light having a predetermined wavelength, or the like is acquired. The white light image and the near-infrared light image may be captured by the surgical camera 110 provided with a plurality of imaging elements and different imaging elements for the white light image and the near-infrared light image, or the light source device 130. The light source may be switched to capture a white light image and a near-infrared light image with a single imager. The image signal of the affected area 230 captured by the surgical camera 110 is transmitted to the CCU 100 as RAW data via the camera cable. The acquired image may be stored in the recorder 170 or the storage unit 1250 of the CCU 100 as shown in FIG. The surgical camera 110 and the CCU 100 may be connected by a wireless communication method in addition to the camera cable. As the surgical camera 110, for example, an endoscope or a surgical microscope may be used. In the illustrated example, the surgical camera 110 configured as a so-called rigid mirror having a rigid lens barrel is illustrated, but the surgical camera 110 is configured as a so-called flexible mirror having a flexible lens barrel. May be good. As the camera cable, a wired transmission cable such as an electric signal cable compatible with electric signal communication, an optical fiber compatible with optical communication, or a composite cable thereof may be used.

A light source device 130 is connected to the surgical camera 110, and the light generated by the light source device 130 is transmitted to the mirror by a light guide extending inside a lens barrel provided at the tip of the surgical camera 110. The light is guided to the tip of the cylinder and irradiated toward the observation target in the patient's body cavity through the objective lens.

(CCU100)
The CCU 100 is composed of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like, and controls the operations of the surgical camera 110 and the output device 140 in an integrated manner. Specifically, the CCU 100 performs various image processing for displaying an image based on the image signal, such as development processing (demosaic processing), on the image signal received from the surgical camera 110. The CCU 100 provides the image signal subjected to the image processing to the output device 140. Further, the CCU 100 transmits a control signal to the surgical camera 110 and controls the driving thereof. The control signal may include information about imaging conditions such as magnification and focal length.

(Light source device 130)
The light source device 130 has a white light source that irradiates white light. The light source device 130 may be configured to be able to supply light in a predetermined wavelength band corresponding to special light observation, if necessary. In addition to the white light source, the light source device 130 is, for example, at least one of a near-infrared light source that irradiates near-infrared light having a wavelength belonging to the near-infrared light band and a laser light source that irradiates laser light having a predetermined wavelength. It is preferable to have a laser. Further, the light source device 130 may have a plurality of near-infrared light sources that irradiate near-infrared light having different wavelengths. The light source device 130 supplies the irradiation light for photographing the surgical portion to the surgical camera 110. In the special light observation, as will be described later, the bleeding position of the surgical site may be specified, the amount of bleeding may be calculated, and the like. Further, in special steel observation, for example, by utilizing the wavelength dependence of light absorption in body tissue, light in a narrow band is irradiated as compared with the irradiation light (that is, white light) in normal observation. So-called narrow band imaging (NBI), in which a predetermined tissue such as a blood vessel on the surface layer of the mucous membrane is photographed with high contrast, may be performed. Alternatively, in the special light observation, fluorescence observation may be performed in which an image is obtained by fluorescence generated by irradiating with excitation light. In fluorescence observation, the body tissue is irradiated with excitation light to observe the fluorescence from the body tissue (autofluorescence observation), or a reagent such as indocyanine green (ICG) is locally injected into the body tissue and the body tissue is injected. An excitation light corresponding to the fluorescence wavelength of the reagent may be irradiated to obtain a fluorescence image. The light source device 130 may be configured to be capable of supplying narrow band light and / or excitation light corresponding to such special light observation.

(Output device 140)
The output device 140 displays an image based on an image signal that has been image-processed by the CCU 100 under the control of the CCU 100. Further, the output device emits a predetermined sound under the control of the CCU 100.

(Treatment tool control device 150)
The treatment tool control device 150 controls the drive of the energy treatment tool 210 for cauterizing, incising, sealing a blood vessel, or the like of a tissue. The treatment tool control device 150 is, for example, a high-frequency output device that outputs a high-frequency current to the energy treatment tool 210 that cuts the affected portion 230.

(Pneumoperitoneum device 160)
The pneumoperitoneum device 160 is a device provided with air supply and inspiration means, and air is supplied to the patient's body, for example, the abdominal region to inflate the patient's body cavity. By expanding the body cavity of the patient by the pneumoperitoneum device 160, it is possible to secure the field of view and the working space of the operator by the surgical camera 110.

(Recorder 170)
The recorder 170 is a device capable of recording various information related to surgery. The recorder 170 may record an image acquired by the surgical camera 110, and various information acquired by the feature point detection unit 1210 and the feature amount detection unit 1220 of the CCU 100, which will be described later, are recorded as history information. You may. Further, the recorder 170 may appropriately record various parameters that need to be saved when the image processing system 10 according to the present embodiment performs some processing, the progress of the processing, and the like.

(Printer 180)
The printer 180 is a device capable of printing various information related to surgery in various formats such as texts, images, and graphs.

(Foot switch 250)
The foot switch 250 controls the CCU 100 and the treatment tool control device 150 by using an operation of an operator, an assistant, or the like as a trigger signal. The control of the CCU 100 and the treatment tool control device 150 is not limited to the foot switch 250, and may be performed by a touch panel, voice input, or the like.

The schematic configuration of the endoscopic surgery system 1 to which the technique according to the present disclosure can be applied has been described above. The technique according to the present disclosure can be applied not only to the endoscopic surgery system as described above, but also to the microscopic surgery system used for surgery using a microscope.

<1-2. Configuration of image processing system 10>
Next, a configuration example of the image processing system 10 according to the embodiment of the present disclosure will be described. FIG. 2 is a diagram showing a configuration example of the image processing system 10 according to the embodiment of the present disclosure. The image processing system 10 according to the embodiment of the present disclosure can be implemented as one function of, for example, the endoscopic surgery system 1 as shown in FIG. In the following, a detailed description will be given by taking as an example a case where the image processing system 10 described in detail below is mounted on the endoscopic surgery system 1. As shown in FIG. 2, the image processing system 10 according to the embodiment of the present disclosure includes an image processing device 120. The image processing device 120 is mounted on the CCU 100 described above, for example. In this case, the image processing device 120 according to the embodiment of the present disclosure cooperates with the surgical camera 110, the light source device 130, and the output device 140 mounted on the endoscopic surgery system 1 while coordinating with each other. Perform various processes. Further, the image processing device 120 according to the embodiment of the present disclosure is not the surgical camera 110, the light source device 130, and the output device 140 in the endoscopic surgery system 1 as shown in FIG. 1, but these surgical cameras. It is also possible to carry out various processes while coordinating with those having the same functions as the 110, the light source device 130 and the output device 140.

The image processing device 120 according to the embodiment of the present disclosure is a part of a living body under surgery, illuminated by a light source device having at least a white light source, acquired by a surgical camera 110 at different times. Using a plurality of images of the surgical site, the bleeding position in the surgical site is specified based on the time change of the surgical site between the plurality of white light images. Further, the image processing device 120 further uses at least one of an image other than the white light image (for example, a near-infrared light image or a laser light image) acquired at different time points by the surgical camera 110 in the surgical site. It is also possible to specify the bleeding position and the like. In the following, a plurality of images of various types (white light image, near infrared light image, laser light image, etc.) acquired at different time points by the surgical camera 110. The image processing apparatus 120 according to the embodiment of the present disclosure includes a feature point detection unit 1210, a feature amount detection unit 1220, and a feature amount detection unit 1220, as shown in FIG. It includes an output control unit 1230, a light source switching unit 1240, and a storage unit 1250.

(Feature point detection unit 1210)
The feature point detection unit 1210 according to the embodiment of the present disclosure uses a plurality of white light images acquired by the surgical camera 110 and analyzes each of the plurality of white light images to characterize the white light image. A certain "feature point" is detected from each of a plurality of white light images. As shown in FIG. 2, the feature point detection unit 1210 according to the embodiment of the present disclosure includes a three-dimensional information acquisition unit 1211 and a blood detection unit 1212. The three-dimensional information acquisition unit 1211 acquires the three-dimensional information of the region displayed on the white light image acquired by the surgical camera 110. The blood detection unit 1212 detects blood that may exist in the region displayed on the white light image, and acquires blood distribution information regarding the blood distribution. The three-dimensional information refers to the position information regarding the position of the image region acquired by the surgical camera 110 and the shape information regarding the three-dimensional shape of the region displayed on the image. The feature point detection unit 1210 transmits the acquired three-dimensional information and blood position information to the feature amount detection unit 1220.

(Three-dimensional information acquisition unit 1211)
The three-dimensional information acquisition unit 1211 estimates the position and orientation of the surgical camera 110 by analyzing the white light image acquired by the surgical camera 110, and in the three-dimensional space of the part displayed on the white light image. Acquire 3D information by grasping the position of. Such estimation of the position and posture of the surgical camera 110, grasping the position of the portion displayed on the white light image, and the like can be performed by a known self-position estimation technique such as SLAM (Simultaneus Localization and Mapping). .. FIG. 3 is an explanatory diagram for explaining the feature points extracted from the image taken by the camera. For example, as shown in FIG. 3, in general images including images of surgical sites, there are various types that characterize the structures and positional relationships of various objects existing in the images. There are feature points. Therefore, the three-dimensional information acquisition unit 1211 detects a feature point (feature point FP) existing in the image, and tracks the time passage of the feature point FP to estimate the position and orientation of the camera and to estimate the posture. , It is possible to grasp the position of the part displayed on the image. Using such a method, the three-dimensional information acquisition unit 1211 can estimate the position and posture of the surgical camera 110, grasp the position of the portion displayed on the image, and the like. By using the self-position estimation technique as described above, the three-dimensional information acquisition unit 1211 is displayed on the image even when the surgical camera 110 is moved due to the body movement of the patient during the operation or the operation by the operator. It is possible to grasp the position of the area and acquire three-dimensional information.

Further, if any hypothesis about the movement of the surgical camera 110 can be obtained by applying the self-position estimation technique as described above to the technique of the present disclosure, the position of the future surgical camera 110 based on the hypothesis and the position of the future surgical camera 110 based on the hypothesis Using the posture, the positions of the feature points estimated so far can be projected on the image, and the appearance position of the feature point FP can be predicted. FIG. 4 is a schematic diagram for explaining a method of predicting the appearance position of the feature point FP. Surgical camera 110 located at the position C ₁ takes an image of the object P, when acquiring an image of an area A _1, the display area A ₁ of the acquired image, the feature point FP ₁ relates to an object P is extracted. After that, when the surgical camera 110 _{moves from the position C 1} to the position C ₂ , the surgical camera 110 extracts the _{feature point FP 2} related to the object P existing in the display area A _{2 displayed from the position C 2.} do. Further, when the surgical camera 110 _{moves from the position C 2} to the position C ₃ , the surgical camera 110 extracts the _{feature point FP 3} related to the object P existing in the display area A _{3 displayed from the position C 3.} do. Here, when it is expected to move to the position _{C 4} followed by surgical camera 110, the display position of the feature point FP ₁ in the area _{A 1,} the position of the feature point FP ₂ in the display area _{A 2,} and a display the position of the object P to be estimated from each of the position of the feature point FP ₃ in the area a _3, the positional relationship between the position C ₄ and the position C _3, based on the display position of the feature point FP ₄ estimated area a by projecting the _4, it is possible to predict the occurrence position of the feature point in the display area a _4.

The three-dimensional information acquisition unit 1211 can estimate the position and posture of the surgical camera 110 by using the above-mentioned self-position estimation technique. As a result, the three-dimensional information acquisition unit 1211 saves the estimated position at any time for a certain period of time even when the surgical camera 110 moves due to the body movement of the patient during the operation and the surgical unit is not displayed on the output device 140. By doing so, the position of the surgical site can be grasped. 5A and 5B are explanatory views for explaining a method of acquiring an image of the surgical site by the surgical camera 110 whose position and posture are estimated. For example, even if the image captured by the surgical camera 110 is deviated from the surgical site and the area of blood B that the user wants to grasp is not displayed on the output device 140, the three-dimensional information acquisition unit 1211 can use the blood B. The position of the area can be predicted. Therefore, the user can move the surgical camera 110 to a desired position and cause the output device 140 to display the blood B region. For example, as shown in FIG. 5B, the user can grasp the position of the image display region. However, a specific part can be enlarged and displayed.

(Blood detection unit 1212)
The blood detection unit 1212 is a region displayed on the white light image based on the color information constituting the white light image acquired by the surgical camera 110 and the three-dimensional information acquired by the three-dimensional information acquisition unit 1211. Detects blood in and acquires blood distribution information. Specifically, a discriminator for identifying blood existing in the white light image from the white light image is prepared in advance, and the created discriminator is applied to the white light image acquired by the surgical camera 110. Then, blood distribution information can be obtained. Here, a method of acquiring blood distribution information will be described with reference to FIGS. 6A to 6C. FIG. 6A is an explanatory diagram for explaining an example of a method for detecting blood. FIG. 6B is a schematic view showing an example of a feature plane created for detecting blood. FIG. 6C is a schematic diagram showing an image showing the distribution of blood.

First, the image processing device 120 is made to learn the blood region by using the learning image for creating the classifier. FIG. 6A is a diagram showing an example of a learning image. FIG. 6A shows an image in which blood B is present on a part of the surface of the organ as a learning image. In FIG. 6A, point P1 indicates a portion trained as blood, and point P2 indicates a portion trained to be an organ region. The discriminator is, for example, color information such as RGB of the blood region and regions other than the blood region displayed on the learning image, three-dimensional information of the blood region, and three-dimensional information of the organ region on which the organ surface is displayed. It is created based on the surface roughness calculated from and the standard deviation of the surface roughness. Using a plurality of learning images as shown in FIG. 6A, the blood region and the organ region are trained as described above, and learning result data is created. As the learning result data, for example, as shown in FIG. 6B, a feature plane having color information on the vertical axis and surface roughness on the horizontal axis is created, and the color information and surface roughness indicated by the blood vessel region are obtained. The identification surface f indicating the boundary between the feature amount region occupied by the combination and the feature amount region occupied by the combination of the color information and the surface roughness indicated by the organ region is specified. For example, in this way, a discriminator that discriminates the blood present in the image from the image may be constructed. Then, the blood detection unit 1212 may use the constructed discriminator (for example, a feature amount as shown in FIG. 6B). The identification surface f) on the plane is used to identify the location of blood in the image acquired by the surgical camera 110. Specifically, the blood detection unit 1212 divides the image acquired by the surgical camera 110 into a predetermined size, and from the three-dimensional information acquired by the three-dimensional information acquisition unit 1211, for each divided range of the image. RGB and surface roughness are calculated. After that, the blood detection unit 1212 plots the obtained combination of RGB and surface roughness in any region on the feature amount plane for each divided range based on the information regarding the identification surface f. It is determined whether or not each divided range of the image is blood. As described above, as shown in FIG. 6C, blood distribution information regarding the distribution of blood B in the image acquired by the surgical camera 110 can be acquired.

The blood detection unit 1212 can also detect the presence of a foreign substance by the above method. For example, when the surgical instrument is left in the surgical site, the color information and surface roughness of the area where the surgical instrument is displayed are significantly different from the color information and surface roughness of the blood region and the color information and surface roughness of the organ region. .. Therefore, the positions plotted in the feature plane corresponding to the area where the surgical instrument is displayed are significantly different from the positions plotted as the blood area and the organ area. Therefore, the blood detection unit 1212 can detect the presence of a foreign substance. Further, the presence of a foreign substance can be detected by using various known object recognition techniques instead of the above-mentioned method using a discriminator. For example, various surgical tools used for surgery can be detected as an object. By recognizing it, it is possible to detect the neglect of the surgical instrument.

(Feature quantity detection unit 1220)
The feature amount detection unit 1220 according to the embodiment of the present disclosure uses at least the three-dimensional information and the blood distribution information acquired from the feature point detection unit 1210 to determine the feature amount related to blood in each region displayed in the image. To detect. As shown in FIG. 2, the feature amount detecting unit 1220 according to the embodiment of the present disclosure includes a bleeding determination unit 1221, a bleeding position specifying unit 1222, and a blood volume calculating unit 1223. The bleeding determination unit 1221 determines whether or not bleeding has occurred in the area displayed in the image acquired by the surgical camera 110. The bleeding position specifying unit 1222 has a function of specifying a bleeding position in the surgical site based on a time change of the surgical site between the plurality of images using a plurality of images acquired at different time points. The blood volume calculation unit 1223 calculates the blood volume of the blood displayed in the image acquired by the surgical camera 110.

(Bleeding determination unit 1221)
The bleeding determination unit 1221 determines whether or not bleeding has occurred in the region displayed on the image based on the blood distribution information of the plurality of white light images acquired at different times. Specifically, the bleeding determination unit 1221 calculates the area of the region where blood exists and the amount of change in the area based on the blood distribution information of the white light image at each time point, and changes in the area of the region where blood exists. When the amount exceeds a predetermined threshold, it can be determined that bleeding has occurred. In addition, the bleeding determination unit 1221 can determine the presence or absence of bleeding based on the amount of change in blood volume calculated by the blood volume calculation unit 1223, which will be described later. Here, the area of the blood region can be calculated from the number of pixels of the blood region in the acquired white light image and the magnification of the displayed white light image, and the amount of change in the area is the calculated blood region. It can be calculated from the time change of the area.

Here, with reference to FIG. 7, a method for determining the presence or absence of bleeding will be specifically described. FIG. 7 is an explanatory diagram for explaining a method for determining the presence or absence of bleeding. The graph shown in the upper part of FIG. 7 is a graph showing the transition of the area of the blood region on the acquired white light image, with the vertical axis representing the area of the blood region and the horizontal axis representing the time. The graph shown in the middle of FIG. 7 is a graph showing the transition of the amount of change in the area of the blood region on the acquired white light image, with the vertical axis representing the amount of change in the area of the blood region and the horizontal axis representing the time. The lower part of FIG. 7 shows white light images acquired at _{each time t 1 to t 6.} Here, each time t _{1 to t 6} is, for example, a time when 10 frames have elapsed. As shown in the middle part of FIG. 7, the threshold value T (the threshold value T may be set in advance by a third party or is set by the user) with respect to the amount of change in the area of the blood region. When the amount of change in the area of the blood region exceeds the threshold value T, the bleeding determination unit 1221 can determine that bleeding has occurred. In Figure 7, bleeding is determined to have occurred at time t _2. The bleeding determination unit 1221 performs image processing on the white light image at each time point and calculates the difference in the area of the blood region between the images at each time point. As shown in the lower part of FIG. 8, the amount of change in the area of the blood region is the difference in the area of the blood region between the images at each time point. Therefore, when the position of the surgical camera 110 deviates from the position at the time immediately before, the blood distribution displayed in the image changes according to the deviation, so that the area of the blood region and the blood region in the predetermined surgical site There is a possibility that the amount of change in area cannot be calculated accurately. In that case, the area of the blood region can be accurately calculated by correcting the positional deviation based on the three-dimensional information acquired by the three-dimensional information acquisition unit 1211.

(Bleeding position identification part 1222)
The bleeding position specifying unit 1222 identifies the bleeding position by tracing back the transition of the blood distribution state based on the blood distribution information at each time point when a plurality of white light images are acquired. FIG. 8 is an explanatory diagram for explaining a method of identifying a bleeding position. The graph shown in the upper part of FIG. 8 is a graph showing the area of the blood region on the acquired white light image, with the vertical axis representing the area of the blood region and the horizontal axis representing the time. The middle part of FIG. 8 shows white light images acquired at _{each time t 1 to t 6.} Here, each time t _{1 to t 6} is, for example, a time when 10 frames have elapsed. The lower part of FIG. 8 is a diagram showing the difference between the white light images shown in the middle part of FIG. For example, the left view of the lower 8 shows the difference of the obtained white light image at the time of the obtained white-light image and the time t ₂ at time t _3. The second diagram from the left in FIG. 8 bottom shows the difference of the obtained white light image at the time of the white light image and the time t ₃ when being acquired at the time point t _4. Likewise, the fourth diagram from the left in the third figures and 8 in the lower left of the lower 8 white obtained at the time of the acquired white-light image and the time t ₄ at each time t ₅ shows the optical image difference, and the difference of the obtained white light image at the time of the white light image and the time t ₅ that is acquired at the time point t _6. As the area of the blood region in the region displayed on the white light image increases, the displayed blood region expands as shown in the middle figure of FIG. Since the blood region expands from the bleeding position as the starting point, the bleeding position specifying portion 1222 can identify the bleeding position by tracing back the blood region displayed in the white light image at each time point in the time direction. .. For example, as shown in the lower part of FIG. 8, it can be confirmed that bleeding at time t ₂ the difference between the blood region occurs occurs. Then, it is possible from an image showing the difference of the obtained white light image at the time of the white light image and the time t ₂ acquired at the time point t _3, to identify a bleeding position. As a result, the user can recognize the bleeding position more accurately as compared with the conventional technique.

Further, as described above, the bleeding position specifying unit 1222 can grasp the transition of the blood distribution with the passage of time. Therefore, the bleeding position specifying unit 1222 compares the blood region displayed on the white light image at the past time point with the blood region displayed on the white light image at the latest time point, and the direction and expansion of the blood region are expanded. By calculating the amount of change in the blood region, it is possible to estimate the future distribution of blood. Bleeding position specifying unit 1222, for example, as shown in the graph of FIG. 8 the upper, it can be estimated blood region area at the time of the upcoming time t _7, as shown in FIG. 8 middle right, the future it can be estimated blood distribution at time t _7. The bleeding position specifying unit 1222 may perform image processing on the white light image at each time point and calculation of the difference in the area of the blood region between the white light images at each time point.

Further, the bleeding position specifying unit 1222 bleeds using a plurality of laser light images obtained by irradiating the surgical site with a laser beam having a predetermined wavelength from the light source device 130 and irradiating the laser beam at different time points. It is also possible to specify the position. For example, the light source device 130 has a laser light source that generates a laser beam having high coherence, and when the laser beam is irradiated to blood, the laser beam is scattered by the blood. Since the scattered lights have different phases, the scattered lights randomly interfere with each other on the imaging surface of the surgical camera 110, and a spotted light intensity distribution pattern (so-called speckle pattern) is obtained. Since the pattern of the light intensity distribution changes dynamically according to the movement of the object, this pattern of the light intensity distribution can be used as blood flow information. In the bleeding position, the area becomes a region where the movement is large due to the outflow of blood. Therefore, in the speckle pattern as described above, the region closer to the bleeding position has a larger brightness value in the speckle image obtained from the speckle pattern. Should be. Therefore, the bleeding position specifying unit 1222 can calculate the bleeding position by various known methods, and can specify the bleeding position based on the distribution of the brightness value in the calculated speckle image.

Further, the bleeding position specifying unit 1222 identifies the bleeding position by using various methods as described above, and comprehensively determines the information on the bleeding position obtained from the various methods to make more accurate bleeding. It is also possible to specify the position.

(Bleeding amount calculation unit 1223)
The blood volume calculation unit 1223 calculates the volume of blood displayed in the image acquired by the surgical camera 110. A white light image, which is an image of the surgical site irradiated with white light, or a white light image and a near-infrared light image are used for calculating the blood volume. When a white light image is used, the blood volume calculation unit 1223 assumes, for example, that the thickness of the blood displayed in the white light image is constant, and the thickness of the blood and the blood region displayed in the white light image. The product with the area can be the blood volume in the indicated area.

In addition, the blood volume calculation unit 1223 can also calculate the blood volume using a near-infrared light image which is an image of the surgical site irradiated with near-infrared light. In this case, a plurality of near-infrared light images may be used for calculating the blood volume. A method for calculating the volume of blood will be described using near-infrared light images with reference to FIGS. 9A-9D, 10A and 10B. The left figure of FIG. 9A is a schematic view showing the surgical site irradiated with white light from above from the side surface. The left figure of FIG. 9B, the left figure of FIG. 9C, and the left figure of FIG. 9D are schematic views showing from the side the surgical site irradiated with near-infrared light having _{wavelength λ 1} , wavelength λ _2, and wavelength λ _{3, respectively.} .. The right figure of FIG. 9A is a schematic view showing an image of the surgical site irradiated with white light acquired by the surgical camera 110. 9B right, 9C right and 9D right are the surgical sites irradiated with near-infrared light having _{wavelength λ 1} , wavelength λ ₂ and wavelength λ _{3, respectively, acquired by the surgical camera 110.} It is a schematic diagram which showed the image of. Here, it is assumed that the relationship of λ ₁ <λ ₂ <λ _{3 is established for the wavelength λ 1} , the wavelength λ ₂ and the wavelength λ _3. The left figure of FIG. 10A is a schematic view showing a white light image, and the right figure of FIG. 10A is a schematic view showing a blood region detected by the blood detection unit 1212 using the white light image. The left figure of FIG. 10B is a schematic view showing a near-infrared light image of the same region as the image shown on the left side of FIG. 10A, and the right figure of FIG. 10B is a blood using the near-infrared light image. It is a schematic diagram which showed the blood region detected by the detection part 1212.

Since the white light WL reflects light having a wavelength corresponding to red on the surface of the substance, it is reflected on the surfaces of blood B1, blood B2, and blood B3 having different thicknesses, as shown in FIG. 9A. Therefore, blood B1, blood B2, and blood B3 are projected on the acquired white light image. On the other hand, near-infrared light is absorbed by hemoglobin in the blood. The shorter the wavelength of near-infrared light, the greater the degree of absorption by hemoglobin in blood, and the longer the wavelength of near-infrared light, the smaller the degree of absorption by hemoglobin in blood. Therefore, as shown in FIG. 9B, in the near-infrared light image near-infrared light of wavelength lambda ₁ is acquired by irradiation, the thickness of small blood B1, many of near infrared light having a wavelength lambda ₁ transmitted Blood B1 is not recognized by the surgical camera 110 because it is reflected on the surface of the organ below the blood. Further, in blood B2 and blood B3 having a thickness larger than that of blood B1, most of the near-infrared light having a _{wavelength of λ 1 is absorbed.} Therefore, in the near-infrared light image obtained by irradiating the near-infrared light having the wavelength λ ₁ , the blood B2 and the blood B3 are displayed separately from the parts other than the blood B2 and the blood B3. In the near-infrared light image near-infrared light of wavelength lambda ₂ is acquired by irradiation, as shown in FIG. 9C, the blood B1 and blood B2, many of near infrared light having a wavelength lambda ₂ passes Is reflected on the surface of the organ below the blood. Therefore, blood B1 and blood B2 are not recognized by the surgical camera 110. Further, in blood B1 and blood B3 having a thickness larger than that of blood B2, most of the near-infrared light having a _{wavelength of λ 2 is absorbed.} Therefore, in the near-infrared light image obtained by irradiating the near-infrared light having the wavelength λ _{2, the blood B3 is displayed separately from the portion other than the blood B3.} Further, in the near-infrared light image obtained by irradiating the near-infrared light of the wavelength λ ₃ , as shown in FIG. 9D, most of the near-infrared light of the _{wavelength λ 3 is transmitted even in the thick blood B3.} Then it is reflected on the surface of the organ below the blood. Therefore, blood B1, blood B2, and blood B3 are not displayed. Therefore, as shown in the left figure of FIG. 10A, both blood B1 and blood B2 are displayed in the white light image. Therefore, as shown in the right figure of FIG. 10A, the blood detection unit 1212 has both blood B1 and blood B2. Can be detected. On the other hand, as shown in the left figure of FIG. 10B, the blood B1 having a small thickness is not displayed in the near-infrared light image irradiated with the near-infrared light having _{the wavelength λ 1, and therefore, as shown in the right figure of FIG. 10B.} The blood detection unit 1212 can detect only blood B2.

As described above, the blood displayed in the near-infrared light image differs depending on the wavelength of the irradiated near-infrared light. Therefore, it is possible to associate the wavelength with the thickness of blood by taking the difference between a plurality of near-infrared light images obtained by irradiating the surgical site with near-infrared light having different wavelengths from the light source device 130. , It becomes possible to calculate the thickness of blood. As described above, the product of the calculated blood thickness and the area of the blood region having the thickness can be obtained, and the total of the products can be used as the blood volume in the displayed region.

As described above, the blood volume calculation method using the near-infrared light image can calculate the blood volume more accurately than the above-mentioned blood volume calculation method using the white light image. Then, the blood volume calculation unit 1223 can calculate the amount of change in blood volume by performing the above processing on the near-infrared light images acquired at a plurality of time points. Since white light is reflected on the surface of blood, a white light image may be used instead of the near-infrared light image obtained by irradiating with near-infrared light of a certain wavelength. In this way, the blood volume calculation unit 1223 compares the blood volume calculated in at least two or more of the plurality of near-infrared light images and the white light image, and obtains the near-infrared light image or the white light image. The amount of change in blood volume in the displayed area can be calculated.

(Output control unit 1230)
The output control unit 1230 controls the content and output method of the information output by the output device 140. For example, the output control unit 1230 outputs information such as an image acquired by the surgical camera 110, three-dimensional information of the image, a display notifying the user of the occurrence of bleeding or a voice notification, a display regarding the bleeding position, and the amount of bleeding. Output to the output device 140. The type of information output by the output control unit 1230 to the output device 140 may be set in advance by the user. Further, when the output control unit 1230 receives the image information, the three-dimensional information, the information on the bleeding position, the information on the blood volume, etc. acquired by the surgical camera 110, the output instruction may be transmitted to the output device 140. good.

Information to be output to the output device 140 by the output control unit 1230 will be described with reference to FIGS. 11 and 12. FIG. 11 is a schematic diagram showing an example of the information displayed on the output device 140. FIG. 12 is a schematic diagram showing another example of the information output to the output device 140.

For example, FIG. 11 shows a real-time image (MONITOR 1) acquired by the surgical camera 110, an image (MONITOR 2) in which a display showing a bleeding position is superimposed on the real-time image, and a moving image before and after bleeding (MONITOR 3). ), An enlarged image of the bleeding position (MONITOR 4), and a graph of the bleeding amount are displayed on the output device 140. In addition to the above, the output control unit 1230 may display a near-infrared light image and a laser light image which is an image of the surgical part irradiated with the laser light on the output device 140. Further, the output control unit 1230 may superimpose the display indicating the blood region on the white light image and display it on the output device 140. The output control unit 1230 may output a display or a warning sound to the output device 140 when the amount of bleeding exceeds a predetermined value. Further, the output control unit 1230 may display the future blood distribution estimated by the bleeding position specifying unit 1222 on the output device 140.

Since the image showing the bleeding position is superimposed on the real-time image or the moving image before and after the bleeding is displayed on the output device 140, it is difficult to find it because it is covered with thickly accumulated blood in the past. The user can easily recognize the bleeding site and can start the hemostasis work more quickly. By displaying the graph of the amount of bleeding on the output device 140, the user can confirm vital signs, determine the transition from endoscopic surgery to laparotomy, the timing of starting blood transfusion, and the like. By displaying the future blood distribution on the output device 140, the user can recognize the state of the surgical site before the blood is accumulated.

Further, when a plurality of bleeding positions are present in the displayed image, the output control unit 1230 indicates different bleeding positions depending on the degree of change in the blood region near the bleeding position or the moving speed of blood. An image in which the display is superimposed may be displayed. For example, a display indicating a bleeding position existing in the vicinity of blood having a large degree of change in the blood region or a large movement speed may be highlighted. The bleeding position near the blood where the degree of change in the blood region or the moving speed is large is often the bleeding position to be treated with priority. Therefore, the output control unit 1230 causes the output device 140 to highlight the bleeding position existing in the vicinity of the blood having a large degree of change in the blood region or a large movement speed as described above, so that the user can quickly take measures such as hemostasis. It can be carried out.

Further, as shown in FIG. 12, the output control unit 1230 determines the direction of the bleeding position when the bleeding position specified in the past deviates from the area displayed in the real-time image acquired by the surgical camera 110. The indicated display d can be superimposed and displayed on a real-time image. As a result, even if the surgical camera 110 is shaken after bleeding and the bleeding position is not displayed in the acquired image, the user can quickly recognize the bleeding position and perform treatment.

The output image, sound, and the like may be selectively output together with the real-time image by setting the output contents in advance by the user. In addition, an image in which a display showing the bleeding position is superimposed on a real-time image, a moving image before and after the bleeding, an enlarged image of the bleeding position, and the like are output control units when the bleeding determination unit 1221 determines that bleeding has occurred. The 1230 may be displayed by transmitting an output instruction to the output device 140. The bleeding-related information includes information on the occurrence of bleeding, the area of the blood area, the amount of change in the area, the amount of blood, the amount of change in blood volume, the display of the bleeding position, the moving image before and after the occurrence of bleeding, and the bleeding position. It contains all information related to bleeding at the surgical site, such as a display indicating the direction of.

(Light source switching unit 1240)
The light source switching unit 1240 switches the light emitted from the light source device 130. For example, a switching instruction is transmitted from the input device operated by the user to the light source switching unit 1240. In response to such a switching instruction, the light source device 130 switches between a white light light source, a near infrared light source, and a laser light source, and irradiates the surgical site with light desired by the user.

(Memory unit 1250)
An image acquired by the surgical camera 110 is recorded in the storage unit 1250. In the storage unit 1250, various programs, databases, and the like used when the image processing device 120 performs various processes as described above are appropriately recorded. Further, the storage unit 1250 may record various types of information acquired by the feature point detection unit 1210 and the feature amount detection unit 1220 as history information. Further, in the storage unit 1250, for example, the blood detection unit 1212, the bleeding determination unit 1221, the bleeding position identification unit 1222, and the blood volume calculation unit 1223 each need to be stored when performing their respective processes. Parameters, processing progress, etc. may be recorded as appropriate. Not limited to the processing executed by the blood detection unit 1212, the bleeding determination unit 1221, the bleeding position identification unit 1222, and the blood volume calculation unit 1223, it is necessary to save the image processing system 10 according to the present embodiment when performing any processing. Various parameters and the progress of processing may be recorded as appropriate. In the storage unit 1250, the communication unit 1260, the feature point detection unit 1210, the feature amount detection unit 1220, and the like can freely perform read / write processing. The storage unit 1250 may save and overwrite the image to save the image for a time set in advance by the user. In addition, the storage unit 1250 may store images for a time set in advance by the user before and after the time when the bleeding determination unit 1221 determines that bleeding has occurred so as not to be overwritten.

(Communication unit 1260)
The communication unit 1260 has a function of performing information communication with the surgical camera 110 via the network. Specifically, the communication unit 1260 receives an image from the surgical camera 110. Further, the communication unit 1260 has a function of transmitting a control signal to the surgical camera 110 and controlling the driving thereof. The control signal may include information about imaging conditions such as magnification and focal length.

<2. Operation>
Subsequently, an example of the operation of the image processing system 10 according to the embodiment of the present disclosure will be described in detail.

First, the light source switching unit 1240 sets the light source of the light source device 130 to a white light light source, and white light is applied to the surgical part where the operation is being performed. The surgical camera 110 acquires a white light image of the surgical site. The acquired white light image is transmitted to the feature point detection unit 1210, the output control unit 1230, and the storage unit 1250 via the communication unit 1260 of the image processing device 120. Here, the output control unit 1230 transmits the received white light image to the output device 140, and the output device 140 displays the received white light image as a real-time image. Further, the white light image transmitted to the storage unit 1250 is recorded for a time preset by the user.

Next, the three-dimensional information acquisition unit 1211 provided in the feature point detection unit 1210 acquires three-dimensional information regarding the position and shape of the region displayed in the acquired white light image. The blood detection unit 1212 provided in the feature point detection unit 1210 is based on the color information constituting the white light image acquired by the surgical camera 110 and the three-dimensional information acquired by the three-dimensional information acquisition unit 1211. Detects blood in the area displayed in and generates blood distribution information. Specifically, the image acquired by the surgical camera 110 is divided into predetermined sizes, and the surface roughness is calculated for each divided range of the image from the three-dimensional information acquired by the three-dimensional information acquisition unit 1211. .. RGB and the calculated surface roughness for each divided range of the image are plotted on a feature plane with the vertical axis representing the color information and the horizontal axis representing the surface roughness, and each of the divided images is plotted. Determine if the range of is the blood region. The three-dimensional information and the blood distribution information are transmitted to the feature amount detection unit 1220 and the storage unit 1250. The blood distribution information recorded in the storage unit 1250 is transmitted to the output control unit 1230 together with the white light image recorded in the storage unit 1250. The blood distribution information is applied to the white light image, and the display indicating the blood region is output by the output control unit 1230 as an image displayed superimposed on the white light image.

Subsequently, the blood volume calculation unit 1223 provided in the feature amount detection unit 1220 calculates the blood volume using the near-infrared light image. The light source switching unit 1240 switches the light source of the light source device 130 from a white light source to a near infrared light source. The switching of the light source is performed, for example, by operating the foot switch 250 of the user. The light source device 130 irradiates the surgical site with near-infrared light having a different wavelength. The blood volume calculation unit 1223 calculates the thickness of blood by taking the difference between a plurality of near-infrared light images obtained by irradiating the surgical site with near-infrared light having different wavelengths. The product of the calculated blood thickness and the area of the blood region having the thickness is obtained, and the total of the products is taken as the blood volume in the displayed region. The blood volume calculation unit 1223 compares the blood volumes calculated by two or more images, and calculates the amount of change in the blood volume in the region displayed on the near-infrared light image or the white light image.

The bleeding determination unit 1221 determines the occurrence of bleeding from the amount of change in blood volume calculated by the blood volume calculation unit 1223. Specifically, a threshold value is set by the user for the amount of change in blood volume, and when the amount of change in blood volume exceeds the threshold value, the bleeding determination unit 1221 determines that bleeding has occurred. In addition, the bleeding determination unit 1221 may determine the presence or absence of bleeding based on the amount of change in the area of the blood region. Specifically, the bleeding determination unit 1221 performs image processing on the white light image at each time point and calculates the difference in the area of the blood region between the images at each time point. A threshold value is preset by the user for the amount of change in the area of the blood region, and when the amount of change in the area of the blood region exceeds the threshold value, the bleeding determination unit 1221 may determine that bleeding has occurred.

Subsequently, the bleeding position specifying unit 1222 reads out a plurality of white light images stored in the storage unit 1250 and blood distribution information at each time point when the white light images are acquired. Using the read white light image and blood distribution information, the transition of the blood distribution state is grasped. The state of this blood distribution is traced back, and the position where blood is generated is used to identify the bleeding position.

Information such as the blood volume calculated by the blood volume calculation unit, the time when bleeding occurs, and the specified bleeding position is transmitted to at least one of the output control unit 1230 or the storage unit 1250.

Then, the output control unit 1230 appropriately displays a real-time image transmitted from the surgical camera 110, a bleeding amount transmitted from the blood volume calculation unit, an image displaying the bleeding position specified by the bleeding position specifying unit, and the like. Output to the output device 140. The image showing the bleeding position is read out from the storage unit 1250. The above process is executed at any time.

<3. Hardware configuration>
The embodiments according to the present disclosure have been described above. The above-mentioned image processing is realized by the collaboration between the software and the hardware of the information processing apparatus described below.

Next, a hardware configuration example of the image processing device 120 according to the embodiment of the present disclosure will be described. FIG. 13 is a block diagram showing a hardware configuration example of the image processing device 120 according to the embodiment of the present disclosure. Referring to FIG. 13, the image processing apparatus 120 includes, for example, a CPU 1270, a ROM 1271, a RAM 1272, a host bus 1273, a bridge 1274, an external bus 1275, an interface 1276, an input device 1277, and a display device 1278. It has an audio output device 1279, a storage device 1280, a drive 1281, a connection port 1282, and a removable storage medium 1283. The hardware configuration shown here is an example, and some of the components may be omitted. Further, components other than the components shown here may be further included.

(CPU1270)
The CPU 1270 functions as, for example, an arithmetic processing device or a control device, and controls all or a part of the operation of each component based on various programs recorded in the ROM 1271, the RAM 1272, the storage device 1280, or the removable storage medium 1283. ..

(ROM1271, RAM1272)
The ROM 1271 is a means for storing a program read into the CPU 1270, data used for calculation, and the like. The RAM 1272 temporarily or permanently stores, for example, a program read into the CPU 1270, various parameters that change as appropriate when the program is executed, and the like.

(Host bus 1273, bridge 1274, external bus 1275, interface 1276)
The CPU 1270, ROM 1271, and RAM 1272 are connected to each other via, for example, a host bus 1273 capable of high-speed data transmission. On the other hand, the host bus 1273 is connected to the external bus 876, which has a relatively low data transmission speed, via, for example, a bridge 1274. Further, the external bus 1275 is connected to various components via the interface 1276.

(Input device 1277)
For the input device 1277, for example, a mouse, a keyboard, a touch panel, buttons, switches, levers, and the like can be used. Further, as the input device 1277, a remote controller capable of transmitting a control signal using infrared rays or other radio waves may be used. Further, the input device 1277 includes a voice input device such as a microphone. In the endoscopic surgery system 1 to which the technique of the present disclosure can be applied, it is preferable that the foot switch 250 is used as the input device 1277.

(Display device 1278, audio output device 1279)
The display device 1278 is, for example, a display device such as a CRT (Cathode Ray Tube), an LCD, or an organic EL, a printer, or the like, and the audio output device 1279 is an audio output device such as a speaker or a headphone. The display device 1278 and the voice output device 1279 are both devices capable of visually or audibly notifying the user of the acquired information.

(Storage device 1280)
The storage device 1280 is a device for storing various types of data. As the storage device 1280, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like is used.

(Drive 1281)
The drive 1281 is a device that reads information recorded on a removable storage medium 1283 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable storage medium 1283.

(Removable storage medium 1283)
The removable storage medium 1283 is, for example, a DVD media, a Blu-ray (registered trademark) media, an HD DVD media, various semiconductor storage media, and the like. Of course, the removable storage medium 1283 may be, for example, an IC card equipped with a non-contact type IC chip, an electronic device, or the like.

(Connection port 1282)
The connection port 1282 is a port for connecting an external connection device 902 such as a USB (Universal Serial Bus) port, an IEEE1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or an optical audio terminal. be.

<4. Conclusion>
Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can come up with various modifications or modifications within the scope of the technical ideas described in the claims. Of course, it is understood that the above also belongs to the technical scope of the present disclosure.

According to the technique of the present disclosure, it is possible to more accurately grasp the bleeding site of the surgical site at the time of surgery. In addition, it is possible to estimate a stable bleeding position even when the patient moves. In addition, the user can recognize the transition of the bleeding amount. As a result, the user can start the hemostasis work quickly, the operation time can be shortened, and the amount of bleeding can be suppressed. In addition, since the user can recognize the amount of bleeding, it is possible to confirm vital signs, determine the transition from endoscopic surgery to laparotomy, and determine the timing of starting blood transfusion. As a result, it is possible to accelerate the postoperative recovery of the patient.

In addition, the effects described herein are merely explanatory or exemplary and are not limited. That is, the technique according to the present disclosure may exhibit other effects apparent to those skilled in the art from the description of the present specification, in addition to or in place of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1)
A light source device that has at least a white light source that irradiates white light and irradiates the surgical site that is a part of the living body in which surgery is performed with the white light.
A surgical camera that acquires a white light image of the surgical site irradiated with the white light by the light source device, and
An image processing device that identifies a bleeding position in the surgical site based on a time change of the surgical site between the plurality of white light images using a plurality of the white light images acquired at different time points.
To prepare
Image processing system.
(2)
The image processing device acquires three-dimensional information of the region displayed on the white light image, and obtains three-dimensional information.
Based on the color information and the three-dimensional information constituting the white light image, blood existing in the region displayed on the white light image is detected, and blood distribution information regarding the distribution of the blood is acquired.
The bleeding position is specified by tracing back the transition of the state of the blood distribution based on the blood distribution information at each time point when the plurality of white light images are acquired.
The image processing system according to (1).
(3)
The image processing device is based on the blood distribution information at each time point, the amount of change in the area of the region where the blood is present, the amount of blood existing in the region displayed in the white light image, and the change in the blood volume. Calculate at least one of the quantities,
The image processing system according to (1) or (2).
(4)
The image processing device determines that bleeding has occurred in the region displayed on the white light image when the amount of change in the area exceeds a predetermined threshold value.
The image processing system according to any one of (1) to (3).
(5)
The image processing device determines that bleeding has occurred in the region displayed on the white light image when the amount of change in blood volume exceeds a predetermined threshold value.
The image processing system according to any one of (1) to (3).
(6)
The image processing device estimates the future distribution of the blood based on the transition of the distribution of the blood at each of the plurality of time points. The light source device has a wavelength belonging to the near-infrared light band. Equipped with a near-infrared light source that irradiates light
The surgical camera further acquires a plurality of near-infrared light images in which the near-infrared light irradiates the surgical site.
The image processing apparatus is based on at least two or more of the plurality of near-infrared light images and the white light image, and the blood in the near-infrared light image or the region displayed in the white light image. Calculate the amount and the amount of change in the blood volume,
The image processing system according to (5).
(7)
The image processing apparatus estimates the future distribution of the blood based on the transition of the distribution of the blood at a plurality of different time points.
The image processing system according to any one of (1) to (3).
(8)
The image processing device detects the presence of foreign matter existing in the region displayed on the white light image based on the three-dimensional information.
The image processing system according to (1) or (2).
(9)
The light source device includes a laser light source that irradiates a laser beam having a predetermined wavelength.
The surgical camera further acquires a laser beam image of the surgical site irradiated with the laser beam.
The image processing apparatus acquires blood flow information regarding blood flow using the previously acquired laser beam image, and identifies the bleeding position based on the blood flow information.
The image processing system according to (1).
(10)
The image processing device outputs bleeding-related information regarding the bleeding when it is determined that bleeding has occurred in the region displayed on the white light image.
The image processing system according to (1).
(11)
The image processing device outputs the specified bleeding position by a predetermined method.
The image processing system according to (1).
(12)
The image processing device outputs the white light image from the starting point to the elapse of a predetermined time, starting from a time point retroactive to a predetermined time from the time when the white light image determined to have caused bleeding has been acquired. do,
The image processing system according to (1).
(13)
The image processing device outputs a predetermined signal for urging the occurrence of bleeding when at least one of the blood volume and the change amount of the blood volume exceeds a predetermined threshold value.
The image processing system according to any one of (1) to (3).
(14)
The image processing device superimposes and displays a display indicating the bleeding position on the latest white light image at the time point.
The image processing system according to (1).
(15)
The image processing device changes the display indicating the bleeding position in the region displayed on the white light image according to the amount of change in the blood volume or the amount of change in the area of the region where the blood is present.
The image processing system according to any one of (1) to (3).
(16)
The image processing device outputs a display indicating the direction of the bleeding position when the bleeding position deviates from the area displayed in the white light image acquired by the surgical camera.
The image processing system according to (1).
(17)
The image processing device outputs a change in the amount of bleeding in the region displayed in the white light image.
The image processing system according to (1).
(18)
The image processing apparatus outputs an estimated future distribution of the blood based on the transition of the distribution of the blood at a plurality of different time points.
The image processing system according to (1) or (2).
(19)
By imaging the surgical site, which is a part of the living body under surgery, illuminated by the white light from a light source device having at least a white light source that irradiates white light, at different time points with a surgical camera. Using the obtained plurality of white light images, a specific portion for identifying a bleeding position in the surgical site based on a time change of the surgical site between the plurality of white light images is provided.
Image processing device.
(20)
By imaging the surgical site, which is a part of the living body under surgery, illuminated by the white light from a light source device having at least a white light source that irradiates white light, at different time points with a surgical camera. An image processing method including identifying a bleeding position in the surgical site based on a time change of the surgical site between the plurality of white light images using the obtained plurality of white light images.
(21)
A light source device that has at least a white light source that irradiates white light and irradiates the surgical site that is a part of the living body undergoing surgery with the white light, and the light source device that irradiates the white light. A computer capable of communicating with both of the surgical cameras that acquire the white light image of the surgical site.
Using the plurality of white light images acquired by the surgical camera at different times, it functions as a specific part for identifying the bleeding position in the surgical part based on the time change of the surgical part between the plurality of images. To make you
program.

1 Endoscopic surgery system 10 Image processing system 100 CCU
110 Surgical camera 120 Image processing device 130 Light source device 140 Output device 1210 Feature point detection unit 1211 Three-dimensional information acquisition unit 1212 Blood detection unit 1220 Feature amount detection unit 1221 Bleeding determination unit 1222 Bleeding position identification unit 1223 Blood volume calculation unit 1230 Output Control unit 1240 Light source switching unit 1250 Storage unit 1260 Communication unit

Claims (21)

A light source device that has at least a white light source that irradiates white light and irradiates the surgical site that is a part of the living body in which surgery is performed with the white light.
A surgical camera that acquires a white light image of the surgical site irradiated with the white light by the light source device, and
An image processing device that identifies a bleeding position in the surgical site based on a time change of the surgical site between the plurality of white light images using a plurality of the white light images acquired at different time points.
To prepare
Image processing system. The image processing device acquires three-dimensional information of the region displayed on the white light image, and obtains three-dimensional information.
Based on the color information and the three-dimensional information constituting the white light image, blood existing in the region displayed on the white light image is detected, and blood distribution information regarding the distribution of the blood is acquired.
The bleeding position is specified by tracing back the transition of the state of the blood distribution based on the blood distribution information at each time point when the plurality of white light images are acquired.
The image processing system according to claim 1. The image processing device is based on the blood distribution information at each time point, the amount of change in the area of the region where the blood is present, the amount of blood existing in the region displayed in the white light image, and the change in the blood volume. Calculate at least one of the quantities,
The image processing system according to claim 2. The image processing device determines that bleeding has occurred in the region displayed on the white light image when the amount of change in the area exceeds a predetermined threshold value.
The image processing system according to claim 3. The image processing device determines that bleeding has occurred in the region displayed on the white light image when the amount of change in blood volume exceeds a predetermined threshold value.
The image processing system according to claim 3. The light source device includes a near-infrared light source that irradiates near-infrared light having a wavelength belonging to the near-infrared light band.
The surgical camera further acquires a plurality of near-infrared light images in which the near-infrared light irradiates the surgical site.
The image processing apparatus is based on at least two or more of the plurality of near-infrared light images and the white light image, and the blood in the near-infrared light image or the region displayed in the white light image. Calculate the amount and the amount of change in the blood volume,
The image processing system according to claim 5. The image processing apparatus estimates the future distribution of the blood based on the transition of the distribution of the blood at a plurality of different time points.
The image processing system according to claim 3. The image processing device detects the presence of foreign matter existing in the region displayed on the white light image based on the three-dimensional information.
The image processing system according to claim 2. The light source device includes a laser light source that irradiates a laser beam having a predetermined wavelength.
The surgical camera further acquires a laser beam image of the surgical site irradiated with the laser beam.
The image processing apparatus acquires blood flow information regarding blood flow using the acquired laser beam image, and identifies the bleeding position based on the blood flow information.
The image processing system according to claim 1. The image processing device outputs bleeding-related information regarding the bleeding when it is determined that bleeding has occurred in the region displayed on the white light image.
The image processing system according to claim 1. The image processing device outputs the specified bleeding position by a predetermined method.
The image processing system according to claim 1. The image processing device outputs the white light image from the starting point to the elapse of a predetermined time, starting from a time point retroactive to a predetermined time from the time when the white light image determined to have caused bleeding has been acquired. do,
The image processing system according to claim 1. The image processing device outputs a predetermined signal for urging the occurrence of bleeding when at least one of the blood volume and the change amount of the blood volume exceeds a predetermined threshold value.
The image processing system according to claim 3. The image processing device superimposes and displays a display indicating the bleeding position on the latest white light image at the time point.
The image processing system according to claim 1. The image processing device changes the display indicating the bleeding position in the region displayed on the white light image according to the amount of change in the blood volume or the amount of change in the area of the region where the blood is present.
The image processing system according to claim 3. The image processing device outputs a display indicating the direction of the bleeding position when the bleeding position deviates from the area displayed in the white light image acquired by the surgical camera.
The image processing system according to claim 1. The image processing device outputs a change in the amount of bleeding in the region displayed in the white light image.
The image processing system according to claim 1. The image processing apparatus outputs an estimated future distribution of the blood based on the transition of the distribution of the blood at a plurality of different time points.
The image processing system according to claim 2. By imaging the surgical site, which is a part of the living body under surgery, illuminated by the white light from a light source device having at least a white light source that irradiates white light, at different time points with a surgical camera. Using the obtained plurality of white light images, a specific portion for identifying a bleeding position in the surgical site based on a time change of the surgical site between the plurality of white light images is provided.
Image processing device. By imaging the surgical site, which is a part of the living body under surgery, illuminated by the white light from a light source device having at least a white light source that irradiates white light, at different time points with a surgical camera. An image processing method including identifying a bleeding position in the surgical site based on a time change of the surgical site between the plurality of white light images using the obtained plurality of white light images. A light source device that has at least a white light source that irradiates white light and irradiates the surgical site that is a part of the living body undergoing surgery with the white light, and the light source device that irradiates the white light. A computer capable of communicating with both of the surgical cameras that acquire the white light image of the surgical site.
Using the plurality of white light images acquired by the surgical camera at different times, it functions as a specific part for identifying the bleeding position in the surgical part based on the time change of the surgical part between the plurality of images. To make you
program.

Images