WO2024004013A1

WO2024004013A1 - Program, information processing method, and information processing device

Info

Publication number: WO2024004013A1
Application number: PCT/JP2022/025668
Authority: WO
Inventors: 雅昭伊藤; 建夫大金; 大地北口; 一幸林; 悠貴古澤
Original assignee: 国立研究開発法人国立がん研究センター
Priority date: 2022-06-28
Filing date: 2022-06-28
Publication date: 2024-01-04

Abstract

Provided are: a program which makes it possible to highly accurately detect the occurrence of bleeding from an image captured during a surgery; and others. A computer acquires a plurality of captured images each having a plurality of color components, which are images of a treated area taken on a time-series basis. The computer also acquires bleeding area images with respect to the acquired plurality of images, in which each of the bleeding area images shows a bleeding area in each of the captured images on the basis of the difference in the color components. The computer calculates a difference of bleeding area between bleeding area images of each of captured images before and after in time series, and determines the presence or absence of bleeding from the treated area on the basis of the difference of bleeding area.

Description

Program, information processing method, and information processing device

The present disclosure relates to a program, an information processing method, and an information processing device.

In the medical field, surgery using an endoscope (rigid scope) is performed as a minimally invasive surgery to reduce the burden on patients. For example, in laparoscopic surgery, the operator inserts a laparoscope (endoscope), forceps, and other treatment instruments into the patient's body through a hole made in the patient's abdomen, and images of the treatment area and the forceps are photographed using the laparoscope. Perform the procedure while checking on the monitor. Patent Document 1 discloses a technique for extracting and displaying useful parts when displaying a recorded medical video. In the technology disclosed in Patent Document 1, points at which a change exceeding a certain level occurs, such as the point at which a surgical tool is used or the point at which bleeding starts, are detected from a medical video or medical information as feature points, and the point at which the video is When displayed, the video is displayed based on the detected feature points.

Japanese Patent Application Publication No. 2011-36370

In Patent Document 1, a region whose area changes or a region with movement in a series of videos is detected as a motion region, and among the detected motion regions, the area of the region increases with the passage of time, In addition, a moving region whose area continues to increase for a certain period of time or more and whose chromaticity is similar to that of blood is detected as a bleeding point. However, in images taken during laparoscopic surgery, the field of view and target organs are constantly moving, and blood vessels and blood pools are visible, so it is difficult to accurately determine bleeding points from the images. difficult.

The present invention has been made in view of the above circumstances, and its purpose is to provide a program etc. that can detect bleeding occurrence with high accuracy from images taken during surgery. There is a particular thing.

A program according to an aspect of the present disclosure acquires a plurality of photographed images having a plurality of color components, which are photographed in time series of a treatment region, and calculates a difference between the plurality of color components for each of the plurality of acquired photographic images. Based on this, a bleeding area image indicating the bleeding area in the photographed image is acquired, a difference in the bleeding area between the bleeding area images acquired for each of the preceding and succeeding photographed images in chronological order is calculated, and the difference in the bleeding area is calculated. Based on the difference, the computer is caused to execute a process of determining the presence or absence of bleeding from the treatment site.

According to one aspect of the present invention, occurrence of bleeding can be detected with high accuracy from images taken during surgery.

FIG. 1 is a block diagram showing a configuration example of an endoscopic surgery system. 3 is a flowchart illustrating an example of a bleeding detection processing procedure. It is an explanatory diagram of bleeding detection processing. It is an explanatory diagram of bleeding detection processing. It is an explanatory diagram of bleeding detection processing. It is an explanatory diagram of bleeding detection processing. It is an explanatory diagram of bleeding detection processing. 3 is a flowchart illustrating an example of an alert output processing procedure. 12 is a flowchart illustrating an example of a bleeding detection processing procedure of Modification 1. FIG. 6 is an explanatory diagram of bleeding detection processing according to modification 1; FIG. 6 is an explanatory diagram of bleeding detection processing according to modification 1; 12 is a flowchart illustrating an example of a bleeding detection processing procedure of Modification 2. FIG. 12 is an explanatory diagram of bleeding detection processing in Modification 2. FIG. FIG. 12 is an explanatory diagram of bleeding detection processing in Modification 2. FIG. 12 is a flowchart illustrating an example of a bleeding detection processing procedure according to modification 3. FIG. 12 is an explanatory diagram of bleeding detection processing according to modification 3; 12 is a flowchart illustrating an example of a bleeding detection processing procedure according to modification 4. FIG. 12 is an explanatory diagram of bleeding detection processing in Modification 4. FIG. 12 is a flowchart illustrating an example of a bleeding detection processing procedure according to modification 5. FIG. 9 is an explanatory diagram of bleeding detection processing according to modification 5; 12 is a flowchart illustrating an example of a bleeding detection processing procedure according to modification 6. FIG. 12 is an explanatory diagram of bleeding detection processing according to modification 6; FIG. 12 is an explanatory diagram of bleeding detection processing according to modification 6; FIG. 12 is an explanatory diagram of bleeding detection processing according to modification 6; FIG. 12 is an explanatory diagram of bleeding detection processing according to modification 6; 12 is a flowchart illustrating an example of a bleeding detection processing procedure according to modification 7. FIG. 12 is an explanatory diagram of bleeding detection processing in Modification Example 7; 12 is a flowchart illustrating an example of a bleeding detection processing procedure in Modification 8. FIG. 12 is an explanatory diagram of bleeding detection processing in Modification 8. FIG. 12 is an explanatory diagram of bleeding detection processing in Modification 8. FIG. 12 is an explanatory diagram of bleeding detection processing in Modification 8. FIG. 12 is an explanatory diagram of bleeding detection processing in Modification 8. FIG. 12 is an explanatory diagram of bleeding detection processing in Modification 8. FIG. 2 is an explanatory diagram showing a configuration example of a learning model. 3 is a flowchart illustrating an example of a learning model generation process procedure. 7 is a flowchart illustrating an example of a bleeding detection processing procedure according to the second embodiment. FIG. 7 is an explanatory diagram of bleeding detection processing according to the second embodiment. FIG. 7 is an explanatory diagram of bleeding detection processing according to the second embodiment. FIG. 7 is an explanatory diagram of bleeding detection processing according to the second embodiment. 12 is a flowchart illustrating an example of a bleeding detection processing procedure according to the third embodiment. FIG. 7 is an explanatory diagram of bleeding detection processing according to the third embodiment. FIG. 7 is an explanatory diagram of bleeding detection processing according to the third embodiment. FIG. 7 is an explanatory diagram of bleeding detection processing according to the third embodiment. 12 is a flowchart illustrating an example of a bleeding detection processing procedure according to the fourth embodiment. 13 is a flowchart illustrating an example of a bleeding detection processing procedure according to the fifth embodiment.

Below, a program, an information processing method, and an information processing device of the present disclosure will be described in detail based on drawings showing embodiments thereof. In each of the following embodiments, laparoscopic surgery that treats the abdomen, such as the gastrointestinal tract, will be described as an example of endoscopic surgery. Note that the surgery in each embodiment may be a thoracoscopic surgery or the like in which the chest, such as the heart or lungs, is treated, or may be an open surgery. The configuration of the present disclosure can be used for any surgery as long as the treatment target can be photographed.

(Embodiment 1)
An information processing device that detects the occurrence of bleeding from a treatment site based on captured images of the treatment site during endoscopic surgery will be described. FIG. 1 is a block diagram showing an example of the configuration of an endoscopic surgery system. The endoscopic surgery system shown in FIG. 1 includes an endoscope 20 (laparoscope), an endoscope control device 21 that drives the endoscope 20, an information processing device 10 that controls the operation of the endoscope control device 21, and a display device 40, etc. The endoscopic surgery system also includes a treatment instrument 30, a treatment instrument control device 31 that drives the treatment instrument 30, and an operation section 32 that receives operation inputs for the treatment instrument 30. In endoscopic surgery, the operator inserts an endoscope 20, a treatment tool 30, forceps (not shown), etc. into the patient's body through a hole made in the patient's body, and images are taken using the endoscope 20. While checking images inside the body on the display device 40, a treatment such as resection of the treatment site is performed using the treatment instrument 30.

The endoscope 20 is a rigid scope having a rigid lens barrel, and has an imaging section and a light source section at the tip of the lens barrel. The light source section includes a light source such as an LED (Light Emitting Diode) and an illumination lens, and illumination light (visible light) emitted from the light source is collected by the illumination lens and irradiated onto the subject. The light source of the light source section may be a light source provided in the endoscope control device 21, in addition to the light source provided at the distal end of the endoscope 20. In this case, the endoscope 20 has a light guide that guides illumination light emitted from the light source of the endoscope control device 21 to the distal end of the endoscope 20, and the light guided by the ride guide is directed toward the subject. irradiated. The photographing section has a photographing lens and an image sensor, and the light reflected from the subject by the light emitted by the light source section is received by the image sensor via the photographing lens, and is converted into an image signal by photoelectric conversion by the image sensor. Ru. With such a configuration, the endoscope 20 acquires an image signal (captured image) of a treatment site inside the body by photographing the inside of the body with the imaging unit while the light source unit illuminates the inside of the body. Note that the photographing unit is configured to obtain, for example, 30 frames (30 images) of image signals (video signals) per second.

The endoscope 20 is connected to an endoscope control device 21, and the endoscope control device 21 transmits a control signal for controlling the operation of the endoscope 20 to the endoscope 20. , transmits the image signal photographed by the photographing unit to the endoscope control device 21. The control signal that the endoscope control device 21 transmits to the endoscope 20 includes information regarding photographing conditions, such as zoom magnification (variable magnification), focal length, and photographing direction during photographing, for example. The endoscope control device 21 performs various signal processing on image signals acquired from the endoscope 20 to generate image data. One frame of image data generated here includes an R (red) component, a G (green) component, and a B (blue) component, and the endoscope control device 21 generates an image of the R component from the image signal. data, G component image data, and B component image data. Therefore, through one imaging process by the endoscope 20 (imaging unit), the endoscope control device 21 generates one image including R component image data, G component image data, and B component image data. Obtain data (photographed images). The image data generated by the endoscope control device 21 is output to the information processing device 10, and the information processing device 10 outputs it to the display device 40. Thereby, a photographed image (video) photographed by the photographing section of the endoscope 20 is displayed on the display device 40, and the range illuminated by the light source section of the endoscope 20 can be optically observed.

Signal processing for generating image data from image signals acquired by the imaging unit of the endoscope 20 may be executed by the information processing device 10 in addition to the configuration executed by the endoscope control device 21. In this case, the endoscope control device 21 outputs the image signal acquired from the imaging unit of the endoscope 20 as it is to the information processing device 10, and the information processing device 10 performs predetermined signal processing on the acquired image signal. is executed to generate image data, and the generated image data is sequentially output to the display device 40 to be displayed on the display device 40. Note that the light emitted by the light source section is not limited to visible light, but may also be near-infrared light, special light controlled to a predetermined wavelength, etc., and each light source section irradiates different types of light. A configuration having a plurality of light sources may be used.

The treatment tool 30 is an energy device for incising or resecting living tissue, sealing blood vessels, stopping bleeding, etc. using energy such as high frequency waves, ultrasound waves, and microwave waves. The treatment tool 30 includes, for example, a high-frequency knife, high-frequency scissors, forceps, an electric scalpel, an ultrasonic scalpel, and the like. The treatment tool 30 is connected to a treatment tool control device 31, and the treatment tool control device 31 controls the operation of the treatment tool 30. For example, when the treatment instrument 30 is a device using high frequency, the treatment instrument control device 31 is a high frequency output device that outputs a high frequency current to the treatment instrument 30. An operation section 32 is connected to the treatment instrument control device 31, and the operation section 32 is an input device that receives operation input from medical personnel including, for example, an endoscopic surgery operator. A corresponding control signal is sent to the treatment instrument control device 31. In addition to the treatment tool 30, which is an energy device, the operator manually operates scissors, forceps, and dissection forceps to incise, excise, and detach the tissue of the diseased area.

The information processing device 10 is a device capable of various information processing and transmission/reception of information, and is, for example, a personal computer, a server computer, a workstation, or the like. Note that the information processing device 10 may have a configuration in which distributed processing is performed by a multicomputer including a plurality of computers, or may be implemented by a virtual machine virtually constructed using software within one device. Furthermore, when the information processing device 10 is configured with a server computer, the information processing device 10 may be a local server installed in a medical institution or the like, or a cloud server communicatively connected via a network such as the Internet. It's okay. In this embodiment, the information processing device 10 will be described as one computer.

The information processing device 10 includes a control section 11, a storage section 12, an operation section 13, a communication section 14, a reading section 15, etc., and these sections are interconnected via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit). The control unit 11 performs various information processing, control processing, etc. that the information processing device 10 should perform by appropriately executing the program 12P stored in the storage unit 12.

The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), and the like. The storage unit 12 stores in advance a program 12P (program product) to be executed by the control unit 11 and various data necessary for executing the program 12P. The storage unit 12 also temporarily stores data and the like generated when the control unit 11 executes the program 12P. The program 12P may be written into the storage unit 12 during the manufacturing stage of the information processing device 10.

The operation unit 13 is an input device that accepts operation input from medical personnel, including, for example, an endoscopic surgery operator, and sends a control signal corresponding to the input operation details to the control unit 11. The operation unit 13 may be a keyboard, a mouse, a trackball, a foot switch, a microphone, etc., or may be a sensor that accepts gesture input, gaze input, etc. Further, the operation unit 13 may be a touch panel configured integrally with a display unit (not shown) provided in the information processing device 10. The operation unit 13 accepts operations on the endoscope 20 in addition to operations on the information processing device 10 . Note that the operator's operation on the endoscope 20 is not limited to the configuration in which the operation unit 13 of the information processing device 10 receives and outputs the operation to the endoscope control device 21. The endoscope control device 21 may be configured to accept an operator's operation on the endoscope 20.

The communication unit 14 has a communication module for connecting the endoscope control device 21, transmits control signals from the control unit 11 to the endoscope control device 21, and transmits image data from the endoscope control device 21. Receive (photographed image). Note that the image data received from the endoscope control device 21 is sequentially stored in the storage unit 12, for example. Further, the communication unit 14 has a communication module for connecting the display device 40, and transmits image data to be displayed on the display device 40 to the display device 40 under control from the control unit 11. The communication unit 14, each of the endoscope control device 21 and the display device 40 may be configured to perform wired communication via a cable, or may be configured to perform wireless communication. The communication unit 14 also includes a communication module for connecting to a network such as the Internet or a LAN (Local Area Network) through wired or wireless communication, and transmits and receives information to and from other devices via the network. It may be configured to do so.

The reading unit 15 reads information stored in a portable storage medium 10a including a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc)-ROM, a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, etc. read. The program 12P (program product) and various data stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 10a via the reading unit 15 and stored in the storage unit 12. Further, the program 12P and various data stored in the storage unit 12 may be distributed by a remote server device, which the control unit 11 may download via the communication unit 14 and store in the storage unit 12.

The display device 40 is a liquid crystal display, an organic EL display, or the like, and displays various information appropriately transmitted from the information processing device 10. The information processing device 10 configured as described above performs a process of transmitting a control signal for controlling the operation of the endoscope 20 to the endoscope control device 21 in accordance with an operation by an operator via the operation unit 13. Further, the information processing device 10 performs a process of acquiring a captured image (image data) captured by the endoscope 20 from the endoscope control device 21 and displaying it on the display device 40 . Further, the information processing device 10 performs a process of detecting the occurrence of bleeding from the treatment site based on image data sequentially acquired from the endoscope control device 21 and outputting an alert as necessary.

The endoscopic surgery system configured as described above is installed and used in, for example, an operating room of a medical institution. In addition to the above-mentioned configuration, the endoscopic surgery system includes a pneumoperitoneum device that sends air into the patient's body (abdomen) to inflate the patient's body cavity, a recording device that records images taken with the endoscope 20, etc. The information processing device 10 may have a configuration in which the insufflation device, the recording device, and the like are connected to the information processing device 10.

Below, in the endoscopic surgery system configured as described above, a process of detecting bleeding based on an image taken by the endoscope 20 will be described. FIG. 2 is a flowchart showing an example of a bleeding detection processing procedure, and FIGS. 3A to 4B are explanatory diagrams of the bleeding detection processing. The following processing is executed by the control unit 11 of the information processing device 10 according to the program 12P stored in the storage unit 12.

In the endoscopic surgery system of this embodiment, the endoscope 20 performs imaging by the imaging unit under the control of the endoscope control device 21, and acquires image signals of, for example, 30 frames per second. Output to the control device 21. The endoscope control device 21 performs signal processing on image signals sequentially acquired from the endoscope 20 and outputs processed image data (hereinafter referred to as a photographed image) to the information processing device 10. The control unit 11 (captured image acquisition unit) of the information processing device 10 acquires captured images sequentially output from the endoscope control device 21 via the communication unit 14 (S11). The control unit 11 acquires a photographed image as shown in FIG. 3A, for example. The photographed image shown in FIG. 3A is a photographed image of a treatment region including the treatment instrument 30 and the forceps 30a, and includes a photographed image of an R component, a photographed image of a G component, and a photographed image of a B component.

Based on the acquired captured image, the control unit 11 calculates a difference image by subtracting each pixel value of the G component captured image from each pixel value of the R component captured image (S12). The difference image between the R component photographed image and the G component photographed image can indicate the degree of redness in the photographed image, and in this embodiment, the bleeding area in the photographed image is identified based on this difference image. do. The control unit 11 calculates a difference image by subtracting each pixel value of the B component photographed image from each pixel value of the R component photographed image, and generates a difference image indicating the degree of redness in the photographed image. Good too. The control unit 11 calculates a difference image as shown in FIG. 3B, for example. The difference image shown in FIG. 3B is an example of a difference image between a photographed image of the R component and a photographed image of the G component. For example, the area indicated by the broken line in FIGS. 3A and 3B is a bleeding area with strong redness. In addition to the above-mentioned difference image, the control unit 11 generates an image (R/G) obtained by dividing each pixel value of the G component from each pixel value of the R component, or the logarithm (log(R/G) of the division result). An image representing a feature amount such as G)) may be generated. Further, in addition to the configuration in which the control unit 11 detects a bleeding area based on the images showing each feature amount described above, the control unit 11 may detect a bleeding area by combining a plurality of images.

The control unit 11 (bleeding area acquisition unit) generates a bleeding area image indicating the bleeding area in the captured image based on the color component difference image (S13). Here, the control unit 11 compares each pixel value of the difference image with a predetermined threshold, and assigns white (for example, 1) to pixels whose pixel value is equal to or greater than the predetermined threshold, and to pixels whose pixel value is less than the predetermined threshold. A bleeding area image is generated by assigning black (for example, 0) to each of the images. The control unit 11 generates a bleeding area image as shown in FIG. 3C, for example. In the bleeding area image shown in FIG. 3C, a white area is identified as a bleeding area. Note that the threshold value for determining whether or not it is a bleeding area may be dynamically set based on each pixel value (brightness of each pixel) in the captured image. This makes it possible to set a threshold value in consideration of the imaging conditions, etc., and it is possible to more appropriately identify a bleeding area in a captured image. The control unit 11 performs the above-described processes of steps S11 to S13 on the captured images of each frame sequentially transmitted from the endoscope control device 21, and generates a bleeding area image from each captured image.

The control unit 11 (calculation unit) calculates the difference (increase amount) in the bleeding area between successive frames in time series, based on the bleeding area in the captured images of each frame sequentially transmitted from the endoscope control device 21. is calculated (S14). Specifically, the control unit 11 identifies, among the bleeding areas in the current frame (frame to be processed), an area that was not a bleeding area in the immediately previous frame (hereinafter referred to as the previous frame). An image showing the area (increased bleeding area) is generated, and the number of pixels included in the increased bleeding area is counted. Then, the control unit 11 stores the counted number of pixels as an increase amount (difference) in the bleeding area in the storage unit 12 in association with the imaging time of the current frame. The control unit 11 acquires time-series data as shown by the solid line in FIG. 4A, for example, by sequentially calculating the difference (increase amount) in the bleeding area between frames that precede and follow in time series. The solid line in FIG. 4A shows the time-series change in the difference in the bleeding area between each frame. In the graph shown in FIG. This shows the difference in the bleeding area between the photographed image and the photographed image. Note that the imaging time may be the elapsed time since the endoscope 20 starts the imaging process, and may be, for example, the imaging date and time indicated by a clock included in the endoscope control device 21.

Next, the control unit 11 calculates a moving average of the differences in the bleeding area between each frame (S15). For example, the control unit 11 calculates a simple moving average every predetermined time period of about 3 seconds. Since 30 frames of captured images are captured per second, the control unit 11 calculates a moving average of the differences in bleeding areas in the previous and subsequent captured images, for example, in 90 frames captured in 3 seconds. Specifically, the control unit 11 calculates the average value of the difference in the bleeding area between each frame from the time to a predetermined time for a certain time, associates the calculated average value with the time, and stores it in the storage unit. Store in 12. The control unit 11 calculates the average value of the difference in the bleeding area in a predetermined time for each imaging time, and calculates the moving average of the difference in the bleeding area based on all the frames photographed by the endoscope 20.

Next, for each imaging time, the control unit 11 sets a threshold value to be used when determining the presence or absence of bleeding, based on the moving average of the difference in bleeding area between each frame calculated in step S15 up to each imaging time. Calculate (S16). Here, the control unit 11 calculates a value of +4σ (+4 standard deviation) with respect to the average value of the measured values, using the moving average of the differences in bleeding areas calculated up to each imaging time as a measurement value, and Set the judgment threshold in . The broken line in FIG. 4A indicates a time series change between the −4σ value and the +4σ value with respect to the moving average of the bleeding area difference, and the control unit 11 determines whether the bleeding area difference between frames is If the moving average of the area differences is greater than or equal to the range of -4σ to +4σ (predetermined range), it is determined that bleeding has occurred. Therefore, the control unit 11 sets each value of the upper (+4σ) time series data indicated by the broken line in FIG. 4A as a threshold value used when determining the presence or absence of bleeding occurrence for each imaging time.

The control unit 11 (determination unit) determines the presence or absence of bleeding for this imaging time based on the difference in bleeding area between frames calculated in step S14 and the determination threshold calculated in step S16 (S17 ). Specifically, for each imaging time, the control unit 11 determines whether the difference in the bleeding area calculated in step S14 is greater than or equal to the set determination threshold, and if it is greater than or equal to the determination threshold, it is determined that bleeding has occurred. If it is less than the determination threshold, it is determined that no bleeding has occurred. That is, in this embodiment, the presence or absence of bleeding is determined depending on whether the amount of increase (difference) in the bleeding area is greater than or equal to the determination threshold, and the determination threshold is determined based on the moving average of the amount of increase in the bleeding area. Set based on

Here, the reason why a value of +4σ is used for the moving average of the difference in the bleeding area between each frame as the threshold for determining the occurrence of bleeding will be explained. In FIG. 4B, the solid line shows the time-series change in the bleeding area difference between the previous and subsequent frames, and the broken line shows the time-series change of the -2σ value and +2σ value with respect to the moving average of the bleeding area difference. It shows a series change. For example, if each value of the upper (+2σ) time series data indicated by the broken line in FIG. 4B is used as a threshold for determining the presence or absence of bleeding, the difference in the bleeding area is greater than or equal to the determination threshold at times T1 to T5 in FIG. 4B. Therefore, it is determined that bleeding has occurred. That is, in the state shown in FIG. 4B, after the first bleeding is detected at time T1, a small amount of bleeding at times T2 to T5 is also frequently determined to be bleeding occurrence. In this way, when it is frequently determined that bleeding has occurred, it is not possible to accurately detect the occurrence of bleeding that should be notified to medical personnel, leading to an increase in false detections. In this embodiment, in order to detect the presence or absence of bleeding based on the amount of increase in the bleeding area, the rise of the waveform indicated by the solid line in FIGS. 4A and 4B is detected. Therefore, by using the value of the moving average of the differences in bleeding areas + 4σ instead of the value of the moving average of differences in bleeding areas + 2σ as the threshold for determining the presence or absence of bleeding, it is possible to prevent sudden bleeding even in the state shown in FIG. 4B, for example. It can be determined that bleeding has occurred only during the time T1 when the area is increasing, and it is possible to detect bleeding occurrence with high accuracy.

If it is determined that bleeding has occurred (S17: YES), the control unit 11 specifies that bleeding has started (S18). Note that if it is determined in step S17 that no bleeding has occurred (S17: NO), the control unit 11 skips the process in step S18 and proceeds to the process in step S19. The control unit 11 determines whether or not to end the above-described processing (S19). For example, the control unit 11 receives an instruction to end the above-described process from the operator via the operation unit 13, and determines to end the process when the end instruction is received. When the control unit 11 determines that the above-described process is not to be ended (S19: NO), the process returns to step S11. In this case, the control unit 11 sequentially acquires photographed images from the endoscope control device 21 (S11), and executes the processes of steps S12 to S18 based on the acquired photographed images. If the control unit 11 determines to end the above-described processing (S19: YES), it ends the series of processing.

Next, a process for outputting an alert when the start of bleeding is identified by the bleeding detection process described above will be described. FIG. 5 is a flowchart showing an example of an alert output processing procedure. The control unit 11 of the information processing device 10 executes the following process while executing the above-described bleeding detection process. In the bleeding detection process described above, the control unit 11 determines whether the start of bleeding has been identified (S21), and if it is determined that the start of bleeding has been identified (S21: YES), it starts outputting an alert. (S22). For example, the control unit 11 displays a message on the display device 40 notifying that bleeding has occurred, and notifies the operator's doctor and medical personnel such as nurses. Note that when the information processing device 10 or the display device 40 has a lamp, a buzzer, a speaker, etc., by lighting or blinking the lamp, sounding the buzzer, or outputting a voice message or warning sound from the speaker, Healthcare professionals may be notified. Further, the control unit 11 stores the imaging time at the time when it is determined that bleeding has started (bleeding has occurred) in the storage unit 12 as the bleeding occurrence time (S23).

When the control unit 11 identifies the start of bleeding and starts outputting an alert, it determines whether the alert termination conditions are met (S24). The condition for ending the alert is, for example, when the bleeding has stopped (ended), and the control unit 11 may determine that the bleeding has stopped when the bleeding area between the previous and subsequent frames does not increase in time series. In addition, the control unit 11 performs processes similar to steps S161 to S165 in the bleeding detection process in FIG. 30, which will be described in Embodiment 5, and terminates the alert when it is detected that the bleeding has ended due to the hemostasis operation. It may be determined that the conditions are met. When the control unit 11 determines that the alert end condition is not satisfied (S24: NO), the control unit 11 continues outputting the alert started in step S22 (S25). On the other hand, if the control unit 11 determines that the alert termination condition is satisfied (S24: YES), it terminates the output of the alert started in step S22 (S26). Further, the control unit 11 stores the imaging time at the time when it is determined that the alert end condition is satisfied in the storage unit 12 as the bleeding end time (S27). After that, the control unit 11 returns to the process of step S21. After the process of step S25, the control unit 11 determines whether or not to end the process described above (S28). For example, the control unit 11 receives an instruction to end the above-described process from the operator via the operation unit 13, and determines to end the process when the end instruction is received. When the control unit 11 determines that the above-described process is not to be ended (S28: NO), the process returns to step S24. If the control unit 11 determines to end the above-described processing (S28: YES), it ends the series of processing. Thereby, when the occurrence of bleeding is identified through the bleeding detection process, an alert can be output to notify medical personnel of the occurrence of bleeding. Further, after the alert output is started, if the bleeding has ended and the alert termination condition is satisfied, the alert output can be terminated to notify the medical staff that the bleeding has stopped.

In the present embodiment, bleeding areas can be detected with high accuracy in images taken with the endoscope 20 based on the difference image between the R component image and the G or B component image. In addition, the presence or absence of bleeding is determined based on the amount of increase in the bleeding area detected with high accuracy between the images taken before and after the time series, so the occurrence of bleeding can be detected early and objectively. medical personnel can be notified. Therefore, medical personnel can promptly treat the occurrence of bleeding, and can also determine whether preparations for blood transfusion are necessary or not at an early stage. Although the present embodiment has been described using as an example a configuration that detects the occurrence of bleeding based on an image captured by the endoscope 20 during surgery, the configuration is not limited to such a configuration, and the configuration of the present disclosure can be used, for example. It can also be used in verification processing after surgery.

Further, in the present embodiment, the information processing device 10 displays a graph as shown in FIG. 40 (display unit) may be displayed. For example, after the process in step S18 in FIG. 2, the graph shown in FIG. 4A is generated by plotting the difference in bleeding area between frames and the bleeding determination threshold at each imaging time in association with each imaging time. , the generated graph may be output to the display device 40 and displayed. Further, at this time, the information processing device 10 may indicate the timing (imaging time) at which the occurrence of bleeding was determined in the displayed graph. For example, as shown by the arrow in FIG. 4A, the imaging time at which the occurrence of bleeding was detected can be presented. Note that when the information processing device 10 performs processing for determining the presence or absence of bleeding after surgery based on the photographed images taken with the endoscope 20 during the surgery, the bleeding area between each frame is determined for all the photographed images. After calculating the difference and the bleeding determination threshold based on the moving average of the difference in bleeding areas, a graph as shown in FIG. 4A may be displayed on the display device 40. In this case, in the post-surgery verification process, it becomes possible to verify the timing of bleeding occurrence, etc.

In the present embodiment, the process for determining the presence or absence of bleeding based on the photographed image taken by the endoscope 20 is not limited to a configuration in which the information processing device 10 performs it locally. For example, a server may be provided that executes a process for determining whether or not bleeding has occurred. In this case, the information processing device 10 transmits the photographed images sequentially acquired from the endoscope control device 21 to the server, and the server determines the presence or absence of bleeding based on the photographed images and sends the determination result to the information processing device 10. configured to send to. Even in the case of such a configuration, the same processing as in this embodiment described above is possible and the same effects can be obtained.

(Modification 1)
A first modification of the first embodiment described above will be described. In this modification, when the imaging unit of the endoscope 20 performs imaging, correction is performed to correct the deviation between each frame that follows in time based on the amount of movement (amount of change) of the imaging position by the imaging unit. The information processing device 10 that performs processing will be described. The imaging position and imaging direction of the imaging unit of the endoscope 20 are changed depending on, for example, a change in the state in which the operator holds (grasps) the endoscope 20 . In this modification, the presence or absence of bleeding can be determined more accurately by correcting the shift that occurs between frames in accordance with changes in the imaging position and imaging direction.

FIG. 6 is a flowchart illustrating an example of a bleeding detection processing procedure according to modification 1, and FIGS. 7A and 7B are explanatory diagrams of the bleeding detection processing according to modification 1. The process shown in FIG. 6 is the process shown in FIG. 2 with steps S31 and S32 added between steps S11 and S12. Description of the same steps as in FIG. 2 will be omitted.

In the information processing device 10 of this modification, when the control unit 11 acquires a photographed image photographed by the endoscope 20 from the endoscope control device 21 (S11), the control unit 11 determines the amount of change in the photographing position by the endoscope 20. (S31). For example, the endoscopic surgery system has a holding mechanism that holds the endoscope 20, and the endoscope control device 21 is configured to control the holding state of the endoscope 20 by controlling the operation of the holding mechanism. In some cases, the control unit 11 can acquire the amount of change in the imaging position from the endoscope control device 21. Furthermore, if the endoscope 20 is provided with a sensor that detects the position of the imaging section (for example, the distal end of the endoscope 20), the control section 11 determines the amount of change in the imaging position based on the detection result by the sensor. may be obtained. Note that the amount of change in the photographing position includes the amount of translation and rotation on the surface facing the subject, and the zoom magnification (scale).

The control unit 11 performs correction processing based on the amount of change in the imaging position on the bleeding area image (the bleeding area image of the previous frame) generated from the immediately previous captured image (S32). For example, the control unit 11 moves each pixel by the amount of translation on the surface facing the subject, rotates each pixel by the amount of rotation, and calculates a transformation matrix for performing magnification processing according to the zoom magnification. do. Then, the control unit 11 moves each pixel of the bleeding area image of the previous frame based on the calculated transformation matrix, and generates a bleeding area image after the conversion process. As a result, coordinate transformation can be performed on the bleeding area image of the previous frame using the transformation matrix, and a bleeding area image in which each pixel of the bleeding area image of the previous frame is aligned with the current frame (frame to be processed) is created. can get.

FIG. 7A shows a superimposed display of bleeding area images generated from two frames before and after in chronological order, where the area indicated by a broken line indicates the bleeding area in the previous frame, and the area indicated by a solid line indicates the bleeding area in the subsequent frame (current Figure 2 shows the bleeding area in the frame. The control unit 11 can correct the deviation based on the amount of change in the imaging position between the previous frame and the subsequent frame by performing correction processing on the bleeding area image of the previous frame based on the amount of change in the imaging position. . Therefore, by performing such correction processing on the bleeding area image of the previous frame, it is possible to obtain the bleeding area image of the previous frame in which the deviation based on the amount of change in the imaging position has been corrected, as shown in FIG. 7B. .

After that, the control unit 11 performs the processing from step S12 onwards. Note that in step S14, the control unit 11 calculates the difference between the bleeding area image of the previous frame subjected to the correction process in step S32 and the bleeding area image of the current frame generated in step S13 (S14). Steps S31 to S32 in FIG. 6 may be executed after the process of step S11 and before the process of step S14.

In this modification, it is possible to correct the deviation based on the amount of change in the imaging position in frames that follow in time series, so the difference in bleeding area can be corrected between frames by eliminating the deviation based on the amount of change in the imaging position. can be calculated with high accuracy. Note that in this modification, the deviation of each pixel position due to the amount of change in the imaging position is corrected for the bleeding region image generated from the imaging image, but the present invention is not limited to such a configuration. For example, correction processing based on the amount of change in the photographing position may be performed on the photographed image. In this case, in step S32, the control unit 11 performs a correction process on the captured image of the previous frame, generates a bleeding area image from the captured image based on the captured image after the correction process, and generates a bleeding area image from the captured image. The process in step S14 may be performed using the image as the bleeding area image of the previous frame.

Also in this modification, the same effects as in the first embodiment described above can be obtained. In addition, in this modified example, by correcting the deviation of the entire imaging field that occurs due to the movement of the imaging position, it is possible to eliminate the deviation between frames that occurs during imaging, so the difference in bleeding area between frames can be accurately calculated. It can be calculated. Therefore, the presence or absence of bleeding can be detected with high accuracy based on the difference between the bleeding areas calculated with high accuracy. Further, also in this modification, the modifications described in the above-described first embodiment can be applied.

(Modification 2)
In this modification, an information processing device 10 that performs a correction process to correct a shift between frames that follow each other in time series based on images taken with an endoscope 20 will be described. Modification 1 described above has a configuration that mechanically acquires the amount of change in the imaging position of the endoscope 20 (imaging unit), whereas this modification uses image processing for the captured image to determine the amount of change between each frame. This configuration calculates the amount of deviation and corrects the calculated amount of deviation.

FIG. 8 is a flowchart showing an example of a procedure for bleeding detection processing in Modification 2, and FIGS. 9A and 9B are explanatory diagrams of bleeding detection processing in Modification 2. The process shown in FIG. 8 is the process shown in FIG. 6 with steps S41 to S43 added instead of steps S31 to S32. Description of the same steps as in FIG. 6 will be omitted.

In the information processing device 10 of this modification, when the control unit 11 acquires a photographed image photographed by the endoscope 20 from the endoscope control device 21 (S11), the control unit 11 selects the previous frame for the acquired photographic image. A feature point corresponding to each feature point in is extracted (S41). For example, the control unit 11 uses image processing such as optical flow based on captured images sequentially acquired from the endoscope control device 21 to generate a motion vector (movement amount and direction). Therefore, the control unit 11 can extract feature points corresponding to the feature points extracted in the previous frame from the current frame. FIG. 9A is an example in which the feature points extracted from the captured image are indicated by black circles, and as shown in FIG. 9B, the control unit 11 controls the feature points in the subsequent frame based on the feature points in the previous frame (n-1 frame). Obtain feature points in (n frames). Note that the feature points whose movements are to be tracked may be specified using any method. Thereby, the amount of shift (the amount of movement of each feature point) between the frames that follow in time series can be obtained by image processing on the captured image.

The control unit 11 changes the position of each feature point of the previous frame to the position of each feature point of the current frame based on the position of each feature point of the previous frame and the position of each feature point of the current frame extracted in step S41. A transformation matrix is calculated to bring the value closer to (S42). The control unit 11 calculates a transformation matrix that minimizes the difference between each position after moving each feature point of the previous frame and the position of each feature point of the current frame by a transformation process based on the transformation matrix. do. Then, the control unit 11 performs a correction process on the bleeding area image of the previous frame based on the calculated transformation matrix (S43), and corrects the amount of deviation between it and the subsequent frame. As a result, also in this modification, as shown in FIG. 7B, the misalignment between frames is corrected, and a bleeding area image in which each pixel of the bleeding area image of the previous frame is aligned with the current frame is obtained. After that, the control unit 11 performs the processing from step S12 onwards. Note that in FIG. 8 as well, steps S41 to S43 may be executed after the process of step S11 and before the process of step S14.

Also in this modification, the same effects as in the first embodiment and modification 1 described above can be obtained. Furthermore, in this modification, the amount of shift between frames is calculated by image processing based on the captured images, so there is no need to mechanically obtain the amount of change in the imaging position by the endoscope 20. Therefore, even if, for example, post-surgery detection processing is performed on the occurrence of bleeding based only on images taken with the endoscope 20, it is possible to eliminate the discrepancy between frames. The difference in bleeding areas between frames can be calculated with high accuracy. Further, also in this modification, the modifications described in the above-mentioned Embodiment 1 and Modification 1 can be applied.

(Modification 3)
In this modification, when the amount of change in the zoom magnification (enlargement rate) during imaging by the endoscope 20 is greater than or equal to a predetermined magnification, the determination result of bleeding occurrence based on the photographed images taken before and after the change in zoom magnification is determined. An information processing device 10 that disables the following will be described. The photographing unit of the endoscope 20 is configured to be able to change the zoom magnification during photographing, and when photographing with zoom-in (enlargement), the bleeding area in the photographed image before zooming in will change to the photographed image after zooming in. It will be greatly expanded. That is, the size (number of pixels) of the bleeding area in the captured image increases before and after zooming in, and if the presence or absence of bleeding is detected based on this increase, erroneous detection will occur. Therefore, in this modified example, when the amount of change in the zoom magnification (enlargement rate) is greater than or equal to a predetermined magnification, the determination result of bleeding occurrence based on the captured images before and after changing the zoom magnification is invalidated. false positive detection can be suppressed.

FIG. 10 is a flowchart illustrating an example of a procedure for bleeding detection processing according to modification 3, and FIG. 11 is an explanatory diagram of the bleeding detection processing according to modification 3. The process shown in FIG. 10 is the process shown in FIG. 2 with steps S51 to S53 added between YES in step S17 and step S18. Description of the same steps as in FIG. 2 will be omitted.

In the information processing device 10 of this modification, the control unit 11 executes the processes of steps S11 to S17 in FIG. Acquire the magnification (S51). For example, the control unit 11 acquires from the endoscope control device 21 the zoom magnification at which the endoscope control device 21 has controlled the endoscope 20 . The control unit 11 acquires the zoom magnification of the endoscope 20 each time an image is taken by the endoscope 20, thereby acquiring time-series data of the zoom magnification as shown by the solid line in FIG. 11, for example. The solid line in FIG. 11 shows the time-series change in zoom magnification by the endoscope 20. In the graph shown in FIG. 11, the horizontal axis shows the imaging time, and the vertical axis shows the zoom magnification at each imaging time. . Note that instead of acquiring the zoom magnification set for the endoscope 20, the control unit 11 calculates the amount of change in the zoom magnification (magnification rate) from the previous frame by image processing on the previous and subsequent frames in chronological order. It may be calculated.

Next, the control unit 11 sets a threshold value (magnification threshold value) is calculated (S52). For example, the control unit 11 calculates the moving average of the zoom magnification shown by the solid line in FIG. 11, and sets the moving average of the zoom magnification for each imaging time as the magnification threshold for the imaging time. The moving average of the zoom magnification is, for example, a simple moving average every predetermined period of about 3 seconds. The broken line in FIG. 11 indicates a time series change in the moving average of the zoom magnification at each imaging time, and the control unit 11 changes each value of the time series data shown by the broken line in FIG. Set as the scaling threshold used when determining whether zoom-in processing has been performed.

The control unit 11 determines whether the zoom magnification obtained in step S51 is greater than or equal to the magnification threshold set in step S52 (S53). If the control unit 11 determines that the zoom magnification is equal to or greater than the magnification threshold (S53: YES), the control unit 11 skips the process of step S18 and proceeds to the process of step S19. As a result, if the zoom magnification is equal to or higher than the magnification threshold, that is, if it is determined that the zoom-in process that should invalidate the bleeding occurrence determination process has been performed, the control unit 11 controls the bleeding occurrence determined in the most recent step S17. The determination result of "Yes" is invalidated and it is determined that there is no bleeding. On the other hand, if the control unit 11 determines that the zoom magnification is less than the magnification threshold (S53: NO), it determines that the zoom-in process that should invalidate the bleeding occurrence determination process is not performed, and the process in step S18 is performed. Execute. As a result, if the zoom magnification is less than the magnification threshold, that is, if it is determined that the zoom-in process that should invalidate the bleeding occurrence determination process has not been performed, the control unit 11 Based on the determination result that bleeding has occurred, it is determined that bleeding has started. In the time-series change in zoom magnification shown in FIG. 11, the section indicated by the double-headed arrow indicates the time when it is determined that the zoom-in process has been performed.

Steps S51 to S53 in FIG. 10 only need to be executed after the processing of step S11 and before the processing of step S18. For example, when executed after the processing of step S11, it is determined that the zoom magnification is equal to or higher than the magnification threshold. In this case, the processing of steps S12 to S18 may be skipped. In this case, when the zoom magnification is equal to or greater than the magnification threshold, each process for determining the occurrence of bleeding is not executed, thereby making it possible to reduce the processing load.

Also in this modification, the same effects as in the first embodiment described above can be obtained. In addition, in this modification, it is determined whether zoom-in processing that should invalidate the bleeding occurrence determination processing is performed, and when the zoom-in processing is performed, the captured images taken before and after the zoom-in processing are By invalidating the determination result of bleeding occurrence based on this, it is possible to suppress erroneous detection of bleeding occurrence that occurs when the amount of change in zoom magnification is large. The configuration of this modification can be combined with the configurations of each modification described above, and the same effect can be obtained even when combined with each modification. Further, also in this modification, the modifications described in the above-described first embodiment and each modification can be applied.

(Modification 4)
In this modification, for a bleeding area detected in a captured image, the center of gravity of the bleeding area is traced back for a predetermined period of time, and if the tracked trajectory is included in a predetermined area around the periphery of the captured image, the bleeding area is detected. The following describes an information processing apparatus 10 that excludes objects used in the bleeding occurrence determination process as objects that have entered the imaging range from outside the imaging range due to movement of the imaging position or the like. As the imaging position of the endoscope 20 moves, the imaging range moves, and a bleeding area that was outside the imaging range when imaging the previous frame may come within the imaging range of the current frame. . In this embodiment, the purpose is to accurately detect bleeding that has occurred within the imaging range of the endoscope 20, so the presence or absence of bleeding is detected based on the bleeding area that has entered the imaging range from outside the imaging range. If detected, a false positive will occur. Therefore, in this modification, by excluding the above-mentioned bleeding area from the targets when detecting the occurrence of bleeding, it is possible to suppress erroneous detection of the occurrence of bleeding.

FIG. 12 is a flowchart showing an example of a procedure for bleeding detection processing in Modification 4, and FIG. 13 is an explanatory diagram of bleeding detection processing in Modification 4. The process shown in FIG. 12 is the process shown in FIG. 2 with steps S61 to S63 added between steps S13 and S14. Description of the same steps as in FIG. 2 will be omitted.

In the information processing device 10 of this modification, the control unit 11 executes the processes of steps S11 to S13 in FIG. 2, and generates a bleeding area image indicating the bleeding area in the current frame (S13). The control unit 11 identifies the center of gravity of the bleeding area in the generated bleeding area image (S61). Specifically, the control unit 11 calculates the coordinate values of the center of gravity of the bleeding area in the bleeding area image. Note that the coordinate value of each pixel in the bleeding area image is expressed, for example, with the upper left of the captured image as the origin (0, 0), and the number of pixels from the origin to the right and the number of pixels from the origin to the bottom. . The coordinate value of the center of gravity of the bleeding area may be expressed, for example, by the average value of the coordinate values of all pixels included in the bleeding area, and may be expressed as the coordinate value of the center position in the left and right direction of the leftmost pixel and the rightmost pixel of the bleeding area. , the coordinate value of the center position in the vertical direction of the upper end pixel and the lower end pixel of the bleeding region, or the average value of the coordinate values of each pixel on the outline of the bleeding region. FIG. 13 shows an example of a bleeding area image, and for the bleeding area shown in FIG. 13, the position indicated by the + mark is specified as the position of the center of gravity C of the bleeding area.

Based on the bleeding area in the current frame (bleeding area image), the control unit 11 traces the center of gravity of the bleeding area going back to a frame a predetermined time ago, and obtains the locus of the center of gravity of the bleeding area (S62). . In the bleeding area image shown in FIG. 13, the centroids of bleeding areas traced back to frames a predetermined time ago are each indicated by a + mark. Note that in the bleeding area images in each frame up to a predetermined time ago, the coordinate values of the center of gravity of the bleeding area may already be calculated and stored in the storage unit 12. In this case, the control unit 11 can acquire the locus of the center of gravity of the bleeding area by reading out the coordinate values of the center of gravity of the bleeding area in each frame stored in the storage unit 12. Further, the control unit 11 extracts the bleeding area in the previous frame based on the bleeding area in the bleeding area image of the current frame, and performs the process of calculating the center of gravity of the extracted bleeding area sequentially up to the frame a predetermined time ago. By executing this, the locus of the center of gravity of the bleeding area may be obtained.

The control unit 11 determines whether the locus of the center of gravity of the acquired bleeding area is included in a predetermined area at the periphery of the photographed image (S63). For example, the control unit 11 determines whether the trajectory of the center of gravity of the bleeding area is included in the peripheral area indicated by hatching in the photographed image (bleeding area image) shown in FIG. 13 . If it is determined that the trajectory of the center of gravity of the bleeding area is included in the peripheral area of the photographed image (S63: YES), the control unit 11 skips the processes of steps S14 to S18 and proceeds to the process of step S19. If the peripheral area of the photographed image includes the locus of the center of gravity of the bleeding area, it is considered that the bleeding area has come into the photographing range from outside the photographing range due to movement of the photographing position or the like. Therefore, when such a bleeding area is detected, erroneous detection is avoided by not performing a process for determining the occurrence of bleeding based on this bleeding area. Since the locus of the center of gravity in the bleeding area shown in Figure 13 is included in the lower edge area of the photographed image, it is determined that this bleeding area has come into the photographing range from outside the photographing range due to movement of the photographing position, etc. The bleeding occurrence determination process is skipped.

On the other hand, if the control unit 11 determines that the locus of the center of gravity of the bleeding area is not included in the peripheral area of the photographed image (S63: NO), it determines that the bleeding area has occurred within the photographing range, and steps The processes of S14 to S18 are executed. As a result, for bleeding that occurs within the photographing range of the endoscope 20, the determination process of bleeding occurrence is performed based on the difference in the bleeding area between the frames that follow in time series. Note that when the bleeding area image includes a plurality of bleeding areas, the control unit 11 executes the processes of steps S61 to S63 for each bleeding area. As a result, for each bleeding area, it is determined whether the center of gravity has entered from outside the imaging range, and bleeding areas that have entered from outside the imaging range are excluded from the determination process of bleeding occurrence determination processing. However, it is not used for determining the occurrence of bleeding.

Also in this modification, the same effects as in the first embodiment described above can be obtained. Furthermore, in this modification, a bleeding area that has come into the imaging range from outside the imaging range due to movement of the imaging position or the like is not used in the process of determining the occurrence of bleeding, thereby suppressing erroneous detection of the occurrence of bleeding. The configuration of this modification can be combined with the configurations of each modification described above, and the same effect can be obtained even when combined with each modification. Further, also in this embodiment, the modifications appropriately described in the above-described first embodiment and each modification can be applied.

(Modification 5)
In this modification, an information processing apparatus 10 will be described in which, when a bleeding area in a photographed image rapidly decreases due to the generation of mist, the photographed image (frame) at this time is not subjected to the bleeding occurrence determination process. In the endoscopic surgery system, mist (mist, steam) is generated when living tissue is incised, resected, peeled off, etc. using the treatment tool 30. When mist is generated, the redness of the bleeding area becomes diluted, so that it is not determined to be a bleeding area, and a situation may occur where the area of the bleeding area suddenly decreases. Therefore, in this modification, when mist is generated, the images taken before and after the mist generation are excluded from the targets for determining the presence or absence of bleeding, and the bleeding occurrence determination processing is not executed. Suppress detection.

FIG. 14 is a flowchart showing an example of a procedure for bleeding detection processing according to modification 5, and FIG. 15 is an explanatory diagram of the bleeding detection processing according to modification 5. The process shown in FIG. 14 is the process shown in FIG. 2 with steps S71 and S72 added between steps S15 and S16. Description of the same steps as in FIG. 2 will be omitted.

In the information processing device 10 of this modification, the control unit 11 executes the processes of steps S11 to S15 in FIG. In step S14, the control unit 11 acquires time-series data of differences in bleeding areas as shown by solid lines in FIG. 15, for example, by sequentially calculating differences in bleeding areas between frames that follow in time series. In the graph shown in FIG. 15, the horizontal axis shows the imaging time, and the vertical axis shows the difference in the bleeding area at each imaging time.

Next, for each imaging time, the control unit 11 sets a threshold ( mist determination threshold value) is calculated (S71). Here, the control unit 11 uses the moving average of the differences in bleeding areas calculated up to each imaging time as a measurement value, calculates a value of −2σ with respect to the average value of the measurement values, and sets the mist determination threshold at the imaging time. Set to . The broken line in FIG. 15 indicates a time-series change between the −2σ value and the +2σ value with respect to the moving average of the bleeding area difference, and the control unit 11 determines that the bleeding area difference between frames is When the moving average of the difference is less than -2σ, it is determined that mist has occurred. Therefore, the control unit 11 sets each value of the lower (-2σ) time series data indicated by the broken line in FIG. 15 as a threshold value used when determining whether mist is generated for each imaging time.

The control unit 11 determines whether mist has occurred in the photographed image based on the difference in bleeding area between frames calculated in step S14 and the mist determination threshold calculated in step S71 (S72). Specifically, for each imaging time, the control unit 11 determines whether the difference in the bleeding area calculated in step S14 is less than the set mist determination threshold, and if it is less than the mist determination threshold, If it is determined that mist has occurred, and if the mist determination threshold value is greater than or equal to the mist determination threshold, it is determined that mist has not occurred. That is, in this embodiment, the presence or absence of mist generation is determined depending on whether the amount of increase (difference) in the bleeding area is less than the mist determination threshold. In addition, the presence or absence of mist generation can be determined not only by color changes in captured images (specifically, differences in bleeding areas) but also by, for example, whether the contrast (brightness ratio) of the entire image has suddenly decreased. good. For example, it may be determined that mist has occurred when the contrast of the entire image has decreased by a predetermined value or more.

If it is determined that no mist is generated (S72: NO), the control unit 11 executes the processes from step S16 onwards, and performs the process of determining the occurrence of bleeding. On the other hand, if it is determined that mist is generated (S72: YES), the control unit 11 skips the processes of steps S16 to S18 and proceeds to the process of step S19. Thereby, when mist is generated, the control unit 11 avoids erroneous detection by not performing determination processing of bleeding occurrence based on the photographed image (bleeding area). In the time-series change in the difference in the bleeding area shown in FIG. 15, the section indicated by the arrow indicates the time when it is determined that mist has occurred.

Also in this modification, the same effects as in the first embodiment described above can be obtained. In addition, in this modification, when the occurrence of mist is detected in a photographed image, the erroneous detection of bleeding occurrence is avoided by not performing the determination process of bleeding occurrence based on the photographed images taken before and after the occurrence of mist. It can be suppressed. The configuration of this modification can be combined with the configurations of each modification described above, and the same effect can be obtained even when combined with each modification. Further, also in this modification, the modifications described in the above-described first embodiment and each modification can be applied.

(Modification 6)
In this modification, when a bleeding area that has been hidden by the treatment instrument 30 and gauze in the photographed image is exposed as the treatment instrument 30 and gauze move, the bleeding area is used as a target for determining the occurrence of bleeding. The information processing devices 10 to be excluded from the following will be explained. In endoscopic surgery, bleeding is stopped using the treatment instrument 30 and gauze, and when the treatment instrument 30 and gauze are removed after the bleeding has stopped, the bleeding area that was hidden by the treatment instrument 30 and gauze is exposed, and the bleeding area is exposed. Situations may arise where the area increases significantly. Therefore, in this modification, the bleeding area exposed due to the movement of the treatment tool 30 and the gauze is excluded from the target when detecting the occurrence of bleeding, thereby suppressing the erroneous detection of the occurrence of bleeding.

In addition to the configuration of the first embodiment shown in FIG. 1, the information processing device 10 of this modification stores a learning model for detecting a predetermined object in an image in the storage unit 12. The learning model here is a learning model that has been trained by machine learning to output the region of the treatment instrument 30 and gauze in the photographed image when the photographed image is input. Such learning models include CNN (Convolution Neural Network), R-CNN (Regions with CNN), Fast R-CNN, Faster R-CNN, Mask R-CNN, SSD (Single Shot Multibook Detector), and YOLO (You Only It may be configured with any object detection algorithm such as Look Once), or it may be configured by combining several of these models. The control unit 11 of the information processing device 10 inputs the captured image captured by the endoscope 20 into the learning model described above, and determines the area of the treatment instrument 30 and gauze in the captured image based on the output information from the learning model. can be identified. Note that detection of gauze in a photographed image may be performed by pattern matching using a template, in addition to using a learning model. In this case, a template indicating the image feature amount of the gauze surface is stored in advance in the storage unit 12, and the control unit 11 determines whether or not there is an area matching the template from the captured image. Presence can be detected.

FIG. 16 is a flowchart illustrating an example of a procedure for bleeding detection processing in Modification 6, and FIGS. 17A to 17D are explanatory diagrams of bleeding detection processing in Modification 6. The process shown in FIG. 16 is the process shown in FIG. 2 with steps S81 to S83 added between steps S13 and S14. Description of the same steps as in FIG. 2 will be omitted.

In the information processing device 10 of this modification, the control unit 11 executes the processes of steps S11 to S13 in FIG. 2, and generates a bleeding area image indicating the bleeding area in the current frame (S13). Further, the control unit 11 inputs the photographed image acquired in step S11 to the learning model, and detects the treatment tool 30 and gauze in the photographed image based on the output information from the learning model (S81). For example, as shown in FIG. 17A, when a photographed image in which three treatment instruments 30 are captured is input to the learning model, as shown in FIG. 17B, the learning model Output the image. Therefore, the control unit 11 can grasp the area of each treatment tool 30 based on the treatment tool image output from the learning model. Note that the left side of FIG. 17B shows a treatment tool image showing the area of the treatment tool 30 in the previous frame (n-1 frame), and the right side shows the area of the treatment tool 30 in the current frame (n frame). A treatment tool image is shown.

When the control unit 11 detects the treatment instrument 30 or gauze in the current frame, the control unit 11 specifies the movement area of the detected treatment instrument 30 and gauze from the position in the previous frame to the position in the current frame (S82). . Specifically, the control unit 11 compares the area of the treatment instrument 30 and gauze in the previous frame with the area of the treatment instrument 30 and gauze in the current frame, and replaces the area existing only in the previous frame with the area of the treatment instrument 30 and gauze. and specific areas of gauze movement. For example, in FIG. 17C, a solid line indicates the area of the treatment instrument 30 and gauze in the current frame, and a broken line indicates the area of the treatment instrument 30 and gauze in the previous frame. In this case, as shown in FIG. 17D, the control unit 11 specifies a region that exists only in the front frame as a movement region of the treatment instrument 30 and the gauze. Note that if a treatment instrument image showing the area of the treatment instrument 30 and gauze in the previous frame has already been generated and stored in the storage unit 12, the control unit 11 stores the area of the treatment instrument 30 and gauze in the previous frame from the storage unit 12. You can get the location. Furthermore, the control unit 11 may obtain a treatment tool image indicating the region of the treatment tool 30 and gauze in the previous frame from the learning model by inputting the captured image of the previous frame into the learning model.

The control unit 11 excludes pixels included in the identified movement area of the treatment instrument 30 and gauze from the bleeding area image in the current frame generated in step S13 (S83 ). Thereby, the pixels of the bleeding area included in the moving area of the treatment instrument 30 and the gauze can be excluded from the bleeding occurrence determination process. After that, the control unit 11 performs the processing from step S14 onwards. In addition, in step S14, the control unit 11 performs the following based on the remaining bleeding area from which pixels included in the movement area of the treatment instrument 30 and gauze are excluded from the bleeding area included in the bleeding area image generated in step S13. The difference between the bleeding area in the previous frame and the bleeding area in the current frame is calculated (S14). Thereby, the bleeding area exposed by the movement of the treatment instrument 30 and the gauze can be excluded from the bleeding occurrence determination process, and erroneous detection of bleeding occurrence can be suppressed.

Also in this modification, the same effects as in the first embodiment described above can be obtained. Furthermore, in this modification, the bleeding area exposed due to the movement of the treatment tool 30 and the gauze is not used in the process of determining the occurrence of bleeding, thereby suppressing erroneous detection of the occurrence of bleeding. The configuration of this modification can be combined with the configurations of each modification described above, and the same effect can be obtained even when combined with each modification. Further, also in this embodiment, the modifications appropriately described in the above-described first embodiment and each modification can be applied.

(Modification 7)
In this modified example, the movement of each pixel is detected between frames that precede and follow in time series, and pixels within the bleeding area that have a large amount of movement (movement amount) are excluded from the targets used in the process to determine the occurrence of bleeding. The information processing device 10 will be explained. In endoscopic surgery, the procedure may be performed on the treatment area while moving the organ. For example, if an organ with blood is moved, the area of blood on the organ will move and the area after the movement will be different. There is a possibility that it will be mistakenly detected as a new bleeding area. Therefore, in this modification, by excluding pixels included in a bleeding area that have a large amount of movement between frames that precede and follow in time series from the targets when detecting bleeding occurrence, it is possible to detect the occurrence of bleeding incorrectly. Suppress detection.

FIG. 18 is a flowchart illustrating an example of a procedure for bleeding detection processing according to modification 7, and FIG. 19 is an explanatory diagram of the bleeding detection processing according to modification 7. The process shown in FIG. 18 is the process shown in FIG. 2 with steps S91 to S93 added between steps S13 and S14. Description of the same steps as in FIG. 2 will be omitted.

In the information processing device 10 of this modification, the control unit 11 executes the processes of steps S11 to S13 in FIG. 2, and generates a bleeding area image indicating the bleeding area in the current frame (S13). On the other hand, the control unit 11 detects the amount of movement of each pixel from the previous frame to the current frame based on the captured image acquired in step S11 and the captured image of the previous frame (S91). Specifically, the control unit 11 calculates a motion vector (amount and direction of motion) indicating the movement (movement) of each pixel from the previous frame to the current frame by image processing such as optical flow. FIG. 19 is an example showing a motion vector from the previous frame for each pixel in the current frame. In the motion vector, the direction of the arrow indicates the direction of movement, and the length of the arrow indicates the amount of movement.

Based on the motion vector of each pixel, the control unit 11 identifies pixels with a large movement amount, specifically, pixels whose movement amount is a predetermined amount or more (S92). As a result, pixels that have partially moved in the photographed image are identified. Note that the control unit 11 determines whether the amount of movement is equal to or greater than a predetermined amount only for each pixel included in the bleeding area in the bleeding area image generated in step S13, and determines whether the amount of movement is equal to or greater than a predetermined amount within the bleeding area. Large pixels may also be identified. Then, the control unit 11 excludes pixels that have a large amount of movement from the pixels included in the bleeding area in the bleeding area image in the current frame generated in step S13 (S93). Thereby, of the pixels included in the bleeding area, pixels that have a large amount of movement can be excluded from the bleeding occurrence determination process. After that, the control unit 11 performs the processing from step S14 onwards. In addition, in step S14, the control unit 11 determines the bleeding area in the previous frame and the bleeding area in the previous frame based on the remaining bleeding area from which pixels with a large movement amount are removed from the bleeding area included in the bleeding area image generated in step S13. The difference from the bleeding area in the current frame is calculated (S14). As a result, a bleeding area that has been moved due to movement of a bloody organ, for example, can be excluded from the bleeding occurrence determination processing, and erroneous detection of bleeding occurrence can be suppressed.

Also in this modification, the same effects as in the first embodiment described above can be obtained. Furthermore, in this modification, for example, by not using a bleeding area that has moved due to the movement of a bloody organ in the process of determining the occurrence of bleeding, it is possible to suppress erroneous detection of the occurrence of bleeding. The configuration of this modification can be combined with the configurations of each modification described above, and the same effect can be obtained even when combined with each modification. Further, also in this embodiment, the modifications appropriately described in the above-described first embodiment and each modification can be applied.

(Modification 8)
In this modification, an information processing apparatus 10 will be described in which an area in which blur or blur occurs in a captured image captured by an endoscope 20 is excluded from targets used in a process for determining the occurrence of bleeding. Since the endoscope 20 uses an image sensor to take pictures, blur or blur may occur, and it is difficult to accurately detect bleeding regions in areas where blur or blur occurs. Therefore, in this modification, the area where blurring or blurring occurs in the captured image is identified, the area where blurring or blurring occurs is excluded from the bleeding area in the captured image, and the occurrence of bleeding is detected. By excluding the blood from the target, false detection of bleeding occurrence can be suppressed. Since high-frequency components are not included in the area where the blur or blur occurs, in this modification, the high-frequency components are extracted from the captured image, and the areas in the captured image that do not include the high-frequency components are treated as low-frequency areas. Identify the area where the blur is occurring.

FIG. 20 is a flowchart showing an example of a procedure for bleeding detection processing in Modification 8, and FIGS. 21A to 21E are explanatory diagrams of bleeding detection processing in Modification 8. The process shown in FIG. 20 is the process shown in FIG. 2 with steps S101 to S104 added between steps S13 and S14. Description of the same steps as in FIG. 2 will be omitted.

In the information processing device 10 of this modification, the control unit 11 executes the processes of steps S11 to S13 in FIG. 2, and generates a bleeding area image indicating the bleeding area in the current frame (S13). On the other hand, the control unit 11 performs smoothing processing using a smoothing filter such as a Gaussian filter on the captured image acquired in step S11 (S101). As a result, a smoothed image as shown in FIG. 21B is generated from the photographed image shown in FIG. 21A, for example. Next, the control unit 11 subtracts the smoothed image from the photographed image to extract high frequency components in the photographed image (S102). Specifically, the control unit 11 subtracts the pixel value of each pixel of the smoothed image from the pixel value of each pixel of the captured image, and generates an image having only high frequency components as shown in FIG. 21C. .

The control unit 11 identifies a low frequency region in the captured image based on the image having only high frequency components (S103). For example, the control unit 11 performs binarization processing on an image having only high frequency components, and generates a binary image as shown in FIG. 21D. As a result, a binary image in which high frequency components are converted to pixels with a pixel value of 1 and low frequency components are converted to pixels with a pixel value of 0 is obtained. Further, the control unit 11 divides each pixel of the binary image into cells of, for example, 8 pixels x 8 pixels, 16 pixels x 16 pixels, etc., and in each cell, a pixel with a pixel value of 1 (a pixel value indicating a high frequency component) If the ratio is above a predetermined ratio (for example, 50% to 80%), the pixel values of all pixels included in the cell are converted to 1, and if it is less than the predetermined ratio, the pixel values of all pixels included in the cell are converted to 1. The value is converted to 0 to generate an image as shown in FIG. 21E. Then, based on the image shown in FIG. 21E, the control unit 11 specifies an area where there is no high frequency component as an area with a low frequency component in the photographed image. Note that since there is a possibility that a high frequency component is not included in an appropriate bleeding area, the control unit 11 considers the distribution of areas of low frequency components in the image shown in FIG. 21E, and selects an appropriate bleeding area. It is determined whether there is any blurring or blurring in the area. For example, in the image shown in FIG. 21E, the control unit 11 determines whether the outline of the low frequency component area is blurred or not, and if the outline is not blurred, it is determined that it is an appropriate bleeding area; It is determined that this is an area where blur or blur is occurring. As a result, in the image shown in FIG. 21E, the area surrounded by the broken line is specified as a low-frequency component area where blurring or blurring occurs.

The control unit 11 excludes the region of the low frequency component identified in step S103 from the bleeding region in the bleeding region image in the current frame generated in step S13 (S104). Thereby, among the pixels included in the bleeding area, the pixels in the area where blurring or blurring occurs can be excluded from the target of the bleeding occurrence determination process. After that, the control unit 11 performs the processing from step S14 onwards. In addition, in step S14, the control unit 11 determines the bleeding area in the previous frame based on the remaining bleeding area from which the blurred or blurred area is excluded from the bleeding area included in the bleeding area image generated in step S13. The difference between the bleeding area and the bleeding area in the current frame is calculated (S14). As a result, a region in which blur or blur has occurred among the bleeding regions can be excluded from the bleeding occurrence determination processing, and erroneous detection of bleeding occurrence can be suppressed.

Also in this modification, the same effects as in the first embodiment described above can be obtained. Furthermore, in this modification, by not using a bleeding area where blurring or blurring occurs in the bleeding occurrence determination process, it is possible to suppress erroneous detection of bleeding occurrence. The configuration of this modification can be combined with the configurations of each modification described above, and the same effect can be obtained even when combined with each modification. Further, also in this embodiment, the modifications appropriately described in the above-described first embodiment and each modification can be applied.

In the above-mentioned variations 1 to 8, when determining the presence or absence of bleeding based on the difference in the bleeding area between adjacent frames in time series, the bleeding area in each frame can be detected more accurately, and the bleeding between frames can be detected more accurately. This configuration allows for more accurate calculation of area differences. In the information processing device 10 of the first embodiment described above, modifications for accurately determining whether or not bleeding has occurred are not limited to the first to eighth modifications described above. For example, in the information processing device 10 of the first embodiment described above, erroneous detection may occur in which a red area such as an organ or a blood pool is mistakenly determined as a bleeding area. In order to suppress such false detections, for example, it is possible to determine whether the bleeding area is a true bleeding area or a non-bleeding area such as an organ or blood pool based on the size or shape of the bleeding area detected based on the captured image. May be determined. Specifically, in the process shown in FIG. 2, after the process in step S13, the control unit 11 determines whether the bleeding area is true based on the size or shape of the bleeding area in the bleeding area image generated in step S13. Processing may be performed to determine whether the area is a bleeding area or a non-bleeding area. For example, the bleeding area tends to spread from the bleeding point, so immediately after the bleeding occurs, the area is small and is not widely scattered. Therefore, if the bleeding area at the time of first detection is larger than a predetermined size, if the number of bleeding areas is larger than a predetermined number, or if multiple bleeding areas are spread over a wide area (for example, the entire image), the bleeding area If the shape is not a simple shape, there is a high possibility that it is not a bleeding area. Therefore, the control unit 11 determines whether or not each bleeding area in the bleeding area image is in any of the states described above, and determines that a bleeding area that is not in any of the states is a true bleeding area. Then, the control unit 11 executes the processes from step S14 onward for the bleeding area determined to be a true bleeding area. That is, the control unit 11 determines that a bleeding area in any of the above states is a non-bleeding area, and excludes the bleeding area from the bleeding occurrence determination target. As a result, the true bleeding area can be identified from the red area in the photographed image, so erroneous detection of the bleeding area is suppressed, and as a result, erroneous detection of the occurrence of bleeding is suppressed. Note that, in this embodiment, the configurations of the above-described modifications can be used in appropriate combinations as necessary, thereby making it possible to further improve the accuracy of determining the occurrence of bleeding.

(Embodiment 2)
An information processing device 10 that detects a bleeding area in a photographed image using a learning model will be described. The information processing apparatus of this embodiment has the same configuration as the information processing apparatus 10 of Embodiment 1, so a description of the configuration will be omitted. Note that the information processing device 10 of this embodiment stores in the storage unit 12 a learning model 12M (see FIG. 22) that has been trained with training data, for example, by machine learning. In the learning model 12M of this embodiment, when the pixel value of each pixel of a captured image captured by the endoscope 20 is input, whether this pixel is a pixel included in a bleeding area (bleeding pixel) or not. This is a trained model that has been trained to output information indicating whether a pixel is included in a bleeding area (non-bleeding pixel) (information regarding a bleeding area). The learning model 12M is assumed to be used as a program module that functions as part of artificial intelligence software. The learning model 12M performs predetermined calculations on input values and outputs the calculation results, and the storage unit 12 stores data such as coefficients and threshold values of functions that define this calculation as the learning model 12M. be done.

FIG. 22 is an explanatory diagram showing a configuration example of the learning model 12M. The learning model 12M shown in FIG. 22 includes pixel values of the R component (R component value), pixel values of the G component (G component value), and pixel values of the B component (B component value) is input, and based on the input RGB component values, a calculation is performed to classify whether the pixel is included in a bleeding area or a pixel included in a non-bleeding area, and the calculated result is calculated. It has been learned to output . The learning model 12M may be any function that outputs the probability that a pixel is included in a bleeding area from the RGB component values of one pixel, and may be an algorithm such as logistic regression or linear regression, or a neural network, etc. Constructed using

The learning model 12M determines whether the pixel is included in a bleeding area or in a non-bleeding area based on an input layer into which RGB component values of one pixel are input and the input RGB component values. An intermediate layer that classifies whether the pixel is a pixel, and an output layer that outputs information indicating whether the pixel is included in a bleeding area or a non-bleeding area based on the calculation results of the intermediate layer. and has. The intermediate layer calculates an output value from the RGB component values input via the input layer using various functions, threshold values, and the like. The output layer has two output nodes associated with each of the bleeding area and non-bleeding area, and from each output node, the probability (confidence) that the pixel should be classified as being included in the bleeding area, and outputs the probability (confidence) that the pixel should be classified as being included in a non-bleeding area. The output value from each output node of the output layer is, for example, a value between 0 and 1, and the sum of the probabilities output from each output node is 1.0 (100%).

With the above-described configuration, the learning model 12M of this embodiment, when the RGB component values of each pixel in an image is input, determines whether the pixel is included in a bleeding area or a non-bleeding area. Outputs an output value (confidence) that indicates. In the learning model 12M described above, the information processing device 10 identifies the output node that outputs the larger output value (confidence) among the output values from each output node, and determines that the bleeding area corresponds to the identified output node. The pixel of the input pixel value is classified as a pixel included in a bleeding area or a pixel included in a non-bleeding area, depending on whether the input pixel value is a pixel included in a bleeding area or a pixel included in a non-bleeding area. Note that the output layer of the learning model 12M has a plurality of output nodes that output the probability that a pixel should be classified as a pixel included in a bleeding region and the probability that a pixel should be classified as a pixel included in a non-bleeding region. , a configuration having one output node that outputs information indicating a region (bleeding region or non-bleeding region) with high classification probability (confidence) may be used.

The learning model 12M includes RGB component values (pixel values) for training and information (correct label) indicating whether this pixel is included in a bleeding area or a non-bleeding area. It can be generated by machine learning using training data. The training data is generated by an expert such as a doctor from an annotation image in which a bleeding area or a non-bleeding area is assigned to each pixel of an image taken with the endoscope 20. Specifically, the training data is generated by adding a correct label indicating a bleeding area to the RGB component values of pixels to which a bleeding area is assigned in an annotation image, and to the RGB component values of pixels to which a non-bleeding area is assigned. It is generated by adding a correct label indicating a non-bleeding area to the RGB component values of .

In the learning model 12M, when the RGB component values included in the training data are input, the output value from the output node corresponding to the correct label (bleeding area or non-bleeding area) included in the training data approaches 1, Learn so that the output values from other output nodes approach 0. In the learning process, the learning model 12M performs calculations in the intermediate layer based on input RGB component values, and calculates output values from each output node. The learning model 12M compares the calculated output value of each output node with the value corresponding to the correct label (1 for the output node corresponding to the correct label, 0 for other output nodes), and compares both. The parameters used for calculation processing in the intermediate layer are optimized so that The parameters include weights (coupling coefficients) between nodes in the intermediate layer. The parameter optimization method is not particularly limited, but an error backpropagation method, steepest descent method, etc. can be used. As a result, when an RGB pixel value is input, a learning model 12M that predicts whether this pixel is in a bleeding area or a non-bleeding area and outputs a prediction result is obtained.

Learning of the learning model 12M may be performed by another learning device. The trained learning model 12M generated by performing learning on another learning device is downloaded from the learning device to the information processing device 10 via the Internet or the portable storage medium 10a, and is stored in the storage unit 12, for example. . The learning model 12M is not limited to the configuration shown in FIG. 22. For example, in addition to the R component value, G component value, and B component value of each pixel of the captured image, the value obtained by subtracting the G component value from the R component value (RG), and the value obtained by dividing the G component value from the R component value A configuration in which feature quantities such as a value obtained by dividing the B component value from the R component value (R/B), etc., may be input, and a combination of multiple feature quantities among these feature quantities may be input. The configuration may be input. Further, the learning model 12M may have a configuration in which each feature amount of one pixel in a photographed image is inputted, or a configuration in which in addition to the target pixel, each feature amount of pixels in the vicinity of the target pixel is inputted. In this case, it is possible to classify whether the area is a bleeding area or a non-bleeding area from each pixel value in a minute area including the target pixel. In addition, in the case of such a configuration, the learning model 12M may be configured using algorithms such as SVM (support vector machine), decision tree, random forest, etc., or may be configured by combining multiple algorithms. .

The process of generating the learning model 12M will be described below. FIG. 23 is a flowchart showing an example of the procedure for generating the learning model 12M. The following process is executed by the control unit 11 of the information processing device 10 according to the program 12P stored in the storage unit 12, but may be executed by another learning device. In the following process, the control unit 11 first generates training data based on the annotation image, and performs a learning process on the learning model 12M using the generated training data. It is assumed that the annotation image is stored in the storage unit 12 in advance, for example.

The control unit 11 of the information processing device 10 reads one annotation image from the storage unit 12 (S111). The control unit 11 extracts the pixel value of one pixel in the annotation image (S112), and determines whether the extracted pixel is a pixel to which a bleeding area has been assigned (S113). If it is determined that the pixel is a pixel to which a bleeding area is assigned (S113: YES), the control unit 11 generates training data by adding a correct label (bleeding label) indicating a bleeding area to the extracted pixel value ( S114). If it is determined that the pixel is not assigned to a bleeding area (S113: NO), the control unit 11 generates training data by adding a correct label (non-bleeding label) indicating a non-bleeding area to the extracted pixel value. (S115). The control unit 11 stores the generated training data in, for example, a training DB (not shown) prepared in the storage unit 12 (S116).

The control unit 11 determines whether there are any pixels (unprocessed pixels) that are not used in the training data generation process among the pixels in the annotation image read out in step S111 (S117). If it is determined that there are unprocessed pixels (S117: YES), the control unit 11 returns to the process of step S112 and performs the processes of steps S112 to S116 for the unprocessed pixels. As a result, training data is accumulated in which the pixel value of each pixel in the annotation image is given a correct label (information regarding the bleeding area) indicating whether each pixel is a pixel in a bleeding area or a pixel in a non-bleeding area. be done.

If it is determined that there are no unprocessed pixels (S117: NO), the control unit 11 selects images that are not used in the training data generation process (unprocessed pixels) among the annotation images stored in the storage unit 12. image) is present (S118). If it is determined that there is an unprocessed image (S118: YES), the control unit 11 returns to the process of step S111 and performs the processes of steps S111 to S117 for the unprocessed annotation image. Thereby, training data used for learning the learning model 12M can be generated and stored in the training DB based on the prepared annotation image.

When the control unit 11 determines that there are no unprocessed images (S118: NO), the control unit 11 performs learning of the learning model 12M using the training data accumulated in the training DB as described above. Specifically, the control unit 11 reads out one of the training data accumulated in the training DB through the process described above (S119). Then, the control unit 11 performs a learning process on the learning model 12M based on the training data (S120). Here, the control unit 11 inputs RGB component values included in the training data to the learning model 12M, and acquires an output value output from the learning model 12M by inputting the RGB component values. The learning model 12M performs calculations based on input RGB component values and calculates output values from each output node. The control unit 11 outputs the output value of each output node output from the learning model 12M and a value corresponding to the correct label included in the training data (1 for the output node corresponding to the correct label, and 1 for the other output nodes). 0), and the learning model 12M is trained so that the two approximate each other. In the learning process, the learning model 12M optimizes parameters used for calculation processing in the intermediate layer. For example, the control unit 11 optimizes parameters such as weights (coupling coefficients) between nodes in the intermediate layer using an error backpropagation method that sequentially updates parameters from the output layer to the input layer of the learning model 12M.

The control unit 11 determines whether or not there is unprocessed training data that has not been subjected to learning processing among the training data stored in the training DB (S121). If it is determined that there is unprocessed training data (S121: YES), the control unit 11 returns to the process of step S119, and performs the processes of steps S119 to S120 on the training data that has not been subjected to the learning process. If it is determined that there is no unprocessed training data (S121: NO), the control unit 11 ends the series of processes. Through the learning process described above, when RGB pixel values are input, the probability that the input pixel value is a pixel in a bleeding area (output value from output node 0) and the probability that a pixel with the input pixel value is a pixel in a non-bleeding area are determined. A learning model 12M that outputs a certain possibility (output value from output node 1) is generated.

The learning model 12M can be further optimized by repeatedly performing learning processing using training data as described above. Further, by relearning the already trained learning model 12M using the above-described learning process, a learning model 12M with further improved estimation accuracy can be generated. In the above-described process, the training data generation process in steps S111 to S118 and the learning model 12M generation process in steps S119 to S121 may be performed by separate devices.

Hereinafter, a process of detecting bleeding based on an image taken by the endoscope 20 using the learning model 12M generated by the above-described process will be described. FIG. 24 is a flowchart showing an example of a bleeding detection processing procedure according to the second embodiment, and FIGS. 25A to 26 are explanatory diagrams of the bleeding detection processing according to the second embodiment. The process shown in FIG. 24 is the process shown in FIG. 2 with step S131 added instead of step S12. Description of the same steps as in FIG. 2 will be omitted.

In the information processing device 10 of the present embodiment, when a photographed image photographed by the endoscope 20 is acquired from the endoscope control device 21 (S11), the control unit 11 controls the pixel values (RGB component values) are input into the learning model 12M and classified into bleeding areas or non-bleeding areas based on the output values from the learning model 12M (S131). Specifically, the control unit 11 identifies the larger output value among the output values from the learning model 12M, and identifies the area (bleeding area or A non-bleeding area) is specified in this pixel area. The control unit 11 (bleeding area acquisition unit) generates a bleeding area image indicating the bleeding area in the captured image based on the classification result of each pixel (S13). Here, the control unit 11 generates a bleeding area image by assigning white (for example, 1) to pixels classified as bleeding areas and black (for example, 0) to pixels classified as non-bleeding areas. do. The control unit 11 generates a bleeding area image as shown in FIG. 25B, for example, based on the photographed image shown in FIG. 25A. The control unit 11 performs the above-described processing of steps S11, S131, and S13 on the captured images of each frame sequentially transmitted from the endoscope control device 21, and generates a bleeding area image from each captured image. .

After that, the control unit 11 performs the processing from step S14 onwards. In this embodiment, in step S16, the control unit 11 calculates a value of +2σ with respect to the average value of the measured values, using the moving average of the differences in bleeding areas calculated up to each imaging time as the measured value, and calculates the value of +2σ with respect to the average value of the measured values. It may also be set as a determination threshold for determining whether or not the occurrence has occurred. The solid line in FIG. 26 shows the time-series change in the difference in the bleeding area between frames calculated in step S14, and the broken line in FIG. It shows the time series change with the value of. In this embodiment, the control unit 11 (determination unit) determines that bleeding has occurred when the difference in bleeding areas between frames is equal to or greater than the value of +2σ with respect to the moving average of the differences in bleeding areas. Therefore, the control unit 11 sets each value of the upper (+2σ) time series data indicated by the broken line in FIG. 26 as the threshold value used when determining the presence or absence of bleeding occurrence for each imaging time. Thereby, the presence or absence of bleeding occurrence is determined based on the bleeding area that is accurately detected from each captured image using the learning model 12M. Note that in this embodiment as well, instead of the value of the moving average of the differences in bleeding areas +2σ, the value of the moving average of the differences in bleeding areas +4σ may be used as the threshold for determining the presence or absence of bleeding. With such a configuration, also in this embodiment, erroneous detection of bleeding occurrence is suppressed, and the detection accuracy of bleeding occurrence is improved.

The endoscopic surgery system of this embodiment is the same as the processing performed by the information processing device 10 of Embodiment 1, except that the learning model 12M is used when detecting a bleeding area in a photographed image. The same effect as 1 can be obtained. Furthermore, in the present embodiment, the learning model 12M classifies each pixel in the photographed image into a bleeding area and a non-bleeding area according to the characteristics of each pixel. Therefore, the user using the learning model 12M can detect bleeding regions in the photographed image with high accuracy using the learning model 12M by generating an annotation image from the photographed image and having the learning model 12M learn. Therefore, also in this embodiment, the presence or absence of bleeding can be detected with high accuracy based on the photographed image taken with the endoscope 20.

Also in this embodiment, the variations 1 to 8 described in the first embodiment can be applied. Therefore, the information processing device 10 of the present embodiment uses the learning model 12M to accurately detect bleeding areas in photographed images, and also detects each bleeding area by applying one or more of the configurations of Modifications 1 to 8. It is possible to realize a configuration that more accurately detects bleeding areas in frames and more accurately calculates differences in bleeding areas between frames. Also in this embodiment, the modifications appropriately described in the above-described first embodiment and each modification can be applied.

The learning model 12M of this embodiment is configured to classify whether each pixel is a bleeding area pixel or a non-bleeding area pixel based on the pixel value of each pixel in a captured image. but not limited to. For example, the learning model 12M may be configured to detect a bleeding area in the captured image when the captured image is input. In this case, for example, the learning model 12M can be configured with an object detection algorithm such as CNN, R-CNN, Fast R-CNN, SSD, or YOLO. Furthermore, the learning model 12M may be configured with an algorithm that implements semantic segmentation, such as SegNet, FCN (Fully Convolutional Network), and U-Net.

(Embodiment 3)
An information processing device 10 that is a modification of the second embodiment that detects a bleeding area in a captured image using the learning model 12M will be described. The information processing apparatus of this embodiment has the same configuration as the information processing apparatus 10 of Embodiment 2, so a description of the configuration will be omitted. The information processing device 10 of this embodiment detects a bleeding area in a photographed image using the learning model 12M, and then generates an image showing the difference (increase) in the bleeding area between frames that follow in time series. Here, the information processing device 10 generates an image showing a bleeding area in the current frame that was not a bleeding area in the previous frame. Then, the information processing device 10 divides each pixel into cells of, for example, 8 pixels x 8 pixels, 16 pixels x 16 pixels, etc. in the image showing the difference (increase) in the bleeding area, and in each cell, the pixel value is The number of pixels (bleeding pixels) with a value of 1 (pixel value indicating a bleeding area) is counted. Then, in each cell, the information processing device 10 determines whether or not each cell is a bleeding area according to the ratio of the number of bleeding pixels to the number of pixels included in the cell, and from the result, determines whether or not bleeding has occurred. Determine.

FIG. 27 is a flowchart illustrating an example of a bleeding detection processing procedure according to the third embodiment, and FIGS. 28A to 28C are explanatory diagrams of the bleeding detection processing according to the third embodiment. The process shown in FIG. 27 is the process shown in FIG. 24 with steps S141 to S144 added instead of steps S14 to S16. Description of the same steps as in FIGS. 2 and 24 will be omitted.

In the information processing device 10 of this embodiment, the control unit 11 executes the processes of steps S11, S131, and S13 in FIG. 24, and generates a bleeding area image indicating the bleeding area in the current frame (S13). Then, the control unit 11 (extraction unit) identifies the area (increased bleeding area) that was not a bleeding area in the previous frame among the bleeding areas in the current frame (frame to be processed), and An image showing the increase (difference) in the bleeding area between the preceding and succeeding frames is generated (S141). For example, the control unit 11 generates an image in which the pixel values of pixels in the increased bleeding area are set to 1, and the pixel values of other pixels are set to 0, as shown in FIG. 28A.

Next, the control unit 11 (pixel counting unit) divides each pixel into cells of 8 pixels x 8 pixels, 16 pixels x 16 pixels, etc. in the image showing the increase in the bleeding area, and calculates the pixel value for each cell. The number of pixels (bleeding pixels) where is 1 (pixel value indicating a bleeding area) is counted (S142). FIG. 28B shows the image shown in FIG. 28A in which the pixel values of all pixels in each cell are converted into pixel values according to the number of bleeding pixels in each cell, and the whiter the cell, the larger the number of bleeding pixels. , and the number of bleeding pixels in each cell is represented by a monochrome gradation.

Next, the control unit 11 calculates, for each cell, the ratio of the number of bleeding pixels to the number of pixels included in each cell in the image showing the increase in the bleeding area (S143). Then, the control unit 11 (determination unit) determines each cell as a bleeding area or a non-bleeding area based on the ratio of the number of bleeding pixels calculated for each cell (S144). Here, for each imaging time, the control unit 11 determines whether the ratio of the number of bleeding pixels in each cell calculated in step S143 is equal to or higher than a predetermined ratio (for example, 50% to 80%), and If the ratio is above, it is determined to be a bleeding area, and if it is less than a predetermined ratio, it is determined to be a non-bleeding area. Thereby, the control unit 11 generates a binary image as shown in FIG. 28C. Then, the control unit 11 determines whether bleeding has occurred based on the determination result in step S144 (S17). In this embodiment, the control unit 11 determines whether or not there is a cluster of cells determined in the bleeding area in step S144, and if there is, it is determined that bleeding has occurred, and if there is not, it is determined that bleeding has not occurred. . For example, the control unit 11 counts the number of adjacent cells among the cells determined to be a bleeding area, and if the number is equal to or greater than a predetermined number, the area (cluster of cells) including this cell is determined as a bleeding area. Therefore, in the image shown in FIG. 28C, among the cells determined to be bleeding regions, those cells for which the number of adjacent cells is less than a predetermined number are determined to be non-bleeding regions. As a result, in this embodiment, it is determined whether or not each cell is a bleeding area, and if a predetermined number or more of cells that have been determined to be bleeding areas are adjacent to each other, this area is identified as a bleeding area. , Based on the identified results, the presence or absence of bleeding is detected. After that, the control unit 11 performs the processing from step S18 onwards.

In this embodiment, the number of pixels that change from non-bleeding pixels to bleeding pixels is counted for each cell in a captured image, and cells in which this number of pixels (bleeding pixel count) exceeds a predetermined percentage are identified as bleeding areas. Other than this, the processing is the same as the processing performed by the information processing device 10 of the second embodiment, so that the same effects as the second embodiment can be obtained. Furthermore, in this embodiment, since a bleeding area or a non-bleeding area is determined for each cell, a bleeding area can be detected with high accuracy by determining a cell where pixels determined to be a bleeding area are gathered as a bleeding area. Erroneous detection of bleeding areas can be suppressed.

Also in this embodiment, it is possible to apply any or more of Modifications 1 to 8 described in Embodiment 1, and similar effects can be obtained by applying the configurations of Modifications 1 to 8. Further, also in this embodiment, the modifications described in each of the above-described embodiments and modifications can be applied.

(Embodiment 4)
In the first embodiment described above, a bleeding area in a photographed image is identified based on a difference image of color components in the photographed image (for example, a difference between an R component and a G component, or a difference between an R component and a B component). , the occurrence of bleeding is determined based on the identified bleeding area. In the second embodiment, the learning model 12M is used to identify a bleeding area in a captured image, and the occurrence of bleeding is determined based on the identified bleeding area. In the third embodiment, after identifying a bleeding area in a captured image using the learning model 12M, a bleeding area or a non-bleeding area is determined for each cell according to the ratio of the number of bleeding pixels, and based on the determination result for each cell. This configuration identifies a bleeding area in a photographed image and determines whether bleeding has occurred based on the identified bleeding area. In this embodiment, it is possible to execute the process in Embodiment 1 (first determination process), the process in Embodiment 2 (second determination process), and the process in Embodiment 3 (third determination process). An information processing apparatus 10 that executes at least one determination process among the three processes and identifies the presence or absence of bleeding occurrence based on the result of the executed determination process will be described.

FIG. 29 is a flowchart illustrating an example of a bleeding detection processing procedure according to the fourth embodiment. The process shown in FIG. 29 is the process shown in FIG. 2 with steps S151 to S153 added instead of steps S11 to S16. Description of the same steps as in FIG. 2 will be omitted.

In the information processing device 10 of the present embodiment, the control unit 11 selects a determination process for determining the occurrence of bleeding based on the captured image to be processed (S151). For example, the control unit 11 selects at least one of a plurality of types of determination processing depending on the type, format, imaging method, or surgical method of the image (video) taken by the endoscope 20. In this embodiment, the process for determining the occurrence of bleeding includes the process shown in FIG. 2 described in Embodiment 1 (first determination process) and the process shown in FIG. 24 described in Embodiment 2 (second determination process). It is possible to execute the process shown in FIG. 27 (third determination process) described in the third embodiment, and the control unit 11 selects at least one of the three determination processes.

Next, the control unit 11 executes the selected determination process on the photographed image (S152). Here, when the first determination process is selected, the control unit 11 executes the processes of steps S11 to S18 in FIG. Determine whether bleeding has occurred. Further, when the second determination process is selected, the control unit 11 executes the processes of steps S11, S131, S13 to S18 in FIG. 24, and detects a bleeding area in the captured image using the learning model 12M. , the presence or absence of bleeding is determined based on the difference (increase amount) in the bleeding area between frames that precede and follow in time series. Further, when the third determination process is selected, the control unit 11 executes the processes of steps S11, S131, S13, S141 to S144, and S17 to S18 in FIG. The presence or absence of bleeding is determined based on the bleeding area image in which the area difference (increase amount) is determined for each cell.

Based on the determination result (presence or absence of bleeding occurrence) of the determination process selected in step S151, the control unit 11 determines the probability (certainty) that it should be determined that bleeding has occurred, or the probability (certainty) that it should be determined that bleeding has not occurred. degree) is calculated (S153). For example, if three types of determination processes are executed and two types of determination processes yield a determination result that bleeding has occurred, the control unit 11 calculates 2/3 as the confidence level for determining that bleeding has occurred. . Note that the determination results of each determination process may be weighted to calculate the certainty that bleeding has occurred or the certainty that bleeding has not occurred.

Then, the control unit 11 identifies (determines) the presence or absence of bleeding in the current frame based on the calculated certainty factor (S17). For example, the control unit 11 specifies the determination result as the occurrence of bleeding when the confidence level of occurrence of bleeding is equal to or higher than a predetermined value (for example, a value of 0.5 to 0.8), and when it is less than the predetermined value, the control unit 11 determines that Identify the outcome without bleeding. After that, the control unit 11 performs the processing from step S18 onwards.

In this embodiment, it is possible to execute the determination process performed by the information processing apparatus 10 of the above-described embodiments 1 to 3, and by comprehensively determining the presence or absence of bleeding from the determination results of the three determination processes, it is possible to It is possible to realize a determination process that provides stable detection results. Further, since the three determination processes can be used depending on the type of photographed image, etc., the accuracy of determining the occurrence of bleeding can be further improved by performing the determination process suitable for the photographed image.

(Embodiment 5)
The information processing apparatus 10 according to the first to fourth embodiments determines whether or not a hemostasis operation has been performed when the occurrence of bleeding is detected. In addition to the configuration of Embodiment 1 shown in FIG. 1, information processing device 10 of this embodiment stores a learning model similar to Modification 6 in storage unit 12. The learning model of this embodiment is configured to, when a captured image is input, output the area of the treatment tool 30 used for hemostasis based on the captured image. Note that the learning model 12M of this embodiment receives a plurality of captured images (videos) as input, and performs a calculation to classify whether or not a hemostasis operation has been performed in the captured images based on the input captured images. , may be configured to output the calculated results. The learning model in this case can be constructed using algorithms such as RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), and Transformer.

FIG. 30 is a flowchart illustrating an example of a bleeding detection processing procedure according to the fifth embodiment. The process shown in FIG. 30 is the process shown in FIG. 2 with steps S161 to S163 added between steps S18 and S19. Description of the same steps as in FIG. 2 will be omitted. Note that in FIG. 30, illustrations of steps other than steps S17 to S19 in FIG. 2 are omitted. In the information processing device 10 of the present embodiment, when it is determined that bleeding has occurred (S17: YES), the control unit 11 executes the process of step S18, and then performs the process to stop the bleeding based on the captured image acquired in step S11. It is determined whether execution of an operation is detected (S161). Specifically, the control unit 11 inputs the photographed image into a learning model, and determines whether or not there is a treatment tool 30 for hemostasis (hemostasis device) in the photographed image based on output information from the learning model. do. Note that if the learning model is configured to determine whether or not a hemostasis operation is being performed based on the input captured image, the control unit 11 determines whether or not a hemostasis operation is being performed based on the output information from the learning model. You can determine whether it is being done or not.

If it is determined that the execution of the hemostasis operation is not detected (S161: NO), the control unit 11 repeats the process of step S161 until the execution of the hemostasis operation is detected. Here, the control unit 11 continues the process of determining whether or not the implementation of a hemostasis operation has been detected, based on the captured images sequentially transmitted from the endoscope control device 21. If it is determined that the execution of the hemostasis operation has been detected (S161: YES), the control unit 11 determines whether or not the end of the hemostasis operation has been detected (S162). Here, too, the control unit 11 inputs the photographed image acquired from the endoscope control device 21 into the learning model, and determines whether the hemostasis treatment tool 30 is missing from the photographed image based on the output information from the learning model. to judge. If it is determined that the end of the hemostasis operation has not been detected (S162: NO), the control unit 11 repeats the process of step S162 until the end of the hemostasis operation is detected. Here, the control unit 11 continues the process of determining whether or not the end of the hemostasis operation has been detected, based on the captured images sequentially transmitted from the endoscope control device 21. Note that at this time, the control unit 11 may perform a process of notifying a medical worker that a hemostasis operation is in progress, for example.

If it is determined that the end of the hemostasis operation has been detected (S162: YES), that is, if the removal of the hemostasis device is detected after the hemostasis device is detected in the photographed image, the control unit 11 detects the end of the hemostasis operation. It is determined whether a predetermined time has elapsed since then (S163). If the control unit 11 determines that the predetermined time has not elapsed (S163: NO), the process returns to step S162, and the hemostasis operation is completed until the predetermined time elapses after the end of the hemostasis operation is first detected. Repeat the detection process. The predetermined time here is a time long enough to determine that hemostasis has been successful. If it is determined that the predetermined time has elapsed (S163: YES), the control unit 11 determines that the hemostasis has been successful, and proceeds to the process of step S19. At this time, the control unit 11 may notify the medical personnel that hemostasis has been successful. Thereby, when the occurrence of bleeding is detected based on the photographed image, it can be determined whether a hemostasis operation has been performed. Therefore, it is possible to objectively judge state transitions such as a bleeding state, hemostasis in progress, and successful hemostasis, and notify the medical personnel. Medical personnel, including surgeons, only have to carry out the optimal treatment for each condition, and it becomes possible to support medical personnel and reduce their burden.

Also in this embodiment, it is possible to apply any or more of Modifications 1 to 8 described in Embodiment 1, and similar effects can be obtained by applying the configurations of Modifications 1 to 8. Furthermore, the configuration of this embodiment is applicable to the information processing apparatus 10 of the first to fourth embodiments described above, and similar effects can be obtained even when applied to the information processing apparatus 10 of the first to fourth embodiments. It will be done. Note that when applied to the information processing apparatus 10 of embodiments 2 to 4, the control unit 11 performs the processing of steps S161 to S163 between steps S18 and S19 in each of the processes shown in FIGS. 24, 27, and 29. All you have to do is execute. Further, also in this embodiment, the modifications described in each of the above-described embodiments and modifications can be applied.

The embodiments disclosed herein are illustrative in all respects and should not be considered restrictive. The scope of the present invention is not defined by the above-mentioned meaning, but is indicated by the scope of the claims, and is intended to include meanings equivalent to the scope of the claims and all changes within the scope.

10 information processing device 11 control unit 12 storage unit 13 operation unit 14 communication unit 20 endoscope 21 endoscope control device 30 treatment tool 31 treatment tool control device 40 display device 12M learning model

Claims

Obtain multiple images of the treatment area in chronological order that have multiple color components,
For each of the plurality of acquired captured images, based on the difference between the plurality of color components, obtain a bleeding area image indicating the bleeding area in the captured image,
Calculating the difference in the bleeding area between the bleeding area images acquired for each of the images taken before and after in time series,
A program that causes a computer to execute a process of determining whether or not there is bleeding from the treatment site based on the difference in the bleeding area.
A method for causing the computer to execute a process of determining that there is bleeding from the treatment site when a difference in the bleeding area between the images of the bleeding area exceeds a predetermined range based on a moving average of the difference in the bleeding area. The program described in 1.
Acquire multiple images of the treatment area taken in chronological order,
When a photographed image of a treatment area is input, each of the acquired plurality of photographed images is input to a learning model that has been trained to output information regarding the bleeding area in the photographed image. Get information about the bleeding area,
Calculating the difference in the bleeding area between the photographed images based on information regarding the bleeding area acquired for each of the photographed images before and after the photographed images in time series,
A program that causes a computer to execute a process of determining whether or not there is bleeding from the treatment site based on the difference in the bleeding area.
Acquire multiple images of the treatment area taken in chronological order,
When a photographed image of a treatment area is input, each of the acquired plurality of photographed images is input to a learning model that has been trained to output information regarding the bleeding area in the photographed image. Get information about the bleeding area,
Based on the information regarding the bleeding area acquired for each of the photographed images before and after the photographed images in time series, extracting the increasing area of the bleeding area between the photographed images,
For each cell obtained by dividing the photographed image into cells each having a predetermined number of pixels, count the number of pixels included in the increased area,
A program that causes a computer to execute a process of determining the presence or absence of bleeding from the treatment site based on the number of pixels included in the increased area counted for each cell.
causing the computer to execute a process of specifying, as a bleeding area, a cell in which the number of pixels included in the increased area counted for each cell in the photographed image is equal to or greater than a predetermined ratio to a predetermined number of pixels included in each cell; The program according to claim 4.
Obtaining the amount of change in the photographing position when photographing the photographed image,
The program according to any one of claims 1 to 5, which causes the computer to execute a process of performing image correction on the photographed image based on the amount of change in the photographing position.
Detecting the amount of movement of the feature point in the captured image based on the captured images that are taken before and after in time series,
The program according to any one of claims 1 to 6, which causes the computer to execute a process of performing image correction on the photographed image based on the amount of movement of the feature point.
Detecting whether the magnification was changed when the photographed image was taken;
8. The program according to claim 1, which causes the computer to execute a process of invalidating the determination result of the presence or absence of bleeding based on images taken before and after the magnification is changed.
detecting the center of gravity of the bleeding area in the photographed image;
Identify the locus of the center of gravity of the bleeding area in the images taken before and after the time series,
Any one of claims 1 to 8, wherein the computer executes a process of excluding the bleeding area from the determination target for the presence or absence of bleeding when the locus of the center of gravity of the bleeding area is included in the peripheral area of the photographed image. The program described in.
Determining whether mist has occurred based on the difference in the bleeding area between the photographed images,
The program according to any one of claims 1 to 9, which causes the computer to execute a process of excluding the photographed image from the determination target for the presence or absence of bleeding when it is determined that mist has occurred.
identifying the area of the treatment tool in the photographed image;
Detecting a movement area of the treatment instrument in the photographed image based on the photographed images that precede and follow in chronological order,
The program according to any one of claims 1 to 10, which causes the computer to execute a process of excluding pixels included in a movement area of the treatment instrument from the bleeding area in the captured image.
Detecting the amount of movement of each pixel in the photographed image based on the photographed images that come before and after in time series,
The program according to any one of claims 1 to 11, which causes the computer to execute a process of excluding pixels whose movement amount is equal to or greater than a predetermined value from the bleeding area in the captured image.
extracting a low frequency region in the captured image;
13. The program according to claim 1, which causes the computer to execute a process of excluding pixels included in the low frequency region from the bleeding region in the photographed image.
Determining whether the bleeding area is a non-bleeding area based on the number, size, or shape of the bleeding area in the photographed image,
The program according to any one of claims 1 to 13, which causes the computer to execute a process of excluding the bleeding area determined to be a non-bleeding area from the determination target of the presence or absence of bleeding.
detecting the presence or absence of a hemostatic device in the captured image;
When it is determined that there is bleeding from the treatment site, determining whether removal of the hemostatic device is detected after detecting the hemostatic device in the captured image,
15. The program according to claim 1, which causes the computer to execute a process of determining whether or not there is bleeding from the treatment site when a predetermined period of time has elapsed since removal of the hemostatic device was detected.
Obtain multiple images of the treatment area in chronological order that have multiple color components,
For each of the plurality of acquired captured images, based on the difference between the plurality of color components, obtain a bleeding area image indicating the bleeding area in the captured image,
Calculating the difference in the bleeding area between the bleeding area images acquired for each of the images taken before and after in time series,
An information processing method in which a computer executes a process of determining the presence or absence of bleeding from the treatment site based on the difference in the bleeding area.
Acquire multiple images of the treatment area taken in chronological order,
When a photographed image of a treatment area is input, each of the acquired plurality of photographed images is input to a learning model that has been trained to output information regarding the bleeding area in the photographed image. Get information about the bleeding area,
Calculating the difference in the bleeding area between the photographed images based on information regarding the bleeding area acquired for each of the photographed images before and after the photographed images in time series,
An information processing method in which a computer executes a process of determining the presence or absence of bleeding from the treatment site based on the difference in the bleeding area.
Acquire multiple images of the treatment area taken in chronological order,
When a photographed image of a treatment area is input, each of the acquired plurality of photographed images is input to a learning model that has been trained to output information regarding the bleeding area in the photographed image. Get information about the bleeding area,
Based on the information regarding the bleeding area acquired for each of the photographed images before and after the photographed images in time series, extracting the increasing area of the bleeding area between the photographed images,
For each cell obtained by dividing the photographed image into cells each having a predetermined number of pixels, count the number of pixels included in the increased area,
An information processing method in which a computer executes a process of determining the presence or absence of bleeding from the treatment site based on the number of pixels included in the increased area counted for each cell.
a photographed image acquisition unit that acquires a plurality of photographed images having a plurality of color components, photographing the treatment area in chronological order;
a bleeding area acquisition unit that acquires a bleeding area image indicating a bleeding area in the captured image based on the difference between the plurality of color components for each of the plurality of captured images;
a calculation unit that calculates a difference in the bleeding area between the bleeding area images acquired for each of the images taken before and after in time series;
An information processing apparatus comprising: a determination unit that determines whether or not there is bleeding from the treatment site based on the difference in the bleeding area.
a photographed image acquisition unit that acquires a plurality of photographed images of the treatment area taken in chronological order;
When a photographed image of a treatment area is input, each of the acquired plurality of photographed images is input to a learning model that has been trained to output information regarding the bleeding area in the photographed image. a bleeding area acquisition unit that acquires information about the bleeding area;
a calculation unit that calculates a difference in the bleeding area between the photographed images based on information regarding the bleeding area acquired for each of the photographed images that precede and follow in time series;
An information processing apparatus comprising: a determination unit that determines whether or not there is bleeding from the treatment site based on the difference in the bleeding area.
a photographed image acquisition unit that acquires a plurality of photographed images of the treatment area taken in chronological order;
When a photographed image of a treatment area is input, each of the acquired plurality of photographed images is input to a learning model that has been trained to output information regarding the bleeding area in the photographed image. a bleeding area acquisition unit that acquires information about the bleeding area;
an extraction unit that extracts an area where the bleeding area increases between the photographed images based on information regarding the bleeding area acquired for each of the photographed images that precede and follow in time series;
a pixel counting unit that counts the number of pixels included in the increased area for each cell obtained by dividing the photographed image into cells each having a predetermined number of pixels;
An information processing device comprising: a determining unit that determines whether or not there is bleeding from the treatment site based on the number of pixels included in the increased area counted for each cell.
Obtain multiple images of the treatment area in chronological order that have multiple color components,
For each of the plurality of acquired captured images, a bleeding area image indicating the bleeding area in the captured image is acquired based on the difference between the plurality of color components, and the bleeding area image obtained for each of the captured images before and after in time series is obtained. a first determination process of calculating a difference in bleeding areas between bleeding area images and determining the presence or absence of bleeding from the treatment site based on the difference in bleeding areas;
When a photographed image of a treatment area is input, each of the acquired plurality of photographed images is input to a learning model that has been trained to output information regarding the bleeding area in the photographed image. Obtaining information regarding the bleeding area, calculating a difference in the bleeding area between the captured images based on the information regarding the bleeding area acquired for each of the captured images before and after the time series, and based on the difference in the bleeding area, a second determination process for determining the presence or absence of bleeding from the treatment site;
When a photographed image of a treatment area is input, each of the acquired plurality of photographed images is input to a learning model that has been trained to output information regarding the bleeding area in the photographed image. Information regarding the bleeding area is acquired, and based on the information regarding the bleeding area acquired for each of the photographed images before and after the photographed images, an area where the bleeding area increases between the photographed images is extracted, and the photographed image is divided into a predetermined number of pixels. A step of counting the number of pixels included in the increased area for each divided cell, and determining whether there is bleeding from the treatment site based on the number of pixels included in the increased area counted for each cell. 3. Select at least one judgment process with the judgment process,
Execute the selected judgment process,
An information processing method in which a computer executes a process of specifying the presence or absence of bleeding from the treatment site based on a determination result of an executed determination process.