Classifications

• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
  • A61B1/045—Control thereof

Definitions

• This disclosure relates to a medical support device, an endoscope device, a medical support method, and a program.
• Patent No. 6875038 discloses a method for analyzing multiple frame images that make up an image obtained from an endoscopic camera to determine whether or not a hemostatic instrument is included in the surgical field image. Furthermore, the technology described in Patent No. 6875038 plays a partial video if the surgical field image does not include a hemostatic instrument, and controls so that the partial video is not played if a hemostatic instrument appears in the surgical field image.
• JP 2016-062488 A describes a mode in which a treatment tool used inside a patient's body during an endoscopic examination is detected by image processing from frame images included in a video captured inside the patient's body using an endoscope.
• JP 2016-062488 A also describes that treatment tools to be detected include hemostatic forceps, and that treatment tools may be detected using color information from the sheath.
• JP 2016-062488 A further describes that images from the period from when the treatment tool enters the frame to when it leaves the frame of the endoscopic video are recorded as video recording data.
• Republication No. 2019-181432 describes a surgery support system that uses a trained classifier to predict the occurrence of a dangerous situation. Republication No. 2019-181432 also describes an aspect in which, in generating teacher data used to train the classifier, it is detected that hemostasis is being performed by detecting the characteristics of a surgical instrument used for hemostasis from surgical images obtained from an endoscopic camera or the like.
• a first aspect of the present disclosure is a medical support device that includes a processor, which acquires a plurality of frames obtained in a time series by imaging the inside of a tubular organ into which a hemostat is inserted using a camera, the plurality of frames capturing stages of a hemostatic procedure using the hemostat, and outputs hemostatic period identification information that can identify that the period from when the stage captured in the plurality of acquired frames changes from the stage of preparing to stop hemostatic using the hemostat to the stage of starting to stop hemostatic using the hemostat until a certain amount of time has elapsed is the hemostatic period during which a hemostatic procedure is being performed.
• a second aspect of the present disclosure is a medical support device according to the first aspect, in which the processor detects a change from the hemostasis preparation stage to the hemostasis initiation stage by performing object recognition processing based on the multiple frames acquired, and the hemostasis period is a period from when the change is detected by the processor until a certain amount of time has elapsed.
• a third aspect of the present disclosure is a medical support device according to the second aspect, in which the object recognition process is realized by using a model obtained by performing machine learning on a neural network.
• a fourth aspect of the present disclosure is a medical support device according to any one of the first to third aspects, in which the hemostatic forceps has a sheath, the multiple frames include a first frame and a second frame, the first frame is a frame in which the sheath is visible, the second frame is a frame in which the sheath is not visible, the hemostasis preparation stage is a stage in which the first frame is obtained, and the hemostasis start stage is a stage in which the second frame is obtained following the first frame.
• a fifth aspect of the present disclosure is a medical support device according to any one of the first to third aspects, in which the hemostatic forceps has a sheath and a clip pin, the plurality of frames includes a first frame and a second frame, the first frame is a frame showing the sheath, the second frame is a frame showing the clip pin, the hemostasis preparation stage is a stage in which the first frame is obtained, and the hemostasis start stage is a stage in which the second frame is obtained following the first frame.
• a sixth aspect of the present disclosure is a medical support device according to any one of the first to third aspects, in which the hemostatic forceps has a sheath and a clip pin, the plurality of frames include a first frame and a second frame, the first frame is a frame in which the sheath is visible, the second frame is a frame in which the clip pin is visible but the sheath is not visible, the hemostasis preparation stage is a stage in which the first frame is obtained, and the hemostasis start stage is a stage in which the second frame is obtained following the first frame.
• a seventh aspect of the present disclosure is a medical support device according to any one of the fourth to sixth aspects, in which the multiple frames further include a third frame, the third frame being a frame showing a sheath, and the device is reset if the third frame is obtained before a certain period of time has elapsed since the transition from the hemostasis preparation stage to the hemostasis initiation stage.
• An eighth aspect of the present disclosure is a medical support device according to any one of the first to seventh aspects, in which hemostasis treatment-related information regarding hemostasis treatment during a hemostasis period is stored in a storage area.
• a ninth aspect of the present disclosure is a medical support device according to the eighth aspect, in which the hemostasis treatment-related information includes hemostasis period-specific information or a part of the hemostasis period-specific information.
• a tenth aspect of the present disclosure is a medical support device according to the ninth aspect, the device including, in part, start timing information capable of identifying the start timing of a hemostasis period and/or end timing information capable of identifying the end timing of a hemostasis period.
• An eleventh aspect of the present disclosure is a medical support device according to any one of the eighth to tenth aspects, in which the hemostasis procedure-related information includes hemostasis location-related information regarding the location where the hemostasis procedure was performed within the tubular organ.
• a twelfth aspect of the present disclosure is a medical support device according to the eleventh aspect, in which the hemostasis location-related information includes lesion-related information about a lesion that was present at the location where the hemostasis procedure was performed.
• a thirteenth aspect of the present disclosure is a medical support device according to the twelfth aspect, in which the lesion-related information includes a lesion image obtained by capturing an image of the lesion with a camera.
• a fourteenth aspect of the present disclosure is a medical support device according to any one of the eleventh to thirteenth aspects, in which the hemostasis location-related information includes a post-hemostasis image obtained by capturing an image of the location where the hemostasis procedure was performed by a camera.
• a fifteenth aspect of the present disclosure is a medical support device according to any one of the eleventh to fourteenth aspects, in which the hemostasis location-related information includes a pre-hemostasis image obtained by capturing an image of the location where the hemostasis procedure was performed before hemostasis using a camera.
• a sixteenth aspect of the present disclosure is a medical support device according to any one of the eighth to fifteenth aspects, in which the hemostatic procedure-related information includes frequency information that can identify the number of times that hemostatic procedures have been performed in a tubular organ.
• a seventeenth aspect of the present disclosure is a medical support device according to any one of the first to sixteenth aspects, in which the certain time is a statistically determined time required to perform one hemostatic procedure using hemostatic forceps.
• An 18th aspect of the present disclosure is an endoscope device including a medical support device according to any one of the first to seventeenth aspects and a camera.
• a nineteenth aspect of the present disclosure is a medical support method that includes acquiring a plurality of frames obtained in a time series by imaging the inside of a tubular organ into which a hemostatic forceps is inserted using a camera, the plurality of frames showing stages of a hemostatic procedure using a hemostatic forceps, and outputting hemostatic period identification information that can identify that the period from when the stage shown in the acquired plurality of frames changes from the stage of preparing to stop hemostatic using a hemostatic forceps to the stage of starting to stop hemostatic using a hemostatic forceps until a certain amount of time has elapsed is the hemostatic period during which a hemostatic procedure is being performed.
• a twentieth aspect of the present disclosure is a program for causing a computer to execute a process including acquiring a plurality of frames obtained in a time series by imaging the inside of a tubular organ into which a hemostatic forceps is inserted using a camera, the plurality of frames showing stages of a hemostatic procedure using a hemostatic forceps, and outputting hemostatic period identification information capable of identifying that the period from when the stage shown in the acquired plurality of frames changes from the stage of preparing to hemostatically using a hemostatic forceps to the stage of starting hemostatically using a hemostatic forceps until a certain amount of time has elapsed is the hemostatic period during which a hemostatic procedure is being performed.
• FIG. 1 is a conceptual diagram showing an example of an embodiment in which an endoscope device is used.
• 1 is a conceptual diagram showing an example of an overall configuration of an endoscope apparatus.
• 2 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope apparatus.
• FIG. 2 is a block diagram showing an example of main functions of a processor included in the medical support device according to an embodiment, and an example of information stored in a storage.
• FIG. FIG. 4 is a conceptual diagram showing an example of processing contents of a recognition unit and a control unit.
• FIG. 1 is a graph (i.e., a histogram) showing an example of the relationship between the number of frames required for one hemostatic procedure using a clip pin during endoscopic examination (i.e., the number of frames captured by the camera from the start to the end of one hemostatic procedure using a clip pin) and the number of hemostatic procedures using a hemostatic forceps.
• FIG. 13 is a conceptual diagram showing an example of processing contents in which a hemostasis treatment period is specified by a control unit; 13 is a conceptual diagram showing an example of processing performed by the control unit when a sheath appears in the frame before the hemostasis treatment time has elapsed after the transition from the hemostasis preparation stage to the hemostasis start stage.
• FIG. 13 is a flowchart showing an example of the flow of a medical support process.
• 13 is a conceptual diagram showing an example of how hemostatic treatment-related information is stored in a storage area.
• FIG. 13 is a conceptual diagram showing an example of the content of bleeding control location related information.
• 1 is a conceptual diagram showing an example of a series of processes in which a computer issues a processing execution request to an external device via a network, the external device executes processing in response to the processing execution request, and the computer receives a processing result from the external device.
• CPU is an abbreviation for "Central Processing Unit”.
• GPU is an abbreviation for "Graphics Processing Unit”.
• GPGPU is an abbreviation for "General-Purpose computing on Graphics Processing Units”.
• APU is an abbreviation for "Accelerated Processing Unit”.
• TPU is an abbreviation for "Tensor Processing Unit”.
• RAM is an abbreviation for "Random Access Memory”.
• NVM is an abbreviation for "Non-volatile memory”.
• EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory”.
• ASIC is an abbreviation for "Application Specific Integrated Circuit”.
• PLD is an abbreviation for "Programmable Logic Device”.
• FPGA is an abbreviation for "Field-Programmable Gate Array”.
• SoC is an abbreviation for "System-on-a-chip”.
• SSD is an abbreviation for "Solid State Drive”.
• USB is an abbreviation for "Universal Serial Bus”.
• HDD is an abbreviation for "Hard Disk Drive”.
• EL is an abbreviation for "Electro-Luminescence”.
• CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor”.
• CCD is an abbreviation for "Charge Coupled Device”.
• AI is an abbreviation for "Artificial Intelligence”.
• BLI is an abbreviation for "Blue Light Imaging”.
• LCI is an abbreviation for "Linked Color Imaging”.
• I/F is an abbreviation for "Interface”.
• SSL is an abbreviation for "Sessile Serrated Lesion.”
• LAN is an abbreviation for "Local Area Network.”
• WAN is an abbreviation for "Wide Area Network.”
• 5G is an abbreviation for "5th Generation Mobile Communication System.”
• a coded processor may be a single arithmetic device or a combination of multiple arithmetic devices.
• a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU, a GPU, a GPGPU, an APU, or a TPU.
• a signed memory is at least one memory such as a RAM (e.g., a volatile memory) in which information is temporarily stored and is used by the processor as a working memory.
• a RAM e.g., a volatile memory
• the symbolic storage refers to one or more non-volatile storage devices that store various programs, various parameters, etc.
• Examples of non-volatile storage devices include flash memory, magnetic disks, and magnetic tapes.
• Another example of storage is cloud storage.
• the external I/F with the symbol is responsible for the transmission and reception of various types of information between multiple devices connected to each other.
• An example of the external I/F is a USB interface.
• a communication I/F including a communication processor and an antenna may be applied to the external I/F.
• the communication I/F is responsible for communication between multiple computers.
• An example of a communication standard applied to the communication I/F is a wireless communication standard including 5G, Wi-Fi (registered trademark), or Bluetooth (registered trademark).
• a and/or B is synonymous with “at least one of A and B.”
• a and/or B means that it may be only A, only B, or a combination of A and B.
• the same concept as “A and/or B” is also applied when three or more things are expressed by connecting them with “and/or.”
• FIG. 1 is a conceptual diagram showing an example of how an endoscopic device 10 is used.
• the endoscopic device 10 is used by a doctor 12 in an endoscopic examination.
• the endoscopic examination is assisted by staff such as a nurse 14.
• the endoscopic device 10 is an example of an "endoscopic device" according to the present disclosure.
• the endoscope device 10 is communicatively connected to a communication device (not shown), and information obtained by the endoscope device 10 is transmitted to the communication device.
• a communication device is a server and/or a client terminal (e.g., a personal computer and/or a tablet terminal, etc.) that manages various information such as electronic medical records.
• the communication device receives the information transmitted from the endoscope device 10 and executes processing using the received information (e.g., processing to store in an electronic medical record, etc.).
• the endoscope device 10 includes an endoscope scope 16, a display device 18, a light source device 20, a control device 22, and a medical support device 24.
• the endoscope device 10 is a modality for performing medical treatment on the large intestine 28 contained within the body of a subject 26 (e.g., a patient) using an endoscope scope 16.
• the large intestine 28 is the object observed by the doctor 12 during an endoscopic examination.
• the endoscope 16 is used by the doctor 12 and inserted into a tubular organ of the subject 26.
• the endoscope 16 is inserted into the large intestine 28 of the subject 26.
• the endoscope device 10 causes the endoscope 16 inserted into the large intestine 28 of the subject 26 to capture images of the inside of the large intestine 28 of the subject 26, and performs various medical procedures on the large intestine 28 as necessary.
• the endoscope device 10 captures images of the inside of the large intestine 28 of the subject 26, and outputs images showing the state of the inside of the large intestine 28.
• the endoscope device 10 is an endoscope with an optical imaging function that captures images of reflected light obtained by irradiating light 30 inside the large intestine 28 and reflecting it off the intestinal wall 32 of the large intestine 28.
• an endoscopic examination of the large intestine 28 is illustrated here, this is merely one example, and the present disclosure also applies to endoscopic examination of hollow organs such as the esophagus, stomach, duodenum, or trachea.
• the light source device 20, the control device 22, and the medical support device 24 are installed on a wagon 34.
• the wagon 34 has multiple platforms arranged vertically, with the medical support device 24, the control device 22, and the light source device 20 installed from the lower platform to the upper platform.
• the display device 18 is installed on the top platform of the wagon 34.
• the control device 22 controls the entire endoscope device 10. Under the control of the control device 22, the medical support device 24 performs various image processing on the images obtained by capturing images of the intestinal wall 32 with the endoscope scope 16.
• the display device 18 displays various information including images. Examples of the display device 18 include a liquid crystal display and an EL display. Also, instead of the display device 18, or together with the display device 18, a tablet terminal with a display may be used.
• a screen 35 is displayed on the display device 18.
• the screen 35 includes a plurality of display areas.
• the plurality of display areas are arranged side by side within the screen 35.
• a first display area 36 and a second display area 38 are shown as examples of the plurality of display areas.
• the size of the first display area 36 is larger than the size of the second display area 38.
• the first display area 36 is used as the main display area, and the second display area 38 is used as the sub-display area. Note that the size relationship between the first display area 36 and the second display area 38 is not limited to this, and may be any size relationship that fits within the screen 35.
• the first display area 36 displays an endoscopic moving image 39.
• the endoscopic moving image 39 is a moving image acquired by imaging the intestinal wall 32 within the large intestine 28 of the subject 26 with the endoscope scope 16.
• a moving image showing the intestinal wall 32 is shown as an example of the endoscopic moving image 39.
• the intestinal wall 32 shown in the endoscopic video 39 includes at least one lesion 41 (e.g., one lesion 41 in the example shown in FIG. 1) as an area of interest (i.e., an area to be observed) that is gazed upon by the physician 12, and the physician 12 can visually recognize the appearance of the intestinal wall 32 including the lesion 41 through the endoscopic video 39.
• the lesion 41 is an example of a "lesion" according to the present disclosure.
• lesions 41 there are various types of lesions 41, and examples of the types of lesions 41 include neoplastic polyps and non-neoplastic polyps.
• examples of the types of neoplastic polyps include adenomatous polyps (e.g., SSL).
• examples of the types of non-neoplastic polyps include hamartomatous polyps, hyperplastic polyps, and inflammatory polyps.
• the types exemplified here are types that are anticipated in advance as types of lesions 41 when an endoscopic examination is performed on the large intestine 28, and the types of lesions will differ depending on the organ in which the endoscopic examination is performed.
• the image displayed in the first display area 36 is one frame 40 included in a moving image that is composed of multiple frames 40 in chronological order.
• the first display area 36 displays multiple frames 40 in chronological order at a default frame rate (e.g., several tens of frames per second).
• the frame 40 is an example of a "frame" according to the present disclosure.
• a moving image displayed in the first display area 36 is a moving image in a live view format.
• the live view format is merely one example, and the moving image may be temporarily stored in a memory or the like and then displayed, such as a moving image in a post-view format.
• each frame contained in a recording moving image stored in a memory or the like may be played back and displayed on the screen 35 (for example, the first display area 36) as an endoscopic moving image 39.
• the second display area 38 exists outside the first display area 36.
• the second display area 38 is adjacent to the first display area 36, and is displayed on the right side of the screen 35 when viewed from the front.
• the display position of the second display area 38 may be anywhere different from the first display area 36, but it is preferable that it is displayed in a position that can be compared with the endoscopic video image 39 displayed in the first display area 36.
• the second display area 38 displays medical information 44, which is information related to medical care.
• the medical information 44 include information that assists the doctor 12 in making medical decisions.
• a first example of information that assists the doctor 12 in making medical decisions includes various visible information about the subject 26 into which the endoscope 16 is inserted (e.g., name, sex, medications, medical history, blood pressure, and/or heart rate).
• a second example of information that assists the doctor 12 in making medical decisions includes visible information such as text and/or images (e.g., feature maps and/or information obtained by processing the feature maps) obtained by processing the endoscopic video 39 using AI.
• FIG. 2 is a conceptual diagram showing an example of the overall configuration of the endoscope device 10.
• the endoscope scope 16 is equipped with an operation section 46 and an insertion section 48.
• the insertion section 48 is partially curved by operating the operation section 46.
• the insertion section 48 is inserted into the large intestine 28 (see FIG. 1) while curving in accordance with the shape of the large intestine 28, in accordance with the operation of the operation section 46 by the doctor 12 (see FIG. 1).
• the tip 50 of the insertion section 48 is provided with a camera 52, a lighting device 54, and an opening 56 for a treatment tool.
• the camera 52 and lighting device 54 are provided on the tip surface 50A of the tip 50. Note that, although an example in which the camera 52 and lighting device 54 are provided on the tip surface 50A of the tip 50 is given here, this is merely one example, and the camera 52 and lighting device 54 may be provided on the side surface of the tip 50, so that the endoscope 16 is configured as a side-viewing mirror.
• Camera 52 is inserted into the body cavity of subject 26 to capture an image of the observation area.
• camera 52 captures an image of the inside of subject 26 (e.g., inside large intestine 28) to obtain endoscopic video image 39.
• One example of camera 52 is a CMOS camera. However, this is merely one example, and other types of cameras such as a CCD camera may also be used.
• camera 52 is an example of a "camera" according to the present disclosure.
• the illumination device 54 has illumination windows 54A and 54B.
• the illumination device 54 irradiates light 30 (see FIG. 1) through the illumination windows 54A and 54B.
• Examples of the type of light 30 irradiated from the illumination device 54 include visible light (e.g., white light) and non-visible light (e.g., near-infrared light).
• the illumination device 54 also irradiates special light through the illumination windows 54A and 54B. Examples of the special light include light for BLI and/or light for LCI.
• the camera 52 captures images of the inside of the large intestine 28 by optical techniques while the light 30 is irradiated inside the large intestine 28 by the illumination device 54.
• the treatment tool opening 56 is an opening for allowing the treatment tool 58 to protrude from the distal end 50.
• the treatment tool opening 56 is also used as a suction port for sucking blood and internal waste, and as a delivery port for delivering fluids.
• the operating section 46 is formed with a treatment tool insertion port 60, and the treatment tool 58 is inserted into the insertion section 48 from the treatment tool insertion port 60.
• the treatment tool 58 passes through the insertion section 48 and protrudes to the outside from the treatment tool opening 56.
• the treatment tool 58 include hemostatic forceps 58A, a puncture needle, a high-frequency knife, a snare, a catheter, a guidewire, and a cannula.
• the treatment tool 58 is shown as hemostatic forceps 58A protruding from the treatment tool opening 56.
• the hemostatic forceps 58A is an example of a "hemostatic forceps" according to the present disclosure.
• the hemostatic forceps 58A are used for hemostasis procedures.
• Hemostasis procedures are procedures to stop bleeding from a bleeding site (for example, a bleeding excision site by excising the lesion 41 with a snare or high-frequency knife, etc.).
• clip hemostasis is used for hemostasis procedures.
• Clip hemostasis is a method of stopping bleeding by pinching and applying pressure to the bleeding site (for example, a blood vessel or mucous membrane, etc.) without using high-frequency current, etc.
• ten or more hemostasis procedures may be performed on one lesion 41.
• the hemostatic forceps 58A has a sheath 58A1 and a clip pin 58A2.
• the clip pin 58A2 protrudes from the tip of the sheath 58A1 and clips the bleeding site, thereby stopping bleeding from the bleeding site.
• the clip pin 58A2 detaches from the sheath 58A1 and is left in the large intestine 28.
• the clip pin 58A2 clipped to the bleeding site is expelled from the large intestine 28 together with stool after the endoscopic examination.
• the endoscope scope 16 is connected to the light source device 20 and the control device 22 via a universal cord 62.
• the medical support device 24 and the reception device 64 are connected to the control device 22.
• the display device 18 is also connected to the medical support device 24.
• the control device 22 is connected to the display device 18 via the medical support device 24.
• the medical support device 24 is exemplified here as an external device for expanding the functions performed by the control device 22, an example is given in which the control device 22 and the display device 18 are indirectly connected via the medical support device 24, but this is merely one example.
• the display device 18 may be directly connected to the control device 22.
• the function of the medical support device 24 may be included in the control device 22, or the control device 22 may be equipped with a function for causing a server (not shown) to execute the same processing as that executed by the medical support device 24 (for example, the medical support processing described below) and for receiving and using the results of the processing by the server.
• the reception device 64 receives instructions from the doctor 12 and outputs the received instructions as an electrical signal to the control device 22.
• Examples of the reception device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and/or a remote control device.
• the control device 22 controls the light source device 20, exchanges various signals with the camera 52, and exchanges various signals with the medical support device 24.
• the light source device 20 emits light under the control of the control device 22 and supplies the light to the illumination device 54.
• the illumination device 54 has a built-in light guide, and the light supplied from the light source device 20 passes through the light guide and is irradiated from illumination windows 54A and 54B.
• the control device 22 causes the camera 52 to capture an image, acquires an endoscopic video image 39 (see FIG. 1) from the camera 52, and outputs it to a predetermined output destination (e.g., the medical support device 24).
• the medical support device 24 performs various types of image processing on the endoscopic video image 39 input from the control device 22 to provide medical support (here, endoscopic examination as an example).
• the medical support device 24 outputs the endoscopic video image 39 that has been subjected to various types of image processing to a predetermined output destination (e.g., the display device 18).
• the endoscopic video image 39 output from the control device 22 is output to the display device 18 via the medical support device 24, but this is merely one example.
• the control device 22 and the display device 18 may be connected, and the endoscopic video image 39 that has been subjected to image processing by the medical support device 24 may be displayed on the display device 18 via the control device 22.
• FIG. 3 is a block diagram showing an example of the hardware configuration of the electrical system of the endoscope device 10.
• the control device 22 includes a computer 66, a bus 68, and an external I/F 70.
• the computer 66 includes a processor 72, a memory 74, and a storage 76.
• the processor 72, the memory 74, the storage 76, and the external I/F 70 are connected to the bus 68.
• the processor 72 controls the entire control device 22.
• the memory 74 and the storage 76 are used by the processor 72.
• the external I/F 70 is responsible for the exchange of various information between the processor 72 and one or more devices (hereinafter also referred to as "first external devices") that exist outside the control device 22.
• the camera 52 is connected to the external I/F 70 as one of the first external devices, and the external I/F 70 is responsible for the exchange of various information between the camera 52 and the processor 72.
• the processor 72 controls the camera 52 via the external I/F 70.
• the processor 72 also acquires, via the external I/F 70, endoscopic video images 39 (see FIG. 1) obtained by the camera 52 capturing an image of the inside of the large intestine 28 (see FIG. 1).
• the light source device 20 is connected to the external I/F 70 as one of the first external devices, and the external I/F 70 is responsible for the exchange of various information between the light source device 20 and the processor 72.
• the light source device 20 supplies light to the lighting device 54 under the control of the processor 72.
• the lighting device 54 irradiates the light supplied from the light source device 20.
• the external I/F 70 is connected to the reception device 64 as one of the first external devices, and the processor 72 acquires instructions received by the reception device 64 via the external I/F 70 and executes processing according to the acquired instructions.
• the medical support device 24 includes a computer 78 and an external I/F 80.
• the computer 78 includes a processor 82, a memory 84, and a storage 86.
• the processor 82, the memory 84, the storage 86, and the external I/F 80 are connected to a bus 88.
• the medical support device 24 is an example of a "medical support device” according to the present disclosure
• the computer 78 is an example of a "computer” according to the present disclosure
• the processor 82 is an example of a "processor" according to the present disclosure.
• computer 78 i.e., processor 82, memory 84, and storage 86
• processor 82 the hardware configuration of computer 78
• memory 84 the hardware configuration of computer 66
• storage 86 the hardware configuration of computer 78
• the external I/F 80 is responsible for the exchange of various information between the processor 82 and one or more devices (hereinafter also referred to as "second external devices") that exist outside the medical support device 24.
• the control device 22 is connected to the external I/F 80 as one of the second external devices.
• the external I/F 70 of the control device 22 is connected to the external I/F 80.
• the external I/F 80 is responsible for the exchange of various information between the processor 82 of the medical support device 24 and the processor 72 of the control device 22.
• the processor 82 acquires endoscopic video images 39 (see FIG. 1) from the processor 72 of the control device 22 via the external I/Fs 70 and 80, and performs various image processing on the acquired endoscopic video images 39.
• the display device 18 is connected to the external I/F 80 as one of the second external devices.
• the processor 82 controls the display device 18 via the external I/F 80 to cause the display device 18 to display various information (e.g., endoscopic moving image 39 that has been subjected to various image processing).
• a hemostatic procedure is performed to stop bleeding from the bleeding site (i.e., the site where the lesion 41 was excised) using hemostatic forceps 58A.
• the doctor 12 determines whether or not the next medical procedure (e.g., further hemostatic procedure) is necessary and determines the timing of performing the next medical procedure. Therefore, in an endoscopic examination, it is important for the doctor 12 to accurately determine whether or not hemostatic procedure is currently being performed (in particular, whether or not hemostatic procedure has been completed) in order to achieve appropriate medical procedure.
• FIG. 4 is a block diagram showing an example of the main functions of the processor 82 included in the medical support device 24, and an example of information stored in the storage 86.
• Storage 86 stores a medical support program 90.
• the medical support program 90 is an example of a "program" according to the present disclosure.
• the processor 82 reads out the medical support program 90 from storage 86 and executes the read out medical support program 90 on memory 84 to perform medical support processing.
• the medical support processing is realized by the processor 82 operating as a recognition unit 82A and a control unit 82B in accordance with the medical support program 90 executed on memory 84.
• Storage 86 stores a recognition model 92. As will be described in more detail below, recognition model 92 is used by recognition unit 82A.
• FIG. 5 is a conceptual diagram showing an example of the processing contents of the recognition unit 82A and the control unit 82B.
• the recognition unit 82A and the control unit 82B acquire, from the camera 52, each of a plurality of frames 40 in a time series included in the endoscopic moving image 39 generated by the camera 52 capturing images of the inside of the large intestine 28 at an imaging frame rate (e.g., several tens of frames/second), one frame at a time series.
• an imaging frame rate e.g., several tens of frames/second
• a mode is shown in which the recognition unit 82A acquires a plurality of frames 40 that show a bleeding site 42 (e.g., a site where bleeding occurs due to the resection of a lesion 41) and a stage of hemostasis treatment using a hemostatic forceps 58A.
• the frames 40 acquired from the camera 52 by the recognition unit 82A are acquired by the control unit 82B in synchronization with the recognition unit 82A.
• the stage of hemostasis treatment using the hemostatic forceps 58A refers to, for example, a stage of hemostasis preparation using the hemostatic forceps 58A and a stage of hemostasis start using the hemostatic forceps 58A.
• the hemostasis preparation stage is a stage in which the bleeding site 42 is targeted (i.e., a stage in which the hemostasis procedure is being prepared).
• the sheath 58A1 e.g., the sheath 58A1 in a state in which the clip pin 58A2 does not protrude from the tip
• the hemostasis preparation stage can also be said to be a stage in which the sheath 58A1 (e.g., the sheath 58A1 in a state in which the clip pin 58A2 does not protrude from the tip) is reflected in the frame 40.
• the hemostasis start stage is a stage at which hemostasis treatment is started for the bleeding site 42.
• the frame 40 shows the clip pin 58A2 to which the bleeding site 42 is clipped, and the sheath 58A1 is not shown.
• the hemostasis start stage can also be said to be a stage at which the frame 40 shows the clip pin 58A2 to which the bleeding site 42 is clipped, and the sheath 58A1 is not shown.
• a stage at which the state changes from the first state to the second state is used as an example of the hemostasis start stage.
• the first state refers to a state in which the sheath 58A1 is shown in the frame 40 obtained one frame before the latest frame 40.
• the second state refers to a state in which the latest frame 40 shows the clip pin 58A2 to which the bleeding site 42 is clipped, and the sheath 58A1 is not shown.
• the frame 40 showing the sheath 58A1 is an example of a "first frame” according to the present disclosure
• the frame 40 showing the clip pin 58A2 detached from the sheath 58A1 is an example of a "second frame” according to the present disclosure.
• the control unit 82B outputs the endoscopic moving image 39 to the display device 18.
• the control unit 82B displays the endoscopic moving image 39 as a live view image in the first display area 36. That is, each time the control unit 82B acquires a frame 40 from the camera 52, the control unit 82B displays the acquired frame 40 in sequence in the first display area 36 according to the display frame rate (e.g., several tens of frames per second).
• the control unit 82B also displays medical information 44 in the second display area 38.
• the control unit 82B also updates the display content of the second display area 38 (e.g., medical information 44) in accordance with the display content of the first display area 36.
• the recognition unit 82A uses the endoscopic video 39 acquired from the camera 52 to recognize the hemostat 58A that appears in the endoscopic video 39. That is, the recognition unit 82A recognizes the hemostat 58A that appears in the frame 40 by sequentially performing a recognition process 96 on each of a plurality of frames 40 in chronological order contained in the endoscopic video 39 acquired from the camera 52. For example, the recognition unit 82A recognizes the sheath 58A1 and the clip pin 58A2 individually. For example, the recognition of the clip pin 58A2 is the recognition of the clip pin 58A2 in a state where it has been detached from the sheath 58A1.
• the recognition process 96 is performed by the recognition unit 82A on the acquired frame 40 each time the frame 40 is acquired.
• the recognition process 96 is a process that recognizes each of the sheath 58A1 and the clip pin 58A2 using an AI-based method (i.e., object recognition process using machine learning).
• the recognition process 96 is an object recognition process using AI in a segmentation method (e.g., semantic segmentation, instance segmentation, and/or panoptic segmentation).
• the recognition model 92 is a trained model for object recognition using an AI segmentation method.
• An example of a trained model for object recognition using an AI segmentation method is a model for semantic segmentation.
• An example of a model for semantic segmentation is a model with an encoder-decoder structure.
• An example of a model with an encoder-decoder structure is U-Net or HRNet.
• the recognition process 96 is an example of the "object recognition process" according to the present disclosure.
• the recognition model 92 is optimized by performing machine learning on the neural network using training data.
• the training data is a data set that includes multiple data (i.e., multiple frames of data) in which example data and correct answer data are associated with each other.
• the example data are various images assuming frame 40.
• the correct answer data are correct answer data (i.e., annotations) for the example data.
• annotations that identify the sheaths of various hemostats that appear in the images used as the example data, and annotations that identify the clip pins of various hemostats that appear in the images used as the example data are used as examples of correct answer data.
• the recognition unit 82A acquires a frame 40 from the camera 52 and inputs the acquired frame 40 to the recognition model 92. As a result, each time a frame 40 is input, the recognition model 92 recognizes whether the sheath 58A1 is captured in the input frame 40, and recognizes whether the clip pin 58A2 is captured in the input frame 40. The recognition model 92 then outputs a recognition result 98.
• the recognition result 98 includes sheath presence/absence information 98A and clip pin presence/absence information 98B.
• the sheath presence/absence information 98A is information indicating whether or not a sheath 58A1 is present in the corresponding frame 40 (i.e., the frame 40 input to the recognition model 92).
• the clip pin presence/absence information 98B is information indicating whether or not a clip pin 58A2 is present in the corresponding frame 40 (i.e., the frame 40 input to the recognition model 92).
• the recognition unit 82A acquires the recognition result 98 output from the recognition model 92, and outputs the acquired recognition result 98 to the control unit 82B.
• the control unit 82B executes processing (e.g., see Figures 7 and 8) using the recognition result 98 input from the recognition unit 82A.
• FIG. 6 is a graph (i.e., a histogram) showing an example of the relationship between the number of frames required for one hemostasis procedure using clip pin 58A2 during endoscopic examination (i.e., the number of frames captured by camera 52 from the start to the end of one hemostasis procedure using clip pin 58A2) and the number of hemostasis procedures using hemostatic forceps 58A.
• the vertical axis shown in FIG. 6 is the number of hemostasis procedures using hemostatic forceps 58A (i.e., the number of procedures using clip hemostasis method using clip pin 58A2), and the horizontal axis shown in FIG. 6 is the number of frames. Note that the graph shown in FIG. 6 was created from data obtained by performing multiple endoscopic examinations on the large intestines 28 of multiple subjects 26.
• a hemostasis treatment time 100 is used.
• the hemostasis treatment time 100 is an example of a "fixed time" according to the present disclosure.
• the hemostasis treatment time 100 is a fixed time that is determined in advance as the time required for one hemostasis treatment, and is statistically determined as the time required to perform one hemostasis treatment.
• one hemostasis procedure is completed between the start of the procedure using the clip pin 58A2 and the time when approximately 1,800 frames of images are captured by the camera 52.
• the timing at which one hemostasis procedure using the clip pin 58A2 begins is the timing at which the bleeding site 42 is clipped by the clip pin 58A2 and the sheath 58A1 is framed out of the frame 40. Also, in the example shown in FIG. 6, the timing at which one hemostasis procedure using the clip pin 58A2 ends is the timing at which approximately 1800 frames of images are captured by the camera 52 after one hemostasis procedure using the clip pin 58A2 begins.
• the timing at which approximately 1800 frames are captured is the timing at which sheath 58A1 re-enters the frame (the timing at which sheath 58A1 is captured in frame 40 again) in most endoscopic examinations.
• the re-entering of the sheath 58A1 means that the Nth hemostasis procedure using clip pin 58A2 has been completed and preparation for the N+1th hemostasis procedure has begun (for example, the stage of searching for a location to clip with clip pin 58A2 in the N+1th endoscopic examination (i.e., the bleeding site 42 where the previous hemostasis procedure was performed, or a bleeding site 42 different from the bleeding site 42 where the previous hemostasis procedure was performed)).
• the time required for the camera 52 to capture 1800 frames is used as the hemostasis treatment time 100.
• the hemostasis treatment time 100 may be the time required for the camera 52 to capture the number of frames corresponding to the statistical value of the number of hemostasis treatments in the graph shown in FIG. 6, such as the time required for the camera 52 to capture the number of frames corresponding to the maximum number of hemostasis treatments in the graph shown in FIG. 6, the time required for the camera 52 to capture the number of frames corresponding to the average number of hemostasis treatments in the graph shown in FIG. 6, or the time required for the camera 52 to capture the number of frames corresponding to the median number of hemostasis treatments in the graph shown in FIG. 6.
• the time required to capture the number of frames corresponding to the statistical value is illustrated, but this is merely an example, and the hemostasis treatment time 100 may be a time that is determined in advance for the statistical value.
• the time required for the camera 52 to capture less than 1800 frames, or the time required for the camera 52 to capture more than 1800 frames may be used as the hemostasis treatment time 100.
• the hemostasis treatment time 100 set by default e.g., the hemostasis treatment time 100 set once in the manner described above
• the timing at which the bleeding site 42 is clipped by the clip pin 58A2 and the sheath 58A1 is framed out of the frame 40 is set as the start timing of the hemostasis treatment, and the timing designated as the completion of the hemostasis treatment is set as the end timing of the hemostasis treatment, and the time specified as the end timing of the hemostasis treatment is used as the hemostasis treatment time 100.
• the hemostatic treatment time 100 is a time statistically derived in advance from the history of multiple hemostatic treatments that have actually been performed (e.g., data collected on the time required for each of multiple hemostatic treatments), or a time that has been adjusted from the time derived in this manner.
• the timing for starting the hemostasis procedure does not have to be the timing when the bleeding site 42 is clipped by the clip pin 58A2 and the sheath 58A1 is framed out of the frame 40, but may be the timing when the bleeding site 42 changes from a state in which it is not clipped by the clip pin 58A2 to a state in which it is clipped by the clip pin 58A2.
• the state in which the bleeding site 42 is not clipped by the clip pin 58A2 and the state in which the bleeding site 42 is clipped by the clip pin 58A2 may be recognized by an AI-based object recognition process (e.g., an object recognition process using the recognition model 92) and/or a non-AI-based object recognition process (e.g., an object recognition process using template matching).
• an AI-based object recognition process e.g., an object recognition process using the recognition model 92
• a non-AI-based object recognition process e.g., an object recognition process using template matching
• Hemostasis treatment is a medical treatment performed during the period from when the hemostasis preparation stage changes to when the hemostasis start stage has been reached until 100 hours of hemostasis treatment time has elapsed.
• the start timing of the hemostasis treatment is the timing when the hemostasis preparation stage changes to when the hemostasis start stage has been reached
• the end timing of the hemostasis treatment is the timing at which 100 hours of hemostasis treatment time has elapsed since the hemostasis preparation stage changes to when the hemostasis start stage has been reached.
• the period from when the hemostasis treatment starts to when the hemostasis treatment ends is used as the hemostasis period during which the hemostasis treatment is being performed.
• the control unit 82B refers to the sheath presence/absence information 98A and the clip pin presence/absence information 98B included in the recognition result 98 to determine whether the sheath 58A1 is not shown in the frame 40 and the clip pin 58A2 is shown. Then, when a frame 40 showing the sheath 58A1 is followed by a frame 40 showing the clip pin 58A2 but not the sheath 58A1, the control unit 82B determines that the stage at which a frame 40 showing the sheath 58A1 is followed by a frame 40 showing the sheath 58A1 but not the sheath 58A1 and showing the clip pin 58A2 is the hemostasis start stage.
• An example of the hemostasis period identification information 102 is text 44A that can identify that the hemostasis period 101 is in progress.
• An example of the text 44A is the text "Bleeding is being stopped.”
• the content of the text 44A may be any content that can identify that the current time is the hemostasis period 101.
• text 44A is shown as an example here, it may also be an image that can identify that the current time is the hemostasis period 101.
• the text 44A is displayed in the second display area 38 as part of the medical information 44 during the hemostasis period 101 (i.e., until the hemostasis treatment time 100 has elapsed).
• a sound that can identify the hemostasis period 101 may be output from a speaker (not shown), or information that can identify the hemostasis period 101 may be printed on a medium (e.g., paper) by a printer.
• FIG. 8 is a conceptual diagram showing an example of the processing contents by the control unit 82B when the sheath 58A1 appears in the frame 40 before the hemostasis treatment time 100 has elapsed since the hemostasis preparation stage changed to the hemostasis start stage.
• the sheath 58A1 appears in the frame 40 before the hemostasis treatment time 100 has elapsed since the hemostasis preparation stage changed to the hemostasis start stage.
• the sheath 58A1 may appear in the frame 40 at a timing not intended by the doctor 12 (i.e., a timing when the hemostasis treatment has not been completed).
• the sheath 58A1 may appear in the frame 40 when the doctor 12 attempts to perform the next hemostasis treatment, even though the hemostasis treatment has not been completed.
• the doctor 12 understands that the current time is the hemostasis period 101.
• the control unit 82B therefore refers to the recognition result 98 to determine whether or not a frame 40 in which the sheath 58A1 appears has been obtained by the recognition unit 82A before the hemostasis treatment time 100 has elapsed since the transition from the hemostasis preparation stage to the hemostasis start stage. If a frame 40 in which the sheath 58A1 appears has been obtained by the recognition unit 82A (i.e., the recognition unit 82A recognizes the frame-in of the sheath 58A1), the control unit 82B resets the hemostasis treatment time 100.
• the control unit 82B restarts timing of the hemostasis treatment time 100, thereby starting the hemostasis period 101 again.
• frame 40 in which sheath 58A1 is captured before hemostasis treatment time 100 has elapsed since the transition from hemostasis preparation stage to hemostasis start stage is an example of the "third frame" according to the present disclosure.
• the flow of the medical support process shown in FIG. 9 is an example of a "medical support method" related to the present disclosure.
• the medical support process is performed by the processor 82 when a bleeding site 42 occurs due to the resection of a lesion 41 from the intestinal wall 32 during an endoscopic examination.
• step ST10 the recognition unit 82A determines whether or not one frame of image data has been captured by the camera 52 inside the large intestine 28. If, in step ST10, one frame of image data has not been captured by the camera 52 inside the large intestine 28, the determination is negative and the medical support process proceeds to step ST32. If, in step ST10, one frame of image data has been captured by the camera 52 inside the large intestine 28, the determination is positive and the medical support process proceeds to step ST12.
• step ST12 the recognition unit 82A and the control unit 82B acquire a frame 40 obtained by imaging the large intestine 28 with the camera 52.
• the control unit 82B then displays the frame 40 in the first display area 36 (see Figures 5, 7, and 8).
• step ST14 the medical support processing proceeds to step ST14.
• step ST14 the recognition unit 82A performs recognition processing 96 on the frame 40 acquired in step ST12.
• the control unit 82B acquires the recognition result 98 obtained by the recognition unit 82A performing the recognition processing 96.
• step ST16 the medical support processing proceeds to step ST16.
• step ST16 the control unit 82B refers to the recognition result 98 acquired in step ST14 and determines whether or not the sheath 58A1 is captured in the frame 40 acquired in step ST12. In step ST16, if the sheath 58A1 is not captured in the frame 40 acquired in step ST12, the determination is negative, and the medical support process proceeds to step ST22. In step ST16, if the sheath 58A1 is captured in the frame 40 acquired in step ST12, the determination is positive, and the medical support process proceeds to step ST18.
• step ST18 the control unit 82B determines whether the endoscopic examination is in the hemostasis period 101. In step ST18, if the endoscopic examination is not in the hemostasis period 101 (i.e., if the current time is within the hemostasis period 101), the determination is negative and the medical support process proceeds to step ST32. In step ST18, if the endoscopic examination is in the hemostasis period 101, the determination is positive and the medical support process proceeds to step ST20.
• step ST20 the control unit 82B resets the hemostasis treatment time 100. After the processing of step ST20 is executed, the medical support processing proceeds to step ST32.
• step ST22 the control unit 82B determines whether or not the clip pin 58A2 is shown in the frame 40 acquired in step ST12 by referring to the recognition result 98 acquired in step ST14. If the clip pin 58A2 is not shown in the frame 40 acquired in step ST12 in step ST22, the determination is negative, and the medical support process proceeds to step ST32. If the clip pin 58A2 is shown in the frame 40 acquired in step ST12 in step ST22, the determination is positive, and the medical support process proceeds to step ST24.
• step ST24 the control unit 82B determines whether the sheath 58A1 was captured in the frame 40 acquired in step ST12 one frame before the frame 40 used in the determination in step ST22. In step ST24, if the sheath 58A1 was not captured in the frame 40 acquired in step ST12 one frame before the frame 40 used in the determination in step ST22, the determination is negative and the medical support process proceeds to step ST32. In step ST24, if the sheath 58A1 was captured in the frame 40 acquired in step ST12 one frame before the frame 40 used in the determination in step ST22, the determination is positive and the medical support process proceeds to step ST26.
• step ST26 the control unit 82B starts the hemostasis period 101.
• the start of the hemostasis period 101 is realized by the control unit 82B starting to time the hemostasis treatment time 100.
• the control unit 82B then outputs hemostasis period specific information 102 to the display device 18.
• text 44A is displayed in the second display area 38 (see FIG. 7).
• step ST28 the control unit 82B determines whether or not the hemostasis treatment time 100 has elapsed since the start of timing the hemostasis treatment time 100. If the hemostasis treatment time 100 has not elapsed since the start of timing the hemostasis treatment time 100 in step ST28, the determination is negative and the medical support process proceeds to step ST32. If the hemostasis treatment time 100 has elapsed since the start of timing the hemostasis treatment time 100 in step ST28, the determination is positive and the medical support process proceeds to step ST30.
• step ST30 the control unit 82B ends the hemostasis period 101.
• the end of the hemostasis period 101 is achieved by ending the timing of the hemostasis treatment time 100 by the control unit 82B.
• the control unit 82B then hides the text 44A in the second display area 38. In other words, the text 44A is erased from the second display area 38.
• step ST32 the control unit 82B determines whether or not a condition for terminating the medical support process has been satisfied.
• a condition for terminating the medical support process is a condition in which an instruction to terminate the medical support process has been given to the endoscope device 10 (for example, a condition in which an instruction to terminate the medical support process has been accepted by the acceptance device 64).
• step ST32 If the conditions for terminating the medical support process are not met in step ST32, the determination is negative and the medical support process proceeds to step ST10. If the conditions for terminating the medical support process are met in step ST32, the determination is positive and the medical support process ends.
• the recognition unit 82A acquires a plurality of frames 40 that are obtained in a time series by capturing images of the inside of the large intestine 28 using the camera 52, and the plurality of frames 40 that show the stages of the hemostasis treatment are acquired. Then, the control unit 82B outputs to the display device 18 hemostasis period identification information 102 that can identify the hemostasis period 101 from when the stage of the hemostasis treatment captured in the plurality of frames 40 acquired by the recognition unit 82A changes from the hemostasis preparation stage to the hemostasis start stage until the hemostasis treatment time 100 has elapsed.
• the screen 35 here, as an example, the second display area 38 of the display device 18.
• text 44A that can identify that it is the hemostasis period 101 is displayed. This allows the doctor 12 to understand that the current time is the hemostasis period 101 (i.e., the period during which the hemostasis treatment is being performed using the hemostatic forceps 58).
• Hemostatic procedure time 100 is a statistically determined time required for one hemostatic procedure using hemostatic forceps 58 (here, as an example, a time determined based on the graph shown in FIG. 6). This makes it difficult for the (N+1)th hemostatic procedure to be performed before the Nth hemostatic procedure has been completed, for example. In addition, because doctor 12 can accurately determine the timing at which one hemostatic procedure has ended, the number of times hemostatic procedures have been performed can also be counted with high accuracy.
• control unit 82B detects a change from the hemostasis preparation stage to the hemostasis start stage based on the recognition result 98 obtained by the recognition unit 82A performing recognition processing 96 using a recognition model 92 on multiple frames 40. This makes it possible to detect the change from the hemostasis preparation stage to the hemostasis start stage without placing a burden on the doctor 12, compared to when the doctor 12 determines the change from the hemostasis preparation stage to the hemostasis start stage by observing multiple frames 40.
• the period from when the change from the hemostasis preparation stage to the hemostasis start stage is detected until the hemostasis treatment time 100 has elapsed is set as hemostasis period 101.
• the doctor 12 is notified that hemostasis period 101 exists by displaying text 44A in the second display area 38. In this way, the period from when the change from the hemostasis preparation stage to the hemostasis start stage is detected until the hemostasis treatment time 100 has elapsed is set as hemostasis period 101 and notified to the doctor 12, so that the doctor 12 can be made aware of the hemostasis period 101 without placing a burden on him.
• the stage at which a frame 40 showing the sheath 58A1 is obtained is regarded as the hemostasis preparation stage, and the stage at which a frame 40 showing the sheath 58A1 is obtained followed by a frame 40 showing the sheath 58A1 is regarded as the hemostasis start stage. Therefore, the timing at which the hemostasis procedure is started, i.e., the start timing of the hemostasis period 101, can be determined simply. As a result, the hemostasis period 101 can also be determined simply.
• the stage at which a frame 40 showing the sheath 58A1 is obtained is regarded as the hemostasis preparation stage, and the stage at which a frame 40 showing the clip pin 58A2 detached from the sheath 58A1 is obtained following the frame 40 showing the sheath 58A1 is regarded as the hemostasis start stage. Therefore, the timing at which the hemostasis procedure is started, i.e., the start timing of the hemostasis period 101, can be determined simply. As a result, the hemostasis period 101 can also be determined simply.
• the stage at which a frame 40 showing the sheath 58A1 is obtained is regarded as the hemostasis preparation stage, and the stage at which a frame 40 showing the sheath 58A1 is followed by a frame 40 showing the clip pin 58A2 detached from the sheath 58A1 and a frame 40 showing no sheath 58A1 is regarded as the hemostasis start stage. Therefore, the timing at which the hemostasis procedure is started, i.e., the start timing of the hemostasis period 101, can be determined simply. As a result, the hemostasis period 101 can also be determined simply.
• the hemostasis treatment time 100 is reset. That is, under the condition that the hemostasis treatment time 100 is predetermined as the time required for one hemostasis treatment, the hemostasis treatment time 100 is reset if a frame 40 showing the sheath 58A1 is obtained before the hemostasis treatment time 100 has elapsed since the transition from the hemostasis preparation stage to the hemostasis start stage.
• control unit 82B outputs the hemostasis period specific information 102 to the display device 18 during the hemostasis period 101, but the present disclosure is not limited to this.
• the control unit 82B may store hemostasis treatment-related information 104 relating to the hemostasis treatment during the hemostasis period 101 in the storage area 84A of the memory 84 on the condition that the hemostasis period 101 has started. In this way, processing using the hemostasis treatment-related information 104 can be easily performed.
• the hemostasis treatment related information 104 may include the hemostasis period specific information 102, or may include a portion of the hemostasis period specific information 102. This makes it easier to perform processing using the hemostasis period specific information 102 or a portion of the hemostasis period specific information 102.
• hemostasis start timing information 104A capable of specifying the start timing of the hemostasis period 101 (e.g., information indicating the start time of the hemostasis period 101), and/or hemostasis end timing information 104B capable of specifying the end timing of the hemostasis period 101 (e.g., information indicating the end time of the hemostasis period 101).
• Hemostasis start timing information 104A is an example of "start timing information" according to the present disclosure
• hemostasis end timing information 104B is an example of "end timing information” according to the present disclosure.
• hemostasis start timing information 104A is stored in storage area 84A
• doctor 12, etc. can ascertain the start timing of hemostasis period 101 from hemostasis start timing information 104A stored in storage area 84A.
• hemostasis end timing information 104B is stored in storage area 84A, doctor 12, etc. can ascertain the end timing of hemostasis period 101 from hemostasis end timing information 104B stored in storage area 84A.
• hemostasis procedure-related information 104 may include hemostasis location-related information 104C.
• Hemostasis location-related information 104C is an example of "hemostasis location-related information" according to the present disclosure. Hemostasis location-related information 104C refers to information relating to the location in the large intestine 28 where hemostasis procedure was performed. If hemostasis location-related information 104C is stored in storage area 84A, the doctor 12, etc. can refer to the hemostasis location-related information 104C stored in storage area 84A to determine what medical procedure is required for the location in the large intestine 28 where hemostasis procedure was performed.
• Lesion related information 104C1 is an example of "lesion related information" according to the present disclosure. Lesion related information 104C1 refers to information related to the lesion 41 that existed at the location where the hemostasis procedure was performed. In this way, if the hemostasis location related information 104C includes the lesion related information 104C1, the doctor 12, etc.
• the lesion related information 104C1 stored in the storage area 84A can refer to the lesion related information 104C1 stored in the storage area 84A to determine what medical treatment is required for the lesion 41 that existed at the location where the hemostasis was performed, or what medical treatment is required for the location where the lesion 41 was present (e.g., the bleeding location 42), or what medical treatment is required for the location where the hemostasis was completed (e.g., the location clipped by the clip pin 58A2).
• the hemostasis treatment-related information 104 may include number of times information 104D.
• the number of times information 104D is an example of "number of times information" according to the present disclosure.
• the number of times information 104D is information that can identify the number of times hemostasis treatment has been performed in the large intestine 28 (for example, hemostasis treatment for the bleeding site 42 for one lesion 41, or the number of clip pins 58A2 placed in the large intestine 28). In this way, if the hemostasis treatment-related information 104 includes the number of times information 104D, the doctor 12, etc.
• the necessary medical treatment can be determined according to the number of times hemostasis treatment has been performed in the large intestine 28.
• the hemostasis location related information 104C may include a pre-hemostasis image 40A.
• the pre-hemostasis image 40A is an example of a "pre-hemostasis image" according to the present disclosure.
• the pre-hemostasis image 40A is a frame 40 obtained by capturing an image of a bleeding location 42 before hemostasis, where a hemostasis procedure has been performed, by a camera 52. If the pre-hemostasis image 40A is included in the hemostasis location related information 104C, the doctor 12 or the like can understand the state of the bleeding location 42 before hemostasis by referring to the pre-hemostasis image 40A stored in the storage area 84A.
• the hemostasis location related information 104C may include a post-hemostasis image 40B.
• the post-hemostasis image 40B is an example of a "post-hemostasis image" according to the present disclosure.
• the post-hemostasis image 40B is a frame 40 obtained by capturing an image of the location where the hemostasis procedure has been performed by the camera 52. If the post-hemostasis image 40B is included in the hemostasis location related information 104C, the doctor 12, etc. can understand the state of the location where the hemostasis procedure has been performed by referring to the post-hemostasis image 40B stored in the storage area 84A.
• Lesion-related information 104C1 may also include lesion image 40C.
• Lesion image 40C is an example of a "lesion image” according to the present disclosure.
• Lesion image 40C is a frame 40 obtained by imaging lesion 41 with camera 52. If lesion image 40C is included in lesion-related information 104C1, doctor 12 or the like can refer to frame 40 included in lesion image 40C stored in storage area 84A to determine what medical treatment is required for lesion 41 that existed at the location where hemostasis was performed, what medical treatment is required for the location where lesion 41 existed (e.g., bleeding location 42), or what medical treatment is required for the location where hemostasis was completed (e.g., the location clipped with clip pin 58A2).
• Lesion-related information 104C1 may include lesion type information 104C1a and/or lesion type information 104C1b.
• Lesion type information 104C1a is information indicating the type of lesion 41.
• types of lesion 41 include neoplastic polyps and non-neoplastic polyps.
• types of neoplastic polyps include adenomatous polyps (e.g., SSL).
• types of non-neoplastic polyps include hamartomatous polyps, hyperplastic polyps, and inflammatory polyps. Note that the types exemplified here are types that are assumed in advance as types of lesion 41 when an endoscopic examination is performed on the large intestine 28, and the types of lesion 41 will differ if the organ in which the endoscopic examination is performed is different.
• Lesion type information 104C1b is information indicating the type of lesion 41. Examples of types of lesion 41 include pedunculated, subpedunculated, sessile, surface elevated, surface flat, and surface depressed.
• Lesion-related information 104C1 may include geometric property information 104C1c.
• Geometric property information 104C1c is information indicating the geometric property of lesion 41.
• One example of the geometric property of lesion 41 is information (e.g., coordinates) that can identify the position and/or shape of an image area indicating lesion 41 within lesion image 40C.
• the lesion type information 104C1a, the lesion type information 104C1b, and/or the geometric characteristic information 104C1c are information obtained by performing object recognition processing (e.g., object recognition processing using AI and/or object recognition processing using template matching) on the frame 40.
• the lesion type information 104C1a, the lesion type information 104C1b, and/or the geometric characteristic information 104C1c may be accepted by the reception device 64.
• the control unit 82B causes the lesion type information 104C1a, the lesion type information 104C1b, and/or the geometric characteristic information 104C1c accepted by the reception device 64 to be included in the lesion-related information 104C1.
• Lesion type information 104C1a, lesion type information 104C1b, and/or geometric property information 104C1c may be associated with lesion image 40C.
• control unit 82B displays lesion image 40C on screen 35, and also displays lesion type information 104C1a, lesion type information 104C1b, and/or geometric property information 104C1c associated with lesion image 40C.
• doctor 12 can observe lesion image 40C through screen 35, and can grasp the type of lesion 41, the type of lesion 41, and/or the geometric property of lesion 41 shown in lesion image 40C.
• the hemostasis start timing information 104A and/or hemostasis end timing information 104B may be associated with hemostasis location related information 104C (e.g., hemostasis location related information 104C obtained during the hemostasis period 101 specified by the hemostasis start timing information 104A and/or hemostasis end timing information 104B) related to the hemostasis start timing information 104A and/or hemostasis end timing information 104B.
• hemostasis location related information 104C e.g., hemostasis location related information 104C obtained during the hemostasis period 101 specified by the hemostasis start timing information 104A and/or hemostasis end timing information 104B
• the doctor 12 can grasp the hemostasis location related information 104C through the screen 35, etc., and can also grasp the hemostasis start timing information 104A and/or hemostasis end timing information 104B associated with the hemostasis location related information 104C through the screen 35, etc.
• the hemostasis treatment-related information 104 is stored in storage area 84A, but this is merely one example. At least a portion of the information contained in the hemostasis treatment-related information 104 may be stored in storage 76 and/or 86, etc. At least a portion of the information contained in the hemostasis treatment-related information 104 may be stored in an electronic medical record. At least a portion of the information contained in the hemostasis treatment-related information 104 may be stored in a server, a personal computer, and/or a tablet terminal. At least a portion of the information contained in the hemostasis treatment-related information 104 may be printed on a medium by a printer. At least a portion of the information contained in the hemostasis treatment-related information 104 may be output as audio from a speaker.
• the information included in the hemostasis treatment-related information 104 may be used as input information for the so-called generation AI.
• the information output from the generation AI may be stored in the various storage areas (e.g., storage 76 and/or 86) described above, displayed as medical information 44 on the display device 18, printed on a medium by a printer, or output as audio from a speaker.
• An example of the generation AI is ChatGPT, which uses GPT-4 (Internet search ⁇ https://openai.com/gpt-4>) or GPT-4-32k.
• At least a portion of the information included in the hemostasis treatment-related information 104 is used for ChatGPT, for example, at least a portion of the information included in the hemostasis treatment-related information 104 is used as instruction data (e.g., prompts) to be input to ChatGPT.
• instruction data e.g., prompts
• An example of information output from ChatGPT is information about medical treatment derived from hemostasis treatment-related information 104 entered into ChatGPT (e.g., information indicating the details of medical treatment recommended to be performed after hemostasis treatment).
• recognition processing 96 using AI with a segmentation method is exemplified, but the present disclosure is not limited to this.
• the present disclosure also applies to recognition processing using AI with a bounding box method.
• FIG. 12 is a conceptual diagram showing an example of the configuration of an endoscope device 106.
• the endoscope device 106 is an example of an "endoscope device" according to the present disclosure.
• the endoscope device 106 differs from the endoscope device 10 described in the above embodiment in that it has an external device 108.
• the external device 108 is communicatively connected to the computer 78 via a network 110 (e.g., a WAN and/or a LAN, etc.).
• a network 110 e.g., a WAN and/or a LAN, etc.
• An example of an external device 108 is at least one server that directly or indirectly transmits and receives data to and from the computer 78 via the network 110.
• the external device 108 receives a processing execution instruction provided from the processor 82 of the computer 78 via the network 110.
• the external device 108 then executes processing according to the received processing execution instruction and transmits the processing results to the computer 78 via the network 110.
• the processor 82 receives the processing results transmitted from the external device 108 via the network 110 and executes processing using the received processing results.
• the processing execution instruction may be, for example, an instruction to have the external device 108 execute at least a part of the medical support processing.
• a first example of at least a part of the medical support processing i.e., a processing to be executed by the external device 108, is the recognition processing 96.
• the external device 108 executes the recognition processing 96 in accordance with the processing execution instruction given from the processor 82 via the network 110, and transmits the recognition result 98 to the computer 78 via the network 110.
• the processor 82 receives the recognition result 98, and executes the same processing as in the above embodiment using the received recognition result 98.
• a second example of at least a portion of the medical support process is the process by the control unit 82B.
• the external device 108 executes the process by the control unit 82B according to the process execution instruction given from the processor 82 via the network 110, and transmits the process result (e.g., the hemostasis period specification information 102 and/or the hemostasis treatment related information 104, etc.) to the computer 78 via the network 110.
• the processor 82 receives the process result, and uses the received process result to execute the same process as in the above embodiment (e.g., display using the display device 18 and/or storage in the storage area 84A, etc.).
• the external device 108 is realized by cloud computing.
• cloud computing is merely one example, and the external device 108 may be realized by network computing such as fog computing, edge computing, or grid computing.
• network computing such as fog computing, edge computing, or grid computing.
• at least one personal computer or the like may be used as the external device 108.
• the external device 108 may be a computing device with a communication function equipped with multiple types of AI functions.
• the medical support program 90 may be stored in a portable, computer-readable, non-transitory storage medium such as an SSD or USB memory.
• the medical support program 90 stored in the non-transitory storage medium is installed in the computer 78 of the endoscope device 10.
• the processor 82 executes the medical support process in accordance with the medical support program 90.
• the medical support program 90 may be stored in a storage device such as another computer or server connected to the endoscope device 10 via a network, and the medical support program 90 may be downloaded and installed in the computer 78 in response to a request from the endoscope device 10.
• processors can be used as hardware resources for executing medical support processing.
• processors include a CPU, which is a general-purpose processor that functions as a hardware resource for executing medical support processing by executing software, i.e., a program.
• processors include dedicated electrical circuits, which are processors with a circuit configuration designed specifically for executing specific processing, such as an FPGA, PLD, or ASIC. All of these processors have built-in or connected memory, and all of these processors execute medical support processing by using the memory.
• the hardware resource that executes the medical support processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the medical support processing may be a single processor.
• a configuration using a single processor first, there is a configuration in which one processor is configured using a combination of one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Secondly, there is a configuration in which a processor is used that realizes the functions of the entire system, including multiple hardware resources that execute medical support processing, on a single IC chip, as typified by SoCs. In this way, medical support processing is realized using one or more of the various processors listed above as hardware resources.
• the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.
• the above medical support process is merely one example. It goes without saying that unnecessary steps can be deleted, new steps can be added, and the processing order can be changed without departing from the spirit of the invention.

Landscapes

• Health & Medical Sciences (AREA)
• Life Sciences & Earth Sciences (AREA)
• Surgery (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Biomedical Technology (AREA)
• Optics & Photonics (AREA)
• Pathology (AREA)
• Radiology & Medical Imaging (AREA)
• Biophysics (AREA)
• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Heart & Thoracic Surgery (AREA)
• Medical Informatics (AREA)
• Molecular Biology (AREA)
• Animal Behavior & Ethology (AREA)
• General Health & Medical Sciences (AREA)
• Public Health (AREA)
• Veterinary Medicine (AREA)
• Endoscopes (AREA)

Priority Applications (2)

Applications Claiming Priority (2)

Publications (1)

Family

ID=94374784

Family Applications (1)

Country Status (3)

Citations (4)

• 2024
  • 2024-03-26 JP JP2025535569A patent/JPWO2025022718A1/ja active Pending
  • 2024-03-26 CN CN202480043328.3A patent/CN121443205A/zh active Pending
  • 2024-03-26 WO PCT/JP2024/012108 patent/WO2025022718A1/ja active Pending

Patent Citations (4)

Also Published As

Similar Documents

Legal Events