Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/00002—Operational features of endoscopes
  • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
  • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
  • A61B1/000094—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope extracting biological structures
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/00002—Operational features of endoscopes
  • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
  • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
  • A61B1/000095—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope for image enhancement
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/00002—Operational features of endoscopes
  • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
  • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
  • A61B1/000096—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope using artificial intelligence
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/00002—Operational features of endoscopes
  • A61B1/00043—Operational features of endoscopes provided with output arrangements
  • A61B1/00045—Display arrangement
  • A61B1/0005—Display arrangement combining images e.g. side-by-side, superimposed or tiled
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
  • G06T7/0012—Biomedical image inspection
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/50—Depth or shape recovery
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/60—Analysis of geometric attributes
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/70—Determining position or orientation of objects or cameras
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/20—Image preprocessing
  • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/31—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for the rectum, e.g. proctoscopes, sigmoidoscopes, colonoscopes
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10068—Endoscopic image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20084—Artificial neural networks [ANN]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
  • G06T2207/30096—Tumor; Lesion
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
  • G06V2201/03—Recognition of patterns in medical or anatomical images

Definitions

• the technology disclosed herein relates to a medical support device, an endoscope system, a medical support method, and a program.
• WO 2020/110214 discloses an endoscope system that includes an image input unit, a lesion detection unit, an oversight risk analysis unit, a notification control unit, and a notification unit.
• the lesion detection unit detects the lesion, which is the subject of observation with the endoscope, from the observation images.
• the oversight risk analysis unit determines the degree of oversight risk, which is the risk that the operator will overlook the lesion, based on the observation images.
• the notification control unit controls the notification means and method for the detection of the lesion based on the degree of oversight risk.
• the notification unit notifies the operator of the detection of the lesion based on the control of the notification control unit.
• the oversight risk analysis unit includes a lesion analysis unit that analyzes the oversight risk based on the state of the lesion.
• the lesion analysis unit includes a lesion size analysis unit that estimates the size of the lesion itself.
• One embodiment of the technology disclosed herein provides a medical support device, an endoscope system, a medical support method, and a program that enable a user or the like to visually recognize multiple observation target areas shown in a medical image in a state in which the characteristics of each of the multiple observation target areas can be grasped without impairing the visibility of the medical image.
• a first aspect of the technology disclosed herein is a medical support device that includes a processor, the processor performing a recognition process on a medical image showing multiple observation target regions to acquire the characteristics of each of the multiple observation target regions, displays the medical image in a first display area, and displays multiple extracted images in which the multiple observation target regions are individually extracted from the medical image in a second display area outside the first display area according to the characteristics.
• a second aspect of the technology disclosed herein is a medical support device according to the first aspect, in which the characteristics include size.
• a third aspect of the technology disclosed herein is a medical support device according to the second aspect, in which the second display area displays multiple extracted images in a display manner that allows visual identification of the size relationship between multiple observation target areas.
• a fourth aspect of the technology disclosed herein is a medical support device according to the second or third aspect, in which the size is divided into a plurality of first ranges, and a plurality of extracted images are displayed in a grouped state according to the first range in the second display area.
• a fifth aspect of the technology disclosed herein is a medical support device according to the fourth aspect, in which, when multiple extracted images are grouped according to a first range, an extracted image representative of the first range is displayed in the second display area, and information regarding the number of extracted images grouped into the first range is also displayed.
• a sixth aspect of the technology disclosed herein is a medical support device according to the fourth or fifth aspect, in which the second display area displays the multiple extracted images grouped by the first range when the number of multiple extracted images exceeds a preset number.
• a seventh aspect of the technology disclosed herein is a medical support device according to any one of the first to sixth aspects, in which the characteristic includes depth.
• An eighth aspect of the technology disclosed herein is a medical support device according to the seventh aspect, in which the second display area displays multiple extracted images in a display manner that allows visual identification of the depth relationship between multiple observation target areas.
• a ninth aspect of the technology disclosed herein is a medical support device according to the seventh or eighth aspect, in which the depth is divided into a plurality of second ranges, and a plurality of extracted images are displayed in a grouped state according to the second range in the second display area.
• a tenth aspect of the technology disclosed herein is a medical support device according to the ninth aspect, in which, when multiple extracted images are grouped according to a second range, an extracted image representative of the second range is displayed in the second display area, and information regarding the number of extracted images grouped into the second range is also displayed.
• An eleventh aspect of the technology disclosed herein is a medical support device according to the ninth or tenth aspect, in which the second display area displays the multiple extracted images grouped by second range when the number of multiple extracted images exceeds a preset number.
• a twelfth aspect of the technology disclosed herein is a medical support device according to any one of the first to eleventh aspects, in which a processor displays positional relationship identification information on a screen, and the positional relationship identification information is information capable of identifying a correspondence between a first display position where at least one of a plurality of extracted images is displayed and a second display position where an observation target area shown in an extracted image displayed at the first display position is displayed in the first display area.
• a thirteenth aspect of the technology disclosed herein is a medical support device according to any one of the first to twelfth aspects, in which the second display area displays a plurality of extracted images grouped according to a common characteristic.
• a fourteenth aspect of the technology disclosed herein is a medical support device according to the thirteenth aspect, in which, when multiple extracted images are grouped according to a common characteristic, an extracted image representative of each common characteristic is displayed in the second display area, and information regarding the number of extracted images grouped into the common characteristic is also displayed.
• a fifteenth aspect of the technology disclosed herein is a medical support device according to the thirteenth or fourteenth aspect, in which, when the number of extracted images exceeds a predetermined number, the second display area displays the extracted images grouped according to common characteristics.
• a sixteenth aspect of the technology disclosed herein is a medical support device according to any one of the first to fifteenth aspects, in which the processor displays, in the third display area, position identification information that can identify the display position within the medical image of the observation target area included in the extracted image.
• a seventeenth aspect of the technology disclosed herein is a medical support device according to the sixteenth aspect, in which the position identification information is a map that identifies the display position within the medical image.
• An 18th aspect of the technology disclosed herein is a medical support device according to the 17th aspect, in which the recognition process is an object recognition process using machine learning, and the map is generated based on a probability map obtained by performing the object recognition process.
• a 19th aspect of the technology disclosed herein is a medical support device according to any one of the 16th to 18th aspects, in which the third display area is located in a position different from the first and second display areas.
• a twentieth aspect of the technology disclosed herein is a medical support device according to any one of the first to nineteenth aspects, in which the actual size of each of the multiple observation target regions is measured, and the second display region displays the actual sizes corresponding to the multiple extracted images in a state in which the correspondence with the multiple extracted images can be identified.
• a twenty-first aspect of the technology disclosed herein is a medical support device according to any one of the first to twentieth aspects, in which the extracted image is an image extracted from a medical image using a frame that allows visual discrimination of the difference in size of the observation area between multiple extracted images when the multiple extracted images are compared.
• a twenty-second aspect of the technology disclosed herein is a medical support device according to the twenty-first aspect, in which the frame has a common shape and size among multiple observation target regions.
• a 23rd aspect of the technology disclosed herein is a medical support device according to any one of the 1st to 22nd aspects, in which the medical image is an endoscopic image obtained by capturing an image using an endoscopic scope.
• the 24th aspect of the technology disclosed herein is a medical support device according to any one of the 1st to 23rd aspects, in which the observation target area is a lesion.
• a twenty-fifth aspect of the technology disclosed herein is an endoscope system including a medical support device according to any one of the first to twenty-fourth aspects, and an endoscope scope that is inserted into the body to capture images of the inside of the body to obtain medical images.
• a 26th aspect of the technology disclosed herein is a medical support method that includes performing a recognition process on a medical image showing multiple observation target regions, acquiring the characteristics of each of the multiple observation target regions recognized by the recognition process, displaying the medical image in a first display area, and displaying multiple extracted images in which the multiple observation target regions are individually extracted from the medical image in a second display area outside the first display area according to the characteristics.
• a 27th aspect of the technology disclosed herein is a medical support method according to the 26th aspect, which includes the use of an endoscope that is inserted into the body to capture images of the inside of the body and thereby obtain medical images.
• FIG. 1 is a conceptual diagram showing an example of an aspect in which an endoscope system is used.
• 1 is a conceptual diagram showing an example of an overall configuration of an endoscope system.
• 2 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope system.
• FIG. 2 is a block diagram showing an example of main functions of a processor included in a medical support device according to an embodiment, and an example of information stored in an NVM.
• FIG. FIG. 4 is a conceptual diagram showing an example of processing contents of a recognition unit and a control unit.
• 11 is a conceptual diagram showing an example of processing content in which a recognition unit associates a unique identifier with each of a plurality of segmentation images.
• FIG. 1 is a conceptual diagram showing an example of an aspect in which an endoscope system is used.
• 1 is a conceptual diagram showing an example of an overall configuration of an endoscope system.
• 2 is a block diagram showing an example of a hardware configuration of an electrical system of
• FIG. 13 is a conceptual diagram showing an example of processing contents in which the control unit displays a frame 40 in the first display area, and also displays a plurality of local images in the second display area according to the size of the lesion.
• FIG. 13 is a flowchart showing an example of the flow of a medical support process. This is a continuation of the flowchart shown in Figure 12A.
• 13 is a conceptual diagram showing an example of a process in which an acquisition unit generates third information and stores it in a RAM.
• FIG. 11 is a conceptual diagram showing an example of processing contents in which a control unit displays a frame 40 in a first display area, and also displays a plurality of local images in a second display area according to distance information.
• FIG. 15 is a conceptual diagram showing an example of a manner in which the display contents of the screen shown in FIG. 11 and the display contents of the screen shown in FIG. 14 are switched in accordance with a given instruction.
• 11 is a conceptual diagram showing an example of processing contents in which the control unit groups a plurality of local images by size range and displays them in the second display area.
• FIG. 11 is a conceptual diagram showing an example of processing contents in which the control unit groups a plurality of local images by distance range and displays the grouped local images in the second display area.
• FIG. 13 is a conceptual diagram showing an example of a process performed by a control unit to display a map in a third display area.
• FIG. FIG. 2 is a conceptual diagram showing an example of an output destination of various information.
• CPU is an abbreviation for "Central Processing Unit”.
• GPU is an abbreviation for "Graphics Processing Unit”.
• RAM is an abbreviation for "Random Access Memory”.
• NVM is an abbreviation for "Non-volatile memory”.
• EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory”.
• ASIC is an abbreviation for "Application Specific Integrated Circuit”.
• PLD is an abbreviation for "Programmable Logic Device”.
• FPGA is an abbreviation for "Field-Programmable Gate Array”.
• SoC is an abbreviation for "System-on-a-chip”.
• SSD is an abbreviation for "Solid State Drive”.
• USB is an abbreviation for "Universal Serial Bus”.
• HDD is an abbreviation for "Hard Disk Drive”.
• EL is an abbreviation for "Electro-Luminescence”.
• CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor”.
• CCD is an abbreviation for "Charge Coupled Device”.
• AI is an abbreviation for "Artificial Intelligence”.
• BLI is an abbreviation for "Blue Light Imaging”.
• LCI is an abbreviation for "Linked Color Imaging”.
• I/F is an abbreviation for "Interface”.
• SSL is an abbreviation for "Sessile Serrated Lesion”.
• LAN is an abbreviation for "Local Area Network”.
• WAN is an abbreviation for "Wide Area Network”.
• an endoscope system 10 is used by a doctor 12 in an endoscopic examination or the like.
• the endoscopic examination is assisted by staff such as a nurse 14.
• the endoscope system 10 is an example of an "endoscope system" according to the technology disclosed herein.
• the endoscope system 10 is communicatively connected to a communication device (not shown), and information obtained by the endoscope system 10 is transmitted to the communication device.
• a communication device is a server and/or a client terminal (e.g., a personal computer and/or a tablet terminal, etc.) that manages various information such as electronic medical records.
• the communication device receives the information transmitted from the endoscope system 10 and executes processing using the received information (e.g., processing to store in an electronic medical record, etc.).
• the endoscope system 10 includes an endoscope scope 16, a display device 18, a light source device 20, a control device 22, and a medical support device 24.
• the endoscope scope 16 is an example of an "endoscope scope" according to the technology disclosed herein.
• the endoscope system 10 is a modality for performing medical treatment on the large intestine 28 contained within the body of a subject 26 (e.g., a patient) using an endoscope scope 16.
• a subject 26 e.g., a patient
• an endoscope scope 16 In this embodiment, the large intestine 28 is the object observed by the doctor 12.
• the endoscope 16 is used by the doctor 12 and inserted into the body cavity of the subject 26.
• the endoscope 16 is inserted into the large intestine 28 of the subject 26.
• the endoscope system 10 causes the endoscope 16 inserted into the large intestine 28 of the subject 26 to capture images of the inside of the large intestine 28 of the subject 26, and performs various medical procedures on the large intestine 28 as necessary.
• the endoscope system 10 obtains and outputs an image showing the state inside the large intestine 28 by imaging the inside of the large intestine 28 of the subject 26.
• the endoscope system 10 is an endoscope with an optical imaging function that irradiates light 30 inside the large intestine 28 and captures an image of the reflected light obtained by reflection from the intestinal wall 32 of the large intestine 28.
• the light source device 20, the control device 22, and the medical support device 24 are installed on a wagon 34.
• the wagon 34 has multiple platforms arranged in the vertical direction, and the medical support device 24, the control device 22, and the light source device 20 are installed from the lower platform to the upper platform.
• the display device 18 is installed on the top platform of the wagon 34.
• the control device 22 controls the entire endoscope system 10. Under the control of the control device 22, the medical support device 24 performs various image processing on the images obtained by capturing images of the intestinal wall 32 by the endoscope scope 16.
• the display device 18 displays various information including images. Examples of the display device 18 include a liquid crystal display and an EL display. Also, a tablet terminal with a display may be used in place of the display device 18 or together with the display device 18.
• a screen 35 is displayed on the display device 18.
• the screen 35 includes a plurality of display areas.
• the plurality of display areas are arranged side by side within the screen 35.
• a first display area 36 and a second display area 38 are shown as examples of the plurality of display areas.
• the size of the first display area 36 is larger than the size of the second display area 38.
• the first display area 36 is used as the main display area, and the second display area 38 is used as the sub-display area. Note that the size relationship between the first display area 36 and the second display area 38 is not limited to this, and may be any size relationship that fits within the screen 35.
• screen 35 is an example of a "screen” according to the technology of the present disclosure
• first display area 36 is an example of a “first display area” according to the technology of the present disclosure
• second display area 38 is an example of a "second display area” according to the technology of the present disclosure.
• the first display area 36 displays an endoscopic moving image 39.
• the endoscopic moving image 39 is a moving image acquired by imaging the intestinal wall 32 within the large intestine 28 of the subject 26 using the endoscope scope 16.
• a moving image showing the intestinal wall 32 is shown as an example of the endoscopic moving image 39.
• the intestinal wall 32 shown in the endoscopic video 39 includes multiple lesions 42 (e.g., one lesion 42 in the example shown in FIG. 1) as multiple regions of interest (i.e., multiple observation target regions) gazed upon by the physician 12, and the physician 12 can visually recognize the appearance of the intestinal wall 32 including the multiple lesions 42 through the endoscopic video 39.
• the lesion 42 is an example of an "observation target region" and a "lesion” according to the technology disclosed herein.
• neoplastic polyps examples include neoplastic polyps and non-neoplastic polyps.
• examples of the types of neoplastic polyps include adenomatous polyps (e.g., SSL).
• examples of the types of non-neoplastic polyps include hamartomatous polyps, hyperplastic polyps, and inflammatory polyps. Note that the types exemplified here are types that are anticipated in advance as types of lesions 42 when an endoscopic examination is performed on the large intestine 28, and the types of lesions will differ depending on the organ in which the endoscopic examination is performed.
• a lesion 42 is shown as an example, but this is merely one example, and the area of interest (i.e., the area to be observed) that is gazed upon by the doctor 12 may be an organ (e.g., the duodenal papilla), a marked area, an artificial treatment tool (e.g., an artificial clip), or a treated area (e.g., an area where traces remain after the removal of a polyp, etc.), etc.
• an organ e.g., the duodenal papilla
• an artificial treatment tool e.g., an artificial clip
• a treated area e.g., an area where traces remain after the removal of a polyp, etc.
• the image displayed in the first display area 36 is one frame 40 included in a moving image that is composed of multiple frames 40 in chronological order.
• the first display area 36 displays multiple frames 40 in chronological order at a default frame rate (e.g., several tens of frames per second).
• the frame 40 is an example of a "medical image” and an "endoscopic image” related to the technology disclosed herein.
• a moving image displayed in the first display area 36 is a moving image in a live view format.
• the live view format is merely one example, and the moving image may be temporarily stored in a memory or the like and then displayed, like a moving image in a post-view format.
• each frame included in a recording moving image stored in a memory or the like may be played back and displayed on the screen 35 (for example, the first display area 36) as an endoscopic moving image 39.
• the second display area 38 exists outside the first display area 36.
• the second display area 38 is adjacent to the first display area 36, and is displayed on the right side of the screen 35 when viewed from the front.
• the display position of the second display area 38 may be anywhere different from the first display area 36, but it is preferable that it is displayed in a position that can be compared with the endoscopic video image 39 displayed in the first display area 36.
• the second display area 38 displays medical information 44, which is information related to medical care.
• the medical information 44 include information that assists the doctor 12 in making medical decisions.
• information that assists the doctor 12 in making medical decisions is various information about the subject 26 into which the endoscope 16 is inserted, and/or various information obtained by performing AI-based processing on the endoscope video image 39. Further details of the medical information 44 will be described later.
• the endoscope 16 includes an operating section 46 and an insertion section 48.
• the insertion section 48 is partially curved by operating the operating section 46.
• the insertion section 48 is inserted into the large intestine 28 (see FIG. 1) while curving in accordance with the shape of the large intestine 28, in accordance with the operation of the operating section 46 by the doctor 12 (see FIG. 1).
• the tip 50 of the insertion section 48 is provided with a camera 52, a lighting device 54, and an opening 56 for a treatment tool.
• the camera 52 and lighting device 54 are provided on the tip surface 50A of the tip 50. Note that, although an example in which the camera 52 and lighting device 54 are provided on the tip surface 50A of the tip 50 is given here, this is merely one example, and the camera 52 and lighting device 54 may be provided on the side surface of the tip 50, so that the endoscope 16 is configured as a side-viewing mirror.
• the camera 52 is inserted into the body cavity of the subject 26 to capture an image of the observation area.
• the camera 52 captures an image of the inside of the subject 26 (e.g., inside the large intestine 28) to obtain an endoscopic moving image 39.
• One example of the camera 52 is a CMOS camera.
• this is merely one example, and other types of cameras such as a CCD camera may also be used.
• the illumination device 54 has illumination windows 54A and 54B.
• the illumination device 54 irradiates light 30 (see FIG. 1) through the illumination windows 54A and 54B.
• Examples of the type of light 30 irradiated from the illumination device 54 include visible light (e.g., white light) and non-visible light (e.g., near-infrared light).
• the illumination device 54 also irradiates special light through the illumination windows 54A and 54B. Examples of the special light include light for BLI and/or light for LCI.
• the camera 52 captures images of the inside of the large intestine 28 by optical techniques while the light 30 is irradiated inside the large intestine 28 by the illumination device 54.
• the treatment tool opening 56 is an opening for allowing the treatment tool 58 to protrude from the tip 50.
• the treatment tool opening 56 is also used as a suction port for sucking blood and internal waste, and as a delivery port for delivering fluids.
• the operating section 46 is formed with a treatment tool insertion port 60, and the treatment tool 58 is inserted into the insertion section 48 from the treatment tool insertion port 60.
• the treatment tool 58 passes through the insertion section 48 and protrudes to the outside from the treatment tool opening 56.
• a puncture needle is shown as the treatment tool 58 protruding from the treatment tool opening 56.
• a puncture needle is shown as the treatment tool 58, but this is merely one example, and the treatment tool 58 may be a grasping forceps, a papillotomy knife, a snare, a catheter, a guidewire, a cannula, and/or a puncture needle with a guide sheath, etc.
• the endoscope scope 16 is connected to the light source device 20 and the control device 22 via a universal cord 62.
• the medical support device 24 and the reception device 64 are connected to the control device 22.
• the display device 18 is also connected to the medical support device 24.
• the control device 22 is connected to the display device 18 via the medical support device 24.
• the medical support device 24 is exemplified here as an external device for expanding the functions performed by the control device 22, an example is given in which the control device 22 and the display device 18 are indirectly connected via the medical support device 24, but this is merely one example.
• the display device 18 may be directly connected to the control device 22.
• the function of the medical support device 24 may be included in the control device 22, or the control device 22 may be equipped with a function for causing a server (not shown) to execute the same processing as that executed by the medical support device 24 (for example, the medical support processing described below) and for receiving and using the results of the processing by the server.
• the reception device 64 receives instructions from the doctor 12 and outputs the received instructions as an electrical signal to the control device 22.
• Examples of the reception device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and/or a remote control device.
• the control device 22 controls the light source device 20, exchanges various signals with the camera 52, and exchanges various signals with the medical support device 24.
• the light source device 20 emits light under the control of the control device 22 and supplies the light to the illumination device 54.
• the illumination device 54 has a built-in light guide, and the light supplied from the light source device 20 passes through the light guide and is irradiated from illumination windows 54A and 54B.
• the control device 22 causes the camera 52 to capture an image, acquires an endoscopic video image 39 (see FIG. 1) from the camera 52, and outputs it to a predetermined output destination (e.g., the medical support device 24).
• the medical support device 24 performs various types of image processing on the endoscopic video image 39 input from the control device 22 to provide medical support (here, endoscopic examination as an example).
• the medical support device 24 outputs the endoscopic video image 39 that has been subjected to various types of image processing to a predetermined output destination (e.g., the display device 18).
• the endoscopic video image 39 output from the control device 22 is output to the display device 18 via the medical support device 24, but this is merely one example.
• the control device 22 and the display device 18 may be connected, and the endoscopic video image 39 that has been subjected to image processing by the medical support device 24 may be displayed on the display device 18 via the control device 22.
• the control device 22 includes a computer 66, a bus 68, and an external I/F 70.
• the computer 66 includes a processor 72, a RAM 74, and an NVM 76.
• the processor 72, the RAM 74, the NVM 76, and the external I/F 70 are connected to the bus 68.
• the processor 72 has at least one CPU and at least one GPU, and controls the entire control device 22.
• the GPU operates under the control of the CPU, and is responsible for executing various graphic processing operations and performing calculations using neural networks.
• the processor 72 may be one or more CPUs with integrated GPU functionality, or one or more CPUs without integrated GPU functionality.
• the computer 66 is equipped with one processor 72, but this is merely one example, and the computer 66 may be equipped with multiple processors 72.
• RAM 74 is a memory in which information is temporarily stored, and is used as a work memory by processor 72.
• NVM 76 is a non-volatile storage device that stores various programs and various parameters, etc.
• An example of NVM 76 is a flash memory (e.g., EEPROM and/or SSD). Note that flash memory is merely one example, and other non-volatile storage devices such as HDDs may also be used, or a combination of two or more types of non-volatile storage devices may also be used.
• the external I/F 70 is responsible for transmitting various types of information between the processor 72 and one or more devices (hereinafter also referred to as "first external devices") that exist outside the control device 22.
• first external devices One example of the external I/F 70 is a USB interface.
• the camera 52 is connected to the external I/F 70 as one of the first external devices, and the external I/F 70 is responsible for the exchange of various information between the camera 52 and the processor 72.
• the processor 72 controls the camera 52 via the external I/F 70.
• the processor 72 also acquires, via the external I/F 70, endoscopic video images 39 (see FIG. 1) obtained by the camera 52 capturing an image of the inside of the large intestine 28 (see FIG. 1).
• the light source device 20 is connected to the external I/F 70 as one of the first external devices, and the external I/F 70 is responsible for the exchange of various information between the light source device 20 and the processor 72.
• the light source device 20 supplies light to the lighting device 54 under the control of the processor 72.
• the lighting device 54 irradiates the light supplied from the light source device 20.
• the external I/F 70 is connected to the reception device 64 as one of the first external devices, and the processor 72 acquires instructions received by the reception device 64 via the external I/F 70 and executes processing according to the acquired instructions.
• the medical support device 24 includes a computer 78 and an external I/F 80.
• the computer 78 includes a processor 82, a RAM 84, and an NVM 86.
• the processor 82, the RAM 84, the NVM 86, and the external I/F 80 are connected to a bus 88.
• the medical support device 24 is an example of a "medical support device” according to the technology of the present disclosure
• the computer 78 is an example of a "computer” according to the technology of the present disclosure
• the processor 82 is an example of a "processor" according to the technology of the present disclosure.
• computer 78 i.e., processor 82, RAM 84, and NVM 86
• processor 82, RAM 84, and NVM 86 is basically the same as the hardware configuration of computer 66, so a description of the hardware configuration of computer 78 will be omitted here.
• the external I/F 80 is responsible for transmitting various types of information between the processor 82 and one or more devices (hereinafter also referred to as "second external devices") that exist outside the medical support device 24.
• second external devices One example of the external I/F 80 is a USB interface.
• the control device 22 is connected to the external I/F 80 as one of the second external devices.
• the external I/F 70 of the control device 22 is connected to the external I/F 80.
• the external I/F 80 is responsible for the exchange of various information between the processor 82 of the medical support device 24 and the processor 72 of the control device 22.
• the processor 82 acquires endoscopic video images 39 (see FIG. 1) from the processor 72 of the control device 22 via the external I/Fs 70 and 80, and performs various image processing on the acquired endoscopic video images 39.
• the display device 18 is connected to the external I/F 80 as one of the second external devices.
• the processor 82 controls the display device 18 via the external I/F 80 to cause the display device 18 to display various information (e.g., endoscopic moving image 39 that has been subjected to various image processing).
• the doctor 12 checks the endoscopic video 39 via the display device 18 and determines whether or not medical treatment is required for the lesion 42 shown in the endoscopic video 39, and performs medical treatment on the lesion 42 if necessary.
• the size of the lesion 42 is an important factor in determining whether or not medical treatment is required.
• the doctor 12 when the size of the lesion 42 is measured, it is necessary to communicate the measured size to the doctor 12 accurately and without interfering with the endoscopic examination.
• the doctor 12 when multiple lesions 42 are shown in the frame 40, it is required that the doctor 12 be able to visually recognize the multiple lesions 42 shown in the frame 40 in a state in which the size and other characteristics of each of the multiple lesions 42 can be grasped without impairing the visibility of the frame 40 displayed on the screen 35.
• medical support processing is performed by the processor 82 of the medical support device 24, as shown in FIG. 4.
• NVM 86 stores a medical support program 90.
• the medical support program 90 is an example of a "program" according to the technology of the present disclosure.
• the processor 82 reads the medical support program 90 from NVM 86 and executes the read medical support program 90 on RAM 84 to perform medical support processing.
• the medical support processing is realized by the processor 82 operating as a recognition unit 82A, an acquisition unit 82B, and a control unit 82C in accordance with the medical support program 90 executed on RAM 84.
• the NVM 86 stores a recognition model 92 and a distance derivation model 94.
• the recognition model 92 is used by the recognition unit 82A
• the distance derivation model 94 is used by the acquisition unit 82B.
• the recognition unit 82A and the control unit 82C acquire each of a plurality of frames 40 in chronological order contained in the endoscopic moving image 39 generated by the camera 52 capturing images at an imaging frame rate (e.g., several tens of frames/second) from the camera 52, one frame at a time in chronological order.
• an imaging frame rate e.g., several tens of frames/second
• the control unit 82C outputs the endoscopic moving image 39 to the display device 18. For example, the control unit 82C displays the endoscopic moving image 39 as a live view image in the first display area 36. That is, each time the control unit 82C acquires a frame 40 from the camera 52, the control unit 82C displays the acquired frame 40 in sequence in the first display area 36 according to the display frame rate (e.g., several tens of frames per second). The control unit 82C also displays medical information 44 in the second display area 38. For example, the control unit 82C also updates the display content of the second display area 38 (e.g., medical information 44) in accordance with the display content of the first display area 36.
• the display frame rate e.g., several tens of frames per second.
• the control unit 82C also displays medical information 44 in the second display area 38.
• the control unit 82C also updates the display content of the second display area 38 (e.g., medical information 44) in accordance with the display content of the first display area
• the recognition unit 82A uses the endoscopic video 39 acquired from the camera 52 to recognize the multiple lesions 42 that appear in the endoscopic video 39. That is, the recognition unit 82A recognizes the multiple lesions 42 that appear in the frames 40 by sequentially performing a recognition process 96 on each of the multiple frames 40 in a time series contained in the endoscopic video 39 acquired from the camera 52. For example, the recognition unit 82A recognizes the geometric characteristics (e.g., position and shape, etc.) of each of the multiple lesions 42, the type of each of the multiple lesions 42, and the type of each of the multiple lesions 42 (e.g., pedunculated, subpedunculated, sessile, surface elevated, surface flat, surface depressed, etc.).
• geometric characteristics e.g., position and shape, etc.
• the recognition process 96 is performed by the recognition unit 82A on the acquired frame 40 each time the frame 40 is acquired.
• the recognition process 96 is a process for recognizing multiple lesions 42 using an AI-based method (i.e., object recognition process using machine learning).
• object recognition process using AI in a segmentation method e.g., semantic segmentation, instance segmentation, and/or panoptic segmentation
• a segmentation method e.g., semantic segmentation, instance segmentation, and/or panoptic segmentation
• the recognition model 92 is a trained model for object recognition using an AI segmentation method.
• An example of a trained model for object recognition using an AI segmentation method is a model for semantic segmentation.
• An example of a model for semantic segmentation is a model with an encoder-decoder structure.
• An example of a model with an encoder-decoder structure is U-Net or HRNet.
• the recognition process 96 is an example of the "recognition process" and "object recognition process” related to the technology disclosed herein.
• the recognition model 92 is optimized by performing machine learning on the neural network using the first training data.
• the first training data is a data set including a plurality of data (i.e., a plurality of frames of data) in which the first example data and the first correct answer data are associated with each other.
• the first example data is an image corresponding to frame 40.
• the first correct answer data is correct answer data (i.e., annotations) for the first example data.
• annotations that identify the geometric characteristics, type, and model of the lesion depicted in the image used as the first example data are used as an example of the first correct answer data.
• the recognition unit 82A acquires a frame 40 from the camera 52 and inputs the acquired frame 40 to the recognition model 92. As a result, each time a frame 40 is input, the recognition model 92 identifies the geometric characteristics of each of the multiple lesions 42 depicted in the input frame 40, and outputs information that can identify the geometric characteristics of each of the multiple lesions 42. In the example shown in FIG. 5, a probability map 100 that is information that can identify the position of the lesion 42 in the frame 40 is shown as an example of information that can identify geometric characteristics.
• the recognition unit 82A acquires from the recognition model 92 information indicating the type of each of the multiple lesions 42 depicted in the frame 40 input to the recognition model 92, and information indicating the type of each of the multiple lesions 42 depicted in the frame 40 input to the recognition model 92.
• the recognition unit 82A obtains a probability map 100 for the frame 40 input to the recognition model 92 from the recognition model 92.
• the probability map 100 is a map that expresses the distribution of the positions of the lesions 42 within the frame 40 in terms of probability, which is an example of an index of likelihood. In general, the probability map 100 is also called a reliability map or a certainty map.
• the probability map 100 includes a plurality of segmentation images 102 that define a plurality of lesions 42 recognized by the recognition unit 82A.
• the segmentation images 102 are image regions that identify the positions of the lesions 42 in the frame 40 that have been recognized by performing the recognition process 96 on the frame 40 (i.e., images displayed in a display manner that can identify the positions in the frame 40 where the lesions 42 are most likely to exist).
• Each segmentation image 102 is associated with first position information 98 by the recognition unit 82A.
• An example of the first position information 98 in this case is coordinates that can identify the position of the segmentation image 102 in the probability map 100 (in other words, coordinates that can identify the display position of the lesion 42 in the frame 40).
• the position of the segmentation image 102 in the probability map 100 refers to, for example, the position of the outer contour of the segmentation image 102 in the probability map 100.
• the probability map 100 may be displayed on the screen 35 (e.g., the second display area 38) as medical information 44 by the control unit 82C.
• the probability map 100 displayed on the screen 35 is updated according to the display frame rate applied to the first display area 36. That is, the display of the probability map 100 in the second display area 38 (i.e., the display of the segmentation image 102) is updated in synchronization with the display timing of the endoscopic video 39 displayed in the first display area 36.
• the doctor 12 can grasp the general position of the lesion 42 in the endoscopic video 39 displayed in the first display area 36 by referring to the probability map 100 displayed in the second display area 38 while observing the endoscopic video 39 displayed in the first display area 36.
• the recognition unit 82A associates, with each of the multiple segmentation images 102 in the probability map 100, an identifier 104 that can individually identify each segmentation image 102.
• the identifier 104 is a unique identifier for each segmentation image 102.
• the association of the identifier 104 with the segmentation image 102 is achieved by assigning the identifier 104 to the first position information 98 that corresponds to the segmentation image 102.
• the identifier 104 is an example of "positional relationship identifying information" related to the technology of the present disclosure.
• the control unit 82C creates a first rectangular frame 106 circumscribing the segmentation image 102 based on the first position information 98.
• the control unit 82C then creates multiple second rectangular frames 108 in the frame 40 based on the multiple first rectangular frames 106 created in the probability map 100.
• the second rectangular frame 108 is a rectangular frame that surrounds the lesion 42 in the frame 40, and is assigned to each of the multiple lesions 42 shown in the frame 40.
• the second rectangular frame 108 is obtained by enlarging the largest first rectangular frame 106 of the multiple first rectangular frames 106 created in the probability map 100 by a preset magnification (e.g., a magnification greater than 1).
• the control unit 82C associates the same identifier 104 as the identifier 104 associated with the segmentation image 102 with the lesion 42 corresponding to the segmentation image 102 in the probability map 100. That is, a common identifier 104 is associated with the segmentation image 102 and the lesion 42 whose positions in the probability map 100 correspond to their positions in the frame 40. The association of the identifier 104 with the lesion 42 is achieved by assigning the identifier 104 associated with the segmentation image 102 corresponding to the lesion 42 to the second rectangular frame 108 assigned to the lesion 42 corresponding to the segmentation image 102.
• the control unit 82C extracts multiple local images 110 showing different lesions 42 from a frame 40 in which a second rectangular frame 108 and an identifier 104 are associated with the lesions 42.
• the local images 110 are local images within the frame 40.
• an image surrounded by the second rectangular frame 108 within the frame 40 is shown as an example of a local image 110.
• each of the multiple local images 110 is an image in which each of the multiple lesions 42 shown in the frame 40 is individually extracted (in other words, an image individually cut out) from the frame 40 using the second rectangular frame 108, which is a frame having the same shape and size.
• each of the multiple local images 110 is an image in which each of the multiple lesions 42 is individually extracted from the frame 40 using the second rectangular frame 108, which is a frame having the same shape and size, so that when the multiple local images 110 are compared, the differences in the size of the lesions 42 between the multiple local images 110 can be visually distinguished.
• the local image 110 is an example of an "extracted image” according to the technology disclosed herein.
• the second rectangular frame 108 is an example of a "frame” according to the technology disclosed herein.
• the control unit 82C generates first information 111 for each lesion 42 shown in the frame 40 and stores it in the RAM 84.
• the first information 111 is information in which the identifier 104, the local image 110, and the second position information 109 are associated.
• the control unit 82C generates the first information 111 by assigning the identifier 104 and the second position information 109 to the local image 110 extracted from the frame 40.
• the identifier 104 assigned to the local image 110 is the identifier 104 assigned to the second rectangular frame 108 used to extract the local image 110 from the frame 40.
• the second position information 109 is information (e.g., coordinates) that can identify the position of the local image 110 within the frame 40.
• multiple local images 110 are extracted from the frame 40 using the second rectangular frame 108
• multiple images may be extracted from the frame 40 using a frame of a different shape and size than the second rectangular frame 108.
• a frame of a common shape and size is used among the multiple lesions 42, as with the second rectangular frame 108.
• a frame is used that allows visual differentiation of the difference in size of the lesions 42 among the multiple extracted images when comparing the multiple extracted images extracted from the frame 40 (multiple local images 110 in the example shown in FIG. 8).
• the acquisition unit 82B acquires a frame 40 from the camera 52 and acquires a size 112 of the lesion 42 captured in the frame 40 acquired from the camera 52 (here, as an example, the frame 40 used in the recognition process 96).
• the acquisition of the size 112 of the lesion 42 captured in the frame 40 is realized by the acquisition unit 82B measuring the size 112.
• the acquisition unit 82B measures the size 112 based on the frame 40.
• the acquisition unit 82B measures the size 112 of the lesion 42 in time series based on each of the multiple frames 40 included in the endoscopic video image 39 acquired from the camera 52.
• the size 112 of the lesion 42 refers to the size of the lesion 42 in real space.
• the size of the lesion 42 in real space is also referred to as the "real size".
• the size 112 is an example of the "characteristics”, "size”, and "real size” related to the technology disclosed herein.
• the acquisition unit 82B acquires distance information 114 of the lesion 42 based on the frame 40 acquired from the camera 52.
• the distance information 114 is information indicating the distance from the camera 52 (i.e., the observation position) to the intestinal wall 32 including the lesion 42 (see FIG. 1).
• the distance from the camera 52 to the intestinal wall 32 including the lesion 42 is illustrated as an example, but this is merely one example, and instead of the distance, a numerical value indicating the depth from the camera 52 to the intestinal wall 32 including the lesion 42 (e.g., a plurality of numerical values that define the depth in stages (e.g., numerical values ranging from several stages to several tens of stages)) may be used.
• acquisition unit 82B acquires distance information 114 here is that even if the lesion 42 is the same size, the size of the lesion 42 on the image becomes smaller the farther the position of the lesion 42 is from the camera 52, and therefore, when calculating the actual size, it is necessary to take into account how far the position of the lesion 42 is from the camera 52.
• Distance information 114 is obtained for each of all pixels constituting frame 40. Note that distance information 114 may also be obtained for each block of frame 40 that is larger than a pixel (for example, a pixel group made up of several pixels to several hundred pixels).
• the acquisition unit 82B acquires the distance information 114, for example, by deriving the distance information 114 using an AI method.
• a distance derivation model 94 is used to derive the distance information 114.
• the distance derivation model 94 is optimized by performing machine learning on the neural network using the second training data.
• the second training data is a data set including multiple data (i.e., multiple frames of data) in which the second example data and the second answer data are associated with each other.
• the second example data is an image corresponding to frame 40.
• the second correct answer data is correct answer data (i.e., annotation) for the second example data.
• an annotation that specifies the distance corresponding to each pixel in the image used as the second example data is used as an example of the second correct answer data.
• the acquisition unit 82B acquires the frame 40 from the camera 52, and inputs the acquired frame 40 to the distance derivation model 94.
• the distance derivation model 94 outputs distance information 114 in pixel units of the input frame 40. That is, in the acquisition unit 82B, information indicating the distance from the position of the camera 52 (e.g., the position of an image sensor or objective lens mounted on the camera 52) to the intestinal wall 32 shown in the frame 40 is output from the distance derivation model 94 as distance information 114 in pixel units of the frame 40.
• the acquisition unit 82B generates a distance image 116 based on the distance information 114 output from the distance derivation model 94.
• the distance image 116 is an image in which the distance information 114 is distributed in pixel units contained in the endoscopic moving image 39 (i.e., frame 40).
• the acquisition unit 82B acquires the first position information 98 assigned to the segmentation image 102 in the probability map 100 obtained by the recognition unit 82A.
• the acquisition unit 82B refers to the first position information 98 and extracts distance information 114 from the segmentation corresponding region 116A in the distance image 116.
• the segmentation corresponding region 116A is a region corresponding to a position identified from the first position information 98 in the distance image 116.
• the distance information 114 extracted from the segmentation corresponding region 116A may be, for example, distance information 114 corresponding to the position (e.g., center of gravity) of the lesion 42, or a statistical value (e.g., median, average, or mode) of the distance information 114 for multiple pixels (e.g., all pixels) included in the lesion 42.
• the acquisition unit 82B extracts a number of pixels 118 from the frame 40.
• the number of pixels 118 is the number of pixels on a line segment 120 that crosses an image area (i.e., an image area showing the lesion 42) at a position identified from the first position information 98 among all image areas of the frame 40 input to the distance derivation model 94.
• An example of the line segment 120 is the longest line segment parallel to a long side of a rectangular frame 122 that circumscribes the image area showing the lesion 42. Note that the line segment 120 is merely an example, and instead of the line segment 120, the longest line segment parallel to a short side of the rectangular frame 122 that circumscribes the image area showing the lesion 42 may be applied.
• the acquisition unit 82B calculates the size 112 of the lesion 42 based on the distance information 114 extracted from the segmentation corresponding area 116A in the distance image 116 and the number of pixels 118 extracted from the frame 40.
• the calculation of the size 112 uses an arithmetic expression 124.
• the arithmetic expression 124 is an arithmetic expression in which the distance information 114 and the number of pixels 118 are independent variables and the size 112 is a dependent variable.
• the acquisition unit 82B inputs the distance information 114 extracted from the distance image 116 and the number of pixels 118 extracted from the frame 40 to the arithmetic expression 124.
• the arithmetic expression 124 outputs the size 112 corresponding to the input distance information 114 and number of pixels 118.
• the acquisition unit 82B generates second information 126.
• the second information 126 is generated by associating the identifier 104 with the size 112 output from the arithmetic expression 124.
• the identifier 104 associated with the size 112 is the identifier 104 associated with the segmentation image 102 that corresponds to the segmentation corresponding area 116A used to calculate the size 112.
• the length of the lesion 42 in real space is exemplified as the size 112, but the technology of the present disclosure is not limited to this, and the size 112 may be the surface area or volume of the lesion 42 in real space.
• an arithmetic formula 124 is used in which the number of pixels in the entire image area showing the lesion 42 and the distance information 114 are independent variables, and the surface area or volume of the lesion 42 in real space is a dependent variable.
• the acquisition unit 82B acquires sizes 112 for other lesions 42 captured in the frame 40 in a manner similar to the example shown in FIG. 9, and generates second information 126 by assigning to the acquired size 112 an identifier 104 associated with the segmentation image 102 corresponding to the segmentation corresponding area 116A used to calculate the size 112.
• the acquisition unit 82B then stores in the RAM 84 each piece of second information 126 generated for each of the multiple lesions 42 captured in the frame 40.
• the control unit 82C acquires the size 112 from the acquisition unit 82B.
• the control unit 82C also acquires the frame 40 used to measure the size 112 by the acquisition unit 82B from the camera 52.
• the control unit 82C displays the frame 40 acquired from the camera 52 in the first display area 36.
• the control unit 82C also acquires from the RAM 84 a plurality of pieces of first information 111 and a plurality of pieces of second information 126 corresponding to a plurality of lesions 42 shown in the frame 40.
• the control unit 82C displays a plurality of identifiers 104 within the frame 40 displayed in the first display area 36 based on the plurality of pieces of first information 111 and the plurality of pieces of second information 126 acquired from the RAM 84.
• the control unit 82C also displays a plurality of identifiers 104, a plurality of local images 110, and a plurality of sizes 112 in the second display area 38 as part of the medical information 44 based on the plurality of pieces of first information 111 and the plurality of pieces of second information 126 acquired from the RAM 84.
• the control unit 82C displays the latest identifier 104, the latest local image 110, and the latest size 112 on the screen 35 each time the acquisition unit 82B acquires the size 112. That is, the identifier 104, the local image 110, and the size 112 displayed on the screen 35 are updated to the latest identifier 104, the latest local image 110, and the latest size 112 each time the acquisition unit 82B acquires the size 112.
• a plurality of identifiers 104 are superimposed on the frame 40 in the first display area 36.
• the plurality of identifiers 104 may be superimposed on the frame 40 using an alpha blending method.
• the position at which each identifier 104 is displayed within the frame 40 is adjacent to the corresponding lesion 42 (hereinafter also referred to as the "lesion-adjacent position").
• the control unit 82C selects one of the multiple pieces of first information 111, and determines the lesion-adjacent position by referring to the second position information 109 included in the selected first information, which is the selected first information 111.
• the control unit 82C displays the identifier 104 included in the selected first information at the lesion-adjacent position.
• the "lesion-adjacent position" is an example of a "second display position" according to the technology of the present disclosure.
• the control unit 82C displays the multiple local images 110 contained in the multiple pieces of first information 111 acquired from the RAM 84 according to the multiple sizes 112 contained in the multiple pieces of second information 126 acquired from the RAM 84.
• the position at which the local image 110 is displayed in the second display area 38 is an example of a "first display position" according to the technology of the present disclosure.
• the multiple local images 110 included in the multiple pieces of first information 111 acquired from the RAM 84 are displayed in a comparatively arranged state.
• the multiple local images 110 are displayed along the vertical direction (in other words, the up-down direction when viewed from the front).
• the multiple local images 110 are displayed in a display manner that allows the relative size 112 to be visually identified.
• the multiple local images 110 are arranged from top to bottom in descending order of size 112.
• the multiple sizes 112 included in the multiple pieces of first information 111 acquired from the RAM 84 are displayed in a state in which their correspondence with the multiple local images 110 can be identified.
• each of the multiple sizes 112 is displayed in a position adjacent to the corresponding local image 110.
• the correspondence between the local image 110 included in the first information 111 and the size 112 included in the second information 126 is determined by matching the identifier 104 included in the first information 111 with the identifier 104 included in the second information 126.
• an identifier 104 corresponding to each local image 110 is displayed in a position adjacent to each local image 110. This allows the doctor 12 to visually identify where in the frame 40 the lesion 42 depicted in the local image 110 is located by comparing the identifier 104 displayed in the lesion-adjacent position in the first display area 36 with the identifier 104 displayed in the second display area 38.
• the identifier 104 is displayed to the left of the corresponding local image 110 when viewed from the front, and the size 112 is displayed to the right of the corresponding local image 110 when viewed from the front, but this is merely an example, and the identifier 104, local image 110, and size 112 may be displayed in the second display area 38 in a layout that allows the correspondence between them to be identified.
• the multiple local images 110 may be displayed in the second display area 38 in a layout that allows the size relationship between the multiple lesions 42 in terms of size 112 to be visually identified.
• a form example is given in which the correspondence between the lesion 42 shown in the frame 40 and the local image 110 is visually identified by the identifier 104, but this is merely one example.
• a first rectangular frame 106 (see FIG. 7) may be displayed within the frame 40, and a second rectangular frame 108 of the local image 110 displayed in the second display area 38 may be displayed in the same display mode (e.g., color and/or brightness, etc.) as the corresponding first rectangular frame 106.
• the lesion 42 in the frame 40 and the corresponding local image 110 may be displayed in a linked state via a line, etc.
• the size 112 is displayed in the second display area 38, but this is merely one example, and the size 112 may be displayed within the frame 40 displayed in the first display area 36. In this case, for example, the size 112 may be displayed superimposed on the frame 40 using an alpha blending method.
• FIG. 11 The flow of the medical support process shown in Figure 11 is an example of a "medical support method" related to the technology of the present disclosure.
• step ST10 the recognition unit 82A and the control unit 82C acquire a frame 40 obtained by imaging the large intestine 28 with the camera 52.
• the control unit 82C displays the frame 40 in the first display area 36 (see FIGS. 5 and 11).
• the following description will be given on the assumption that multiple lesions 42 are shown in the frame 40.
• step ST12 the recognition unit 82A performs a recognition process 96 on the frame 40 acquired in step ST10 to recognize the lesion 42 shown in the frame 40 (see FIG. 5). After the process of step ST12 is executed, the medical support process proceeds to step ST14.
• step ST14 the recognition unit 82A obtains the probability map 100 from the recognition model 92 (see FIG. 6). After the processing of step ST14 is executed, the medical support processing proceeds to step ST16.
• step ST16 the recognition unit 82A assigns first position information 98 to each of the multiple segmentation images 102 in the probability map 100 acquired in step ST14 (see FIG. 6). After the processing of step ST16 is executed, the medical support processing proceeds to step ST18.
• step ST18 the recognition unit 82A assigns an identifier 104 to each of the first position information 98 assigned to each of the multiple segmentation images 102 in the probability map 100, thereby associating the identifier 104 with each of the multiple segmentation images 102 (see FIG. 6).
• step ST20 the medical support processing proceeds to step ST20.
• step ST20 the control unit 82C sets a second rectangular frame 108 for each of the multiple image regions showing the multiple lesions 42 in the frame 40 based on the first position information 98 assigned to each of the multiple segmentation images 102 in the probability map 100 in step ST16 (see FIG. 7).
• step ST20 the medical support processing proceeds to step ST22.
• step ST22 the control unit 82C extracts multiple local images 110 from the frame 40 using the multiple second rectangular frames 108 set for the frame 40 in step ST20 (see FIG. 8). After the processing of step ST22 is executed, the medical support processing proceeds to step ST24.
• step ST24 the control unit 82C generates a plurality of pieces of first information 111 by assigning an identifier 104 and second position information 109 to each of the plurality of local images 110 extracted from the frame 40, and stores the first information 111 in the RAM 84 (see FIG. 8).
• step ST24 the medical support processing proceeds to step ST26 shown in FIG. 12B.
• step ST26 the acquisition unit 82B acquires the size 112 of each of the multiple lesions 42 shown in the frame 40 based on the frame 40 acquired in step ST10 and the probability map 100 in which the first position information 98 is associated with the segmentation image 102 (see FIG. 9).
• step ST28 the medical support processing proceeds to step ST28.
• step ST28 the acquisition unit 82B generates second information 126 by associating the size 112 acquired in step ST26 for each of the multiple lesions 42 shown in the frame 40 with the identifier 104 associated with the segmentation image 102 used to acquire the size 112, and stores the second information 126 in the RAM 84 (see FIG. 10).
• step ST30 the medical support processing proceeds to step ST30.
• step ST30 the control unit 82C displays the identifier 104 at a position adjacent to the lesion in the frame 40 based on the first information 111 stored in the RAM 84 (see FIG. 11).
• step ST32 the medical support processing proceeds to step ST32.
• step ST32 the control unit 82C displays the local image 110 and size 112 for each identifier 104 in the second display area 38 based on the first information 111 and second information 126 stored in the RAM 84 (see FIG. 11).
• step ST34 the medical support processing proceeds to step ST34.
• step ST34 the control unit 82C determines whether or not a condition for terminating the medical support process has been satisfied.
• a condition for terminating the medical support process is a condition in which an instruction to terminate the medical support process has been given to the endoscope system 10 (for example, a condition in which an instruction to terminate the medical support process has been accepted by the acceptance device 64).
• step ST34 If the conditions for terminating the medical support process are not met in step ST34, the determination is negative, and the medical support process proceeds to step ST10 shown in FIG. 12A. If the conditions for terminating the medical support process are met in step ST34, the determination is positive, and the medical support process ends.
• the recognition process 96 is performed on the frame 40 showing the multiple lesions 42, and the size 112 is acquired as a characteristic of each of the multiple lesions 42 recognized.
• the frame 40 is then displayed in the first display area 36, and multiple local images 110 in which the multiple lesions 42 are individually extracted from the frame 40 are displayed in the second display area 38 according to the size 112 of each of the multiple lesions 42. Therefore, the doctor 12 can visually recognize the multiple lesions 42 shown in the frame 40 in a state in which the size 112 of each of the multiple lesions 42 can be grasped without impairing the visibility of the frame 40 displayed in the first display area 36.
• the multiple local images 110 are displayed in the second display area 38 according to the respective sizes 112 (i.e., actual sizes) of the multiple lesions 42, this is merely one example.
• the difference in size 112 between the multiple lesions 42 can also be identified from the difference in size (i.e., size on the image) of the first rectangular frame 106 set for each of the multiple lesions 42, so the multiple local images 110 may be displayed in the second display area 38 according to the size of the first rectangular frame 106 set for each of the multiple lesions 42.
• the multiple local images 110 may be displayed in the second display area 38 according to the sizes of the multiple segmentation images 102 corresponding to the multiple lesions 42.
• the multiple local images 110 are displayed in the second display area 38 in a display format that allows the doctor 12 to visually identify the size relationship of the sizes 112 between the multiple lesions 42.
• the multiple local images 110 and their corresponding sizes 112 are displayed in the second display area 38 in a state in which the correspondence between the multiple local images 110 can be identified. This allows the doctor 12 to visually recognize the differences in the sizes 112 of the multiple lesions 42.
• the doctor 12 can visually recognize the difference in size 112 of each of the multiple lesions 42.
• a frame having a common shape and size among the multiple lesions 42 is used as the second rectangular frame 108. Therefore, by displaying the multiple local images 110 extracted from the frame 40 using the second rectangular frame 108 in a comparable state in the second display area 38, the doctor 12 can visually recognize the differences in the sizes 112 of the multiple lesions 42.
• an identifier 104 is displayed in the second display area 38 at a position adjacent to the local image 110, and an identifier 104 is also displayed in the first display area 36 at a position adjacent to the lesion. Therefore, the doctor 12 can visually recognize the correspondence between the display position of the local image 110 in the second display area 38 and the display position of the lesion 42 shown in the frame 40 in the first display area 36.
• the acquisition unit 82B acquires the sizes 112 of the lesions 42, and the second information 126 is generated by associating the identifiers 104 with the sizes 112.
• the technology disclosed herein is not limited to this.
• the acquisition unit 82B may generate a plurality of third information 128 based on a plurality of distance information 114 acquired for the lesions 42 and store the third information 128 in the RAM 84.
• the third information 128 differs from the second information 126 in that the distance information 114 is used instead of the size 112.
• the distance information 114 used in the third information 128 is the distance information 114 extracted from the segmentation corresponding region 116A when calculating the size 112.
• the distance information 114 is an example of the "depth" related to the technology disclosed herein.
• the control unit 82C acquires the plurality of pieces of first information 111 and the plurality of pieces of third information 128 from the RAM 84, and displays a plurality of identifiers 104, a plurality of local images 110, and a plurality of pieces of distance information 114 in the second display area 38 based on the plurality of pieces of first information 111 and the plurality of pieces of third information 128 acquired from the RAM 84.
• the display content of the second display area 38 shown in FIG. 14 differs from the display content of the second display area 38 shown in FIG.
• the second display area 38 displays a plurality of local images 110 in a display mode that allows visual identification of the depth relationship between a plurality of lesions 42.
• the corresponding distance information 114 i.e., the depth from the camera 52 to the lesion 42 captured in the local image 110
• the doctor 12 can visually recognize the multiple lesions 42 shown in the frame 40 in a state in which the distance information 114 for each of the multiple lesions 42 is graspable, without impairing the visibility of the frame 40 displayed in the first display area 36.
• the doctor 12 since the distance information 114 corresponding to each of the multiple local images 110 is displayed in a position adjacent to the corresponding local image 110, the doctor 12 can visually recognize the relationship in depth from the observation position between the multiple lesions 42.
• FIG. 11 shows an example of a form in which multiple local images 110 are displayed in the second display area 38 according to the size 112
• the example shown in Fig. 14 shows an example of a form in which multiple local images 110 are displayed in the second display area 38 according to the distance information 114, but the display contents of the second display area 38 shown in Fig. 11 and the display contents of the second display area 38 shown in Fig. 14 may be selectively displayed.
• the display contents of the second display area 38 shown in Fig. 11 and the display contents of the second display area 38 shown in Fig. 14 may be switched according to an instruction 129 (e.g., an instruction given by the doctor 12) received by the reception device 64.
• an instruction 129 e.g., an instruction given by the doctor 12
• the number of lesions 42 captured in the frame 40 increases, the number of local images 110 to be displayed in the second display area 38 also increases, and if there are too many lesions 42, it becomes difficult to display all of the local images 110 in the second display area 38. All of the local images 110 may be displayed in the second display area 38 by reducing the size of the local images 110, but this reduces the visibility of the local images 110.
• the control unit 82C displays a plurality of local images 110 in a grouped state in the second display area 38.
• the control unit 82C determines whether the number of local images 110 stored in the RAM 84 (i.e., the number corresponding to the number of lesions 42 recognized by the recognition unit 82A) exceeds a preset number (e.g., four).
• a preset number e.g., four.
• the preset number is a number derived in advance by testing an actual device and/or computer simulation, etc., as the number at which visibility becomes poor when all local images 110 stored in the RAM 84 are displayed in the second display area 38.
• control unit 82C executes the same process as in the above embodiment. If the number of local images 110 stored in the RAM 84 exceeds the preset number, the control unit 82C groups the local images 110 by size range by dividing the local images 110 stored in the RAM 84 into multiple size ranges.
• the multiple size ranges include a first size range 130 and a second size range 132.
• the first size range 130 is a range in which the size 112 is 4.0 mm or more
• the second size range 132 is a range in which the size 112 is less than 4.0 mm.
• the size 112 in the range of 4.0 mm or more and the size 112 in the range of less than 4.0 mm are each an example of a "common characteristic" related to the technology of the present disclosure.
• the multiple size ranges may be determined based on a reference value that is used by the physician 12 for clinical decision-making based on medical knowledge (e.g., a reference value (e.g., 5 mm and/or 10 mm, etc.) that the physician 12 uses to determine whether or not to remove the lesion 42), or may be determined based on a variable value that is changed depending on instructions given by the physician 12 and/or various conditions.
• a reference value e.g., 5 mm and/or 10 mm, etc.
• the control unit 82C groups the local images 110 by size range by dividing the local images 110 into a first size range 130 and a second size range 132 based on the first information 111 and the second information 126 stored in the RAM 84.
• the control unit 82C then displays the local images 110 in the second display area 38 in a state where they are grouped by size range.
• a local image 110 representative of the first size range 130 and a local image 110 representative of the second size range 132 are displayed in the second display area 38.
• the size 112 of the lesion 42 depicted in the local image 110 representative of the first size range 130 and the size 112 of the lesion 42 depicted in the local image 110 representative of the second size range 132 are displayed in the second display area 38.
• first size range 130 and the second size range 132 are examples of the "multiple first ranges" according to the technology disclosed herein.
• the second display area 38 displays first number information 134 indicating the number of local images 110 allocated to the first size range 130 and second number information 136 indicating the number of local images 110 allocated to the second size range 132.
• the first number information 134 displays information indicating that the number of local images 110 allocated to the first size range 130 is two
• the second number information 136 displays information indicating that the number of local images 110 allocated to the second size range 132 is three.
• a specific number is illustrated, but this is merely an example and may be a general indicator that can specify whether the number is large or small.
• the first number information 134 and the second number information 136 are examples of "information regarding the number of extracted images grouped into the first range" related to the technology disclosed herein.
• An example of a local image 110 representative of the first size range 130 is a local image 110 that depicts the lesion 42 with the largest size 112 out of all the local images 110 assigned to the first size range 130.
• An example of a local image 110 representative of the second size range 132 is a local image 110 that depicts the lesion 42 with the largest size 112 out of all the local images 110 assigned to the second size range 132.
• a local image 110 depicting a lesion 42 with the largest size 112 has been exemplified as a local image 110 representative of the size range, but this is merely one example.
• a local image 110 representative of the size range may be a local image 110 depicting a lesion 42 with the smallest size 112, a local image 110 depicting a lesion 42 with a median size 112, a local image 110 depicting a lesion 42 with a mode size 112, or a local image 110 selected randomly.
• an identifier 104 corresponding to a local image 110 representative of the first size range 130 is displayed in the second display area 38
• an identifier 104 corresponding to a local image 110 representative of the second size range 132 is displayed in the second display area 38.
• the multiple sizes 112 for the multiple lesions 42 captured in the multiple local images 110 are divided into a first size range 130 and a second size range 132, and the multiple local images 110 divided into the first size range 130 and the second size range 132 are displayed in a grouped state in the second display area 38. Therefore, better visibility in the second display area 38 can be achieved compared to a case in which all of the local images 110 are displayed separately in the second display area 38.
• a local image 110 representative of all the local images 110 sorted into the first size range 130 is displayed in the second display area 38, and first number information 134 is displayed in the second display area 38 as information regarding the number of local images 110 grouped into the first size range 130.
• a local image 110 representative of all the local images 110 sorted into the second size range 132 is displayed in the second display area 38, and second number information 136 is displayed in the second display area 38 as information regarding the number of local images 110 grouped into the second size range 132. This allows the doctor 12 to roughly grasp the difference between the first size range 130 and the second size range 132.
• the local images 110 stored in the RAM 84 exceeds a preset number, the local images 110 are displayed in the second display area 38 in a grouped state according to size range. Therefore, according to this configuration, a number of local images 110 that does not impair visibility can be displayed in the second display area 38, and a number of local images 110 that impair visibility can be grouped and displayed in the second display area 38. As a result, visual discomfort experienced by the doctor 12 observing the second display area 38 can be suppressed.
• multiple local images 110 are grouped by size range, but the technology of the present disclosure is not limited to this.
• multiple local images 110 may be grouped by distance range.
• the distance information 114 is displayed in the second display area 38, but in the example shown in FIG. 17, instead of the distance information 114, an indicator 141 indicating the depth to the lesion 42 is displayed in the second display area 38.
• the indicator 141 is expressed as "deep” or "shallow”. Note that, in the example shown in FIG.
• control unit 82C divides the local images 110 stored in RAM 84 into multiple distance ranges, thereby grouping the local images 110 by distance range.
• the multiple distance ranges include a first distance range 138 and a second distance range 140.
• first distance range 138 is a range in which the distance indicated by distance information 114 is 4.0 mm or more
• second distance range 140 is a range in which the distance indicated by distance information 114 is less than 4.0 mm.
• each of the distances in the range of 4.0 mm or more and the distances in the range of less than 4.0 mm is an example of a "common characteristic" related to the technology disclosed herein.
• two distance ranges are exemplified here, but this is merely one example, and for example, three or more distance ranges may be used, as long as they are multiple distance ranges that separate multiple pieces of distance information 114.
• the multiple distance ranges may be fixed values, or may be variable values that are changed according to instructions given by the doctor 12 and/or various conditions.
• the control unit 82C groups the local images 110 by distance range by dividing the local images 110 into a first distance range 138 and a second distance range 140 based on the first information 111 and the third information 128 (see FIG. 14) stored in the RAM 84.
• the control unit 82C then displays the local images 110 in the second display area 38 in a state where the local images 110 are grouped by distance range.
• a local image 110 representing the first distance range 138 and a local image 110 representing the second distance range 140 are displayed in the second display area 38.
• the first distance range 138 and the second distance range 140 are examples of "multiple second ranges" according to the technology disclosed herein.
• the second display area 38 displays an index 141 indicating the depth of the lesion 42 captured in the local image 110 assigned to the first distance range 138, and an index 141 indicating the depth of the lesion 42 captured in the local image 110 assigned to the second distance range 140.
• the second display area 38 displays third number information 142 indicating the number of local images 110 assigned to the first distance range 138, and fourth number information 144 indicating the number of local images 110 assigned to the second distance range 140.
• the third number information 142 displays information indicating that the number of local images 110 assigned to the first distance range 138 is two
• the fourth number information 144 displays information indicating that the number of local images 110 assigned to the second distance range 140 is three.
• the third number information 142 and the fourth number information 144 are examples of "information regarding the number of extracted images grouped in the second range" according to the technology disclosed herein.
• An example of a local image 110 representative of the first distance range 138 is a local image 110 in which the lesion 42 at the greatest distance indicated by the distance information 114 is shown among all the local images 110 assigned to the first distance range 138.
• An example of a local image 110 representative of the second distance range 140 is a local image 110 in which the lesion 42 at the greatest distance indicated by the distance information 114 is shown among all the local images 110 assigned to the second distance range 140.
• a local image 110 showing a lesion 42 at the maximum distance has been exemplified as a local image 110 representative of the distance range, but this is merely one example.
• the local image 110 representative of the distance range may be a local image 110 showing a lesion 42 at the minimum distance, a local image 110 showing a lesion 42 at the median distance, a local image 110 showing a lesion 42 at the most frequent distance, or a local image 110 selected randomly.
• an identifier 104 corresponding to a local image 110 representing the first distance range 138 is displayed in the second display area 38
• an identifier 104 corresponding to a local image 110 representing the second distance range 140 is displayed in the second display area 38.
• the multiple distance information 114 for the multiple lesions 42 captured in the multiple local images 110 are divided into a first distance range 138 and a second distance range 140, and the multiple local images 110 divided into the first distance range 138 and the second distance range 140 are displayed in a grouped state in the second display area 38. Therefore, better visibility in the second display area 38 can be achieved compared to a case in which all of the local images 110 are displayed separately in the second display area 38.
• a local image 110 representative of all the local images 110 assigned to the first distance range 138 is displayed in the second display area 38, and third number information 142 is displayed in the second display area 38 as information regarding the number of local images 110 grouped in the first distance range 138.
• a local image 110 representative of all the local images 110 assigned to the second distance range 140 is displayed in the second display area 38, and fourth number information 144 is displayed in the second display area 38 as information regarding the number of local images 110 grouped in the second distance range 140. This allows the doctor 12 to roughly grasp the difference between the first distance range 138 and the second distance range 140.
• the control unit 82C may generate a map 146 that allows the display position within the frame 40 of the lesion 42 included in the local image 110 to be identified, and display the generated map 146 in the third display area 148.
• the map 146 is generated by associating an identifier 104 with each of the multiple segmentation images 102 included in the probability map 100.
• the identifier 104 associated with each of the multiple segmentation images 102 is the same identifier as the identifier 104 shown in FIG. 7.
• the third display area 148 in the screen 35 is a display area different from the first display area 36 and the second display area 38, and is arranged in a position in the screen 35 that can be compared with the first display area 36 and the second display area 38.
• the third display area 148 displays the probability map 100, and multiple identifiers 104 are displayed in the probability map 100. For example, the multiple identifiers 104 are displayed superimposed on the probability map 100.
• the position at which the identifiers 104 are displayed in the probability map 100 is adjacent to the segmentation image 102, and is determined based on the first position information 98 (see FIG. 7).
• the map 146 is an example of the "position identification information" and “map” according to the technology of the present disclosure.
• the third display area 148 is an example of the "third display area” according to the technology of the present disclosure.
• a map 146 is displayed in the third display area 148. Since multiple segmentation images 102 are distributed in the map 146 at positions where multiple lesions 42 are present, the doctor 12 can visually identify the display position within the frame 40 of the lesion 42 included in the local image 110 by referring to the map 146.
• the map 146 is generated based on the probability map 100 obtained by performing the recognition process 96, it is possible to easily obtain a map that can identify the display position within the frame 40 of the lesion 42 contained in the local image 110.
• the map 146 is displayed in the third display area 148, which is different from the first display area 36 and the second display area 38, the visibility of the first display area 36 and the second display area 38 can be maintained better than when the map 146 is displayed in the first display area 36 or when the map 146 is displayed in the second display area 38.
• third display area 148 may also display probability map 100 itself, or a map that has been modified from probability map 100, etc.
• map 146 includes multiple segmentation images 102
• an image in which the frame 40 is thumbnailed, or an image using an outer frame that is homothetic with the outer frame of the frame 40 may be used.
• first display area 36, the second display area 38, and the third display area 148 within the screen 35 may be changed in response to given instructions and/or various conditions.
• one or two of the information displayed in the first display area 36, the information displayed in the second display area 38, and the information displayed in the third display area 148 may be displayed on one or more display devices other than the display device 18.
• the local image 110 may be sorted for each size 112 and for each distance information 114 and displayed in the second display area 38.
• the local image 110 may be sorted for each type of lesion 42 and displayed in the second display area 38, or the local image 110 may be sorted for each type of lesion 42 and displayed in the second display area 38.
• the local image 110 may be sorted for each type of lesion 42 and/or each type of lesion 42 and for each size 112 and displayed in the second display area 38.
• the local image 110 may be sorted for each type of lesion 42 and/or each type of lesion 42 and for each distance information 114 and displayed in the second display area 38.
• the local images 110 may be sorted according to the characteristics (e.g., the above-mentioned size 112, distance information 114, type of lesion 42, and/or model of lesion 42) selected according to instructions received by the reception device 64 (e.g., instructions given by the doctor 12) and displayed in the second display area 38.
• the characteristics e.g., the above-mentioned size 112, distance information 114, type of lesion 42, and/or model of lesion 42
• size 112, distance information 114, type of lesion 42, and form of lesion 42 are exemplified here as characteristics of lesion 42, these are merely examples.
• the characteristics of lesion 42 may be the severity of lesion 42 and/or the state of the mucosa of lesion 42, or may be a combination of the above-mentioned characteristics (e.g., a combination of two or more of size 112, distance information 114, type of lesion 42, form of lesion 42, severity of lesion 42, and state of the mucosa of lesion 42, etc.).
• an example was given in which the identifier 104 and size 112 are displayed on the screen 35, but the technology of the present disclosure is not limited to this, and the technology of the present disclosure can be applied even if the identifier 104 and/or size 112 are not displayed on the screen 35.
• an example shown in FIG. 14 an example was given in which the identifier 104 and distance information 114 are displayed on the screen 35, but the technology of the present disclosure can be applied even if the identifier 104 and/or distance information 114 are not displayed on the screen 35.
• an example was given in which an image cut out from the frame 40 using the second rectangular frame 108 is used as the local image 110, but the technology of the present disclosure is not limited to this.
• an image obtained by performing image processing (e.g., commonly known image processing) on an image cut out from the frame 40 using the second rectangular frame 108 may be used as the local image 110.
• an image cut out from the frame 40 using the largest first rectangular frame 106 out of all the first rectangular frames 106 (see FIG. 7) set for the probability map 100 may be used as the local image 110.
• object recognition processing using AI with a segmentation method is exemplified as the recognition processing 96, but the technology disclosed herein is not limited to this, and the recognition processing 96 may be object recognition processing using AI with a bounding box method.
• a bounding box is used instead of the segmentation image 102 (see FIG. 5), and the bounding box is used as a frame equivalent to the first rectangular frame 106 (see FIG. 7).
• the endoscopic video 39 is displayed in the first display area 36, but the result of performing the recognition process 96 on the endoscopic video 39 may be superimposed on the endoscopic video 39 in the first display area 36. Also, at least a portion of the segmentation image 102 obtained as a result of performing the recognition process 96 on the endoscopic video 39 may be superimposed on the endoscopic video 39.
• One example of superimposing at least a portion of the segmentation image 102 on the endoscopic video 39 is an example in which the outer contour of the segmentation image 102 is superimposed on the endoscopic video 39 using an alpha blending method.
• a bounding box may be superimposed on the endoscopic video 39 in the first display area 36.
• at least a portion of the segmentation image 102 and/or the bounding box may be superimposed on the first display area 36 as information that makes it possible to visually identify which lesion 42 corresponds to the measured size 112.
• the control unit 82C generates the distance image 116 (see FIG. 9) from the frame 40 using the distance derivation model 94 (see FIG. 9), but the technology of the present disclosure is not limited to this.
• the depth of the large intestine 28 in the depth direction may be measured by a depth sensor (e.g., a sensor that measures distance using a laser distance measurement method and/or a phase difference method, etc.) provided at the tip portion 50 (see FIG. 2), and the processor 82 may generate the distance image 116 based on the measured depth.
• the length in real space of the longest range that crosses the lesion 42 along the line segment 120 is measured as the size 112, but the technology disclosed herein is not limited to this.
• the length in real space of the range that corresponds to the longest line segment that is parallel to the short side of the rectangular frame 122 for the image area showing the lesion 42 may be measured as the size 112 and displayed on the screen 35.
• the doctor 12 can be made to understand the length in real space of the longest range that crosses the lesion 42 along the longest line segment that is parallel to the short side of the rectangular frame 122 for the image area showing the lesion 42.
• the actual size of the lesion 42 in terms of the radius and/or diameter of the circumscribing circle for the image area showing the lesion 42 may be measured and displayed on the screen 35.
• the doctor 12 can be made to understand the actual size of the lesion 42 in terms of the radius and/or diameter of the circumscribing circle for the image area showing the lesion 42.
• the size 112 is displayed within the second display area 38, but this is merely one example, and the size 112 may be displayed in a pop-up format from within the second display area 38 to outside the second display area 38, or the size 112 may be displayed outside the second display area 38 on the screen 35.
• the type of lesion and/or the form of lesion may also be displayed within the first display area 36 and/or the second display area 38, or may be displayed on a screen other than the screen 35.
• the size 112 was measured in units of one frame, but this is merely one example, and the size 112 may be measured in units of multiple frames. Furthermore, a representative size (e.g., average, median, maximum, minimum, deviation, standard deviation, and/or mode, etc.) obtained by measuring the size 112 in units of multiple frames may be used for the second information 126 (see Figures 10 and 11).
• a representative size e.g., average, median, maximum, minimum, deviation, standard deviation, and/or mode, etc.
• an AI-based object recognition process is exemplified as the recognition process 96, but the technology disclosed herein is not limited to this, and the lesion 42 shown in the frame 40 may be recognized by the recognition unit 82A by executing a non-AI-based object recognition process (e.g., template matching, etc.).
• a non-AI-based object recognition process e.g., template matching, etc.
• the arithmetic formula 124 was used to calculate the size 112
• the technology disclosed herein is not limited to this, and the size 112 may be measured by performing AI processing on the frame 40.
• a trained model may be used that outputs the size 112 of the lesion 42 when a frame 40 including a lesion 42 is input.
• deep learning may be performed on the neural network using teacher data that has annotations indicating the size of the lesion as correct answer data for the lesions shown in the images used as example data.
• deriving distance information 114 using distance derivation model 94 has been described, but the technology of the present disclosure is not limited to this.
• other methods of deriving distance information 114 using an AI method include a method that combines segmentation and depth estimation (for example, regression learning that provides distance information 114 for the entire image (for example, all pixels that make up the image), or unsupervised learning that learns the distance for the entire image in an unsupervised manner).
• an endoscopic video image 39 is exemplified, but the technology of the present disclosure is not limited to this, and the technology of the present disclosure can also be applied to medical video images other than endoscopic video images 39 (e.g., video images obtained by a modality other than the endoscopic system 10 (e.g., a radiological diagnostic device or an ultrasonic diagnostic device), such as a radiological video image or an ultrasonic video image).
• a modality other than the endoscopic system 10 e.g., a radiological diagnostic device or an ultrasonic diagnostic device
• distance information 114 extracted from the segmentation corresponding region 116A in the distance image 116 is input to the calculation formula 124, but the technology of the present disclosure is not limited to this.
• distance information 114 corresponding to the position identified from the first position information 98 may be extracted from all distance information 114 output from the distance derivation model 94, and the extracted distance information 114 may be input to the calculation formula 124.
• the display device 18 is exemplified as an output destination of the size 112, etc., but the technology of the present disclosure is not limited to this, and the output destination of various information such as the frame 40, medical information 44, and/or map 146 (hereinafter referred to as "various information") may be other than the display device 18.
• an output destination of information that can be output as audio among the various information is an audio playback device 150.
• Information that can be output as audio among the various information may be output as audio by the audio playback device 150.
• an output destination of the various information is a printer 152 and/or an electronic medical record management device 154, etc.
• the various information may be printed as text, etc. on a medium (e.g., paper) by the printer 152, or may be stored in an electronic medical record 156 managed by the electronic medical record management device 154.
• various information is displayed on the screen 35 or is not displayed on the screen 35.
• Displaying various information on the screen 35 means that the information is displayed in a manner that is perceptible to the user (e.g., doctor 12).
• the concept of not displaying various information on the screen 35 also includes the concept of lowering the display level of the information (e.g., the level perceived by the display).
• the concept of not displaying various information on the screen 35 also includes the concept of displaying the information in a manner that is not visually perceptible to the user.
• examples of the display manner include reducing the font size of the information, displaying the information in thin lines, displaying the information in dotted lines, blinking the information, displaying the information for a display time that is not perceptible, and making the information transparent to an imperceptible level.
• the various outputs such as the audio output, printing, and saving described above.
• the medical support processing is performed by the processor 82 included in the endoscope system 10, but the technology disclosed herein is not limited to this, and a device that performs at least a portion of the processing included in the medical support processing may be provided outside the endoscope system 10.
• an external device 160 may be used that is communicatively connected to the endoscope system 10 via a network 158 (e.g., a WAN and/or a LAN, etc.).
• a network 158 e.g., a WAN and/or a LAN, etc.
• An example of the external device 160 is at least one server that directly or indirectly transmits and receives data to and from the endoscope system 10 via the network 158.
• the external device 160 receives a processing execution instruction provided from the processor 82 of the endoscope system 10 via the network 158.
• the external device 160 then executes processing according to the received processing execution instruction and transmits the processing results to the endoscope system 10 via the network 158.
• the processor 82 receives the processing results transmitted from the external device 160 via the network 158 and executes processing using the received processing results.
• the processing execution instruction may be, for example, an instruction to have the external device 160 execute at least a part of the medical support processing.
• a first example of at least a part of the medical support processing i.e., a processing to be executed by the external device 160
• the external device 160 executes the recognition processing 96 in accordance with the processing execution instruction provided from the processor 82 of the endoscope system 10 via the network 158, and transmits the recognition processing result (e.g., the first position information 98 and/or the probability map 100, etc.) to the endoscope system 10 via the network 158.
• the processor 82 receives the recognition processing result and executes the same processing as in the above embodiment using the received recognition processing result.
• a second example of at least a portion of the medical support process is the process by the acquisition unit 82B.
• the process by the acquisition unit 82B refers to, for example, the process of measuring the size 112 of the lesion 42.
• the external device 160 executes the process by the acquisition unit 82B in accordance with a process execution instruction given from the processor 82 of the endoscope system 10 via the network 158, and transmits the measurement process result (e.g., the size 112, etc.) to the endoscope system 10 via the network 158.
• the processor 82 receives the measurement process result, and executes the same process as in the above embodiment using the received measurement process result.
• a third example of at least a portion of the medical support process is at least one of the processes of steps ST12 to ST28 included in the medical support process shown in Figures 12A and 12B.
• a fourth example of at least a portion of the medical support process is the process of generating the third information 128 and storing it in the storage area.
• a fifth example of at least a portion of the medical support process is a process of grouping multiple local images 110 by size range.
• a sixth example of at least a part of the medical support process is a process of grouping multiple local images 110 by distance range.
• a seventh example of at least a portion of the medical support process is a process for generating the display content of the first display area 36, the display content of the second display area 38, and/or the display content of the third display area 148.
• the external device 160 is realized by cloud computing.
• cloud computing is merely one example, and the external device 160 may be realized by network computing such as fog computing, edge computing, or grid computing.
• network computing such as fog computing, edge computing, or grid computing.
• at least one personal computer or the like may be used as the external device 160.
• the external device 160 may be a computing device with a communication function equipped with multiple types of AI functions.
• the medical support program 90 is stored in the NVM 86 , but the technology of the present disclosure is not limited to this.
• the medical support program 90 may be stored in a portable, computer-readable, non-transitory storage medium such as an SSD or USB memory.
• the medical support program 90 stored in the non-transitory storage medium is installed in the computer 78 of the endoscope system 10.
• the processor 82 executes the medical support process in accordance with the medical support program 90.
• the medical support program 90 may be stored in a storage device such as another computer or server connected to the endoscope system 10 via a network, and the medical support program 90 may be downloaded and installed in the computer 78 upon request from the endoscope system 10.
• processors listed below can be used as hardware resources for executing medical support processing.
• An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing medical support processing by executing software, i.e., a program.
• Another example of a processor is a dedicated electrical circuit, which is a processor with a circuit configuration designed specifically for executing specific processing, such as an FPGA, PLD, or ASIC. All of these processors have built-in or connected memory, and all of these processors execute medical support processing by using the memory.
• the hardware resource that executes the medical support processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the medical support processing may be a single processor.
• a configuration using a single processor first, there is a configuration in which one processor is configured using a combination of one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Secondly, there is a configuration in which a processor is used that realizes the functions of the entire system, including multiple hardware resources that execute medical support processing, on a single IC chip, as typified by SoCs. In this way, medical support processing is realized using one or more of the various processors listed above as hardware resources.
• the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.
• the above medical support process is merely one example. It goes without saying that unnecessary steps can be deleted, new steps can be added, and the processing order can be changed without departing from the spirit of the invention.
• a and/or B is synonymous with “at least one of A and B.”
• a and/or B means that it may be just A, or just B, or a combination of A and B.
• the same concept as “A and/or B” is also applied when three or more things are expressed by linking them with “and/or.”

Landscapes

• Engineering & Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Life Sciences & Earth Sciences (AREA)
• Physics & Mathematics (AREA)
• Surgery (AREA)
• Theoretical Computer Science (AREA)
• Medical Informatics (AREA)
• General Physics & Mathematics (AREA)
• General Health & Medical Sciences (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Radiology & Medical Imaging (AREA)
• Animal Behavior & Ethology (AREA)
• Public Health (AREA)
• Optics & Photonics (AREA)
• Pathology (AREA)
• Veterinary Medicine (AREA)
• Biomedical Technology (AREA)
• Heart & Thoracic Surgery (AREA)
• Molecular Biology (AREA)
• Biophysics (AREA)
• Signal Processing (AREA)
• Artificial Intelligence (AREA)
• Evolutionary Computation (AREA)
• Multimedia (AREA)
• Geometry (AREA)
• Computing Systems (AREA)
• Databases & Information Systems (AREA)
• Software Systems (AREA)
• Quality & Reliability (AREA)
• Endoscopes (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Related Child Applications (1)

Publications (1)

Family

ID=92905191

Family Applications (1)

Country Status (4)

Families Citing this family (1)

Citations (7)

• 2024
  • 2024-02-26 CN CN202480022427.3A patent/CN120957648A/zh active Pending
  • 2024-02-26 WO PCT/JP2024/006789 patent/WO2024202789A1/ja not_active Ceased
  • 2024-02-26 JP JP2025509987A patent/JPWO2024202789A1/ja active Pending
• 2025
  • 2025-09-10 US US19/324,171 patent/US20260007302A1/en active Pending

Patent Citations (7)

Also Published As

Similar Documents

Legal Events