WO2023058503A1 - 内視鏡システム、医療画像処理装置及びその作動方法 - Google Patents

内視鏡システム、医療画像処理装置及びその作動方法 Download PDF

Info

Publication number
WO2023058503A1
WO2023058503A1 PCT/JP2022/035958 JP2022035958W WO2023058503A1 WO 2023058503 A1 WO2023058503 A1 WO 2023058503A1 JP 2022035958 W JP2022035958 W JP 2022035958W WO 2023058503 A1 WO2023058503 A1 WO 2023058503A1
Authority
WO
WIPO (PCT)
Prior art keywords
evaluation result
image
evaluation
remission
illumination light
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/035958
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
広樹 渡辺
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujifilm Corp
Original Assignee
Fujifilm Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujifilm Corp filed Critical Fujifilm Corp
Priority to JP2023552817A priority Critical patent/JPWO2023058503A1/ja
Publication of WO2023058503A1 publication Critical patent/WO2023058503A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A61B1/045—Control thereof
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/31—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for the rectum, e.g. proctoscopes, sigmoidoscopes, colonoscopes

Definitions

  • the present invention relates to an endoscope system that outputs multiple diagnostic results from medical images, a medical image processing apparatus, and an operating method thereof.
  • CAD computer-aided diagnosis
  • GOS Geboes histopathology score
  • An object of the present invention is to provide an endoscope system, a medical image processing apparatus, and an operation method thereof that can obtain diagnostic results from endoscopic images according to the purpose of the user.
  • a medical image processing apparatus of the present invention is a medical image processing apparatus comprising a processor, and the processor obtains an endoscopic image captured by illuminating a subject with one or more types of illumination light, and obtains an endoscopic image.
  • the processor uses the mirror image, output at least one of a first evaluation result based on a first evaluation criterion, which is an evaluation criterion for the degree of disease, and a second evaluation result based on a second evaluation criterion different from the first evaluation criterion. configured as follows.
  • the first evaluation criteria are preferably evaluation criteria for the degree of disease based on images captured by an imaging device different from the endoscope.
  • the image captured by an imaging device different from the endoscope is preferably a pathological image of biopsy tissue taken from the subject.
  • the processor is configured to output a first evaluation result by a first determiner using the endoscopic image, the first determiner determining a pre-stored degree of disease based on the endoscopic image and the pathological image. It is preferable to output the first evaluation result using the surface layer morphology classification that is the correspondence relationship between .
  • the processor generates a first frequency-domain image obtained by extracting a first frequency region from the endoscopic image, a second frequency-domain image obtained by extracting a second frequency region different from the first frequency region from the endoscopic image, and a first generating a frequency domain, a second frequency domain, and a third frequency domain image obtained by extracting a third frequency domain different from each other; using the first frequency domain image, the second frequency domain image and the third frequency domain image,
  • the determiner is arranged to output a first evaluation result based on a first evaluation criterion.
  • the processor is configured to output a first evaluation result by a second determiner different from the first determiner using the endoscopic image, the second determiner using the endoscopic image for learning and the pathological image. is preferably learned using the degree of disease based on
  • the second evaluation criteria are preferably evaluation criteria for the degree of disease based on endoscopic images.
  • the processor is configured to use the endoscopic image to output the second evaluation result by a third determiner different from the first determiner and the second determiner, the third determiner being a learning endoscope It is preferable that learning is performed using images and degrees of disease based on endoscopic images.
  • the first evaluation criterion is the evaluation criterion of the degree of disease based on images taken by an imaging device different from the endoscope that evaluates the degree of disease in two stages
  • the second evaluation criterion is the degree of disease in two stages.
  • the evaluation criteria for the degree of disease is based on endoscopic images evaluated in .
  • the first evaluation criteria are evaluation criteria for the degree of disease based on images taken with an imaging device different from the endoscope that evaluates the degree of disease in three or more stages
  • the second evaluation criterion is the degree of disease. It is preferable that the evaluation criteria for the degree of disease are based on endoscopic images, in which the degree is evaluated in three or more stages.
  • the disease is preferably ulcerative colitis.
  • the first evaluation criterion is Geboes histopathology score, Riley score, Modified Riley score or Roberts histopathology index
  • the second evaluation criterion is Mayo endoscopic subscore or UCEIS score.
  • the first evaluation criterion may include the morphology of superficial blood vessels and/or the degree of bleeding
  • the second evaluation criterion may include the degree of mucosal redness, the degree of vascular transparency of deep blood vessels, and/or the degree of erosion or ulceration. preferable.
  • the first criterion and the second criterion have a range of disease in remission and a range of non-remission, wherein the range of disease in remission on the first criterion is the range of disease in remission on the second criterion.
  • the first non-remission boundary value which is smaller than the range and is the lowest value for non-remission of the disease among the first evaluation criteria, is the second non-remission boundary value, which is the lowest value for non-remission of the disease among the second evaluation criteria.
  • the first non-remission threshold is preferably set to a range in which the disease is in remission according to the second criterion.
  • the processor outputs at least one of a first evaluation result and a second evaluation result for each frame of the endoscopic image, and a first evaluation result calculated using the first evaluation result related to at least two frames of the endoscopic image. It is preferable to output and display at least one of a one-multiple-frame evaluation result and a second multiple-frame evaluation result calculated using a second evaluation result relating to at least two frames of endoscopic images. .
  • the processor is preferably configured to divide the endoscopic image into a plurality of regions and output at least one of the first evaluation result and the second evaluation result for the plurality of regions.
  • the processor outputs a region evaluation index for the plurality of regions based on the endoscopic image segmented as a plurality of regions, and uses the endoscopic image composed of the plurality of regions to which the region evaluation index is attached to determine the plurality of regions. is preferably configured to output at least one of the first evaluation result and the second evaluation result for the region.
  • the processor outputs at least one of a first evaluation result and a second evaluation result for each region of the endoscopic image, and a first multi-region evaluation calculated using the first evaluation results for at least two or more regions. It is preferably configured to output at least one of a result and a second multi-region evaluation result calculated using the second evaluation results for at least two or more regions.
  • the processor is preferably configured to output a first multi-region frame evaluation result and a second multi-region frame evaluation result for each frame of endoscopic image.
  • the processor acquires a first illumination light image captured by illuminating the subject with the first illumination light as an endoscopic image, generates a structure-enhanced image using the first illumination light image, and generates the structure-enhanced image. is preferably configured to output a first evaluation result based on the first evaluation criteria.
  • the structure-enhanced image is preferably an image in which the surface structure, color difference, or edge amount of the first illumination light image is emphasized.
  • the processor is preferably configured to output a second evaluation result based on a second evaluation criterion using the first illumination light image.
  • the first illumination light is preferably white light.
  • the processor includes a first illumination light image captured by illuminating the subject with the first illumination light and a second illumination light image captured by illuminating the subject with second illumination light having a spectrum different from that of the first illumination light. It is preferably configured to acquire a plurality of endoscopic images and use the second illumination light image to output a first evaluation result based on a first evaluation criterion.
  • the processor is preferably configured to automatically switch between a first illumination period in which the subject is illuminated with the first illumination light and a second illumination period in which the subject is illuminated with at least one type of second illumination light.
  • the wavelength band of the second illumination light is preferably narrower than that of the first illumination light.
  • the second illumination light preferably contains more specific blue or specific purple than the first illumination light.
  • the processor is preferably configured to divide the second illumination light image into a plurality of second illumination areas and output the first evaluation results for the plurality of second illumination areas.
  • the processor is preferably configured to output the first evaluation result and the second evaluation result.
  • the processor is preferably configured to notify when a combination of the first evaluation result and the second evaluation result satisfies a specific condition.
  • the specific condition for notification is preferably when the first evaluation result is non-remission, or when the second evaluation result is remission and the first evaluation result is non-remission.
  • the processor preferably notifies by displaying a warning on the display on which the endoscopic image is displayed.
  • the processor preferably causes the display to display the endoscopic image, the first evaluation result, and the second evaluation result.
  • the processor is preferably configured to associate and store the part information of the subject that has output the first evaluation result and the second evaluation result, the first evaluation result, and the second evaluation result.
  • An endoscope system of the present invention includes the above-described medical image processing device, light source device, and endoscope.
  • a method of operating a medical image processing apparatus includes the steps of acquiring a medical image captured by illuminating a subject with one or more types of illumination light, outputting at least one of a first evaluation result based on a certain first evaluation criterion and a second evaluation result based on a second evaluation criterion different from the first evaluation criterion.
  • an endoscope system a medical image processing apparatus, and an operating method thereof that can obtain diagnostic results from endoscopic images according to the purpose of the user.
  • FIG. 1 is an explanatory diagram of a configuration of an endoscope system;
  • FIG. 1 is a block diagram showing functions of an endoscope system;
  • FIG. It is a graph which shows the spectrum of 1st illumination light.
  • It is a graph which shows the spectrum of 2nd illumination light.
  • 3 is a block diagram showing functions of an image determination unit;
  • FIG. FIG. 2 is an explanatory diagram showing the vascular morphology, intramucosal hemorrhage, and extramucosal hemorrhage in forms 2 to 6.
  • FIG. FIG. 2 is an explanatory diagram showing tissues of the digestive tract;
  • 3 is a block diagram showing functions of a frequency domain image generator and a first determiner as an image processing unit;
  • FIG. 4 is an explanatory diagram showing a flow of analysis of a frequency domain image from a second illumination light image by a first determiner as an image processing unit;
  • FIG. 10 is an explanatory diagram showing a flow of outputting a first evaluation result by a first determiner as an image processing unit;
  • 3 is a block diagram showing functions of a structure-enhanced image generating section and a first determiner as an image processing unit;
  • FIG. FIG. 4 is an explanatory diagram showing a method of generating first learning data;
  • FIG. 10 is an explanatory diagram showing a case where a frequency domain image is input to a first determiner as an image processing unit and a first evaluation result is output;
  • FIG. 4 is an explanatory diagram showing a case where a structure-enhanced image is input to a first determiner as an image processing unit and a first evaluation result is output;
  • FIG. 11 is an explanatory diagram showing a case where a second illumination light image is input to a first determiner as a learned model and a first evaluation result is output; It is explanatory drawing which shows the case where a 2nd illumination light image is input into a 2nd determination device and a 1st evaluation result is output. It is explanatory drawing which shows the case where a 1st illumination light image is input into a 2nd determination device and a 1st evaluation result is output.
  • FIG. 11 is an explanatory diagram showing a case where a frequency domain image is input to a third determiner and a second evaluation result is output;
  • FIG. 11 is an image diagram showing a display image when displaying an endoscopic image and an evaluation result display field;
  • FIG. 4 is a block diagram showing functions of a comprehensive evaluation result calculation unit;
  • FIG. 10 is an image diagram showing a display image when displaying an endoscopic image, a region currently being observed, and a frame evaluation result;
  • FIG. 10 is an image diagram showing a display image when displaying an endoscopic image, a schematic diagram, and multi-frame part evaluation results;
  • FIG. 10 is an image diagram showing a display image when displaying an endoscopic image, a three-dimensional structure diagram, and multi-frame part evaluation results;
  • FIG. 4 is an image diagram showing a display image when displaying an endoscopic image, a simple schematic diagram, and multi-frame part evaluation results.
  • FIG. 10 is an image diagram showing an example in which an image dividing unit divides an endoscopic image;
  • FIG. 10 is an image diagram showing an example of outputting an evaluation result from an endoscopic image divided into a plurality of regions;
  • FIG. 10 is an image diagram showing a display image when displaying an endoscopic image, a part currently being observed, and multi-region part evaluation results;
  • FIG. 10 is an image diagram showing a display image when displaying an endoscopic image, a schematic diagram, and a multi-region part evaluation result;
  • FIG. 10 is an image diagram showing a display image when displaying an endoscopic image, a three-dimensional structure diagram, and a multi-area part evaluation result;
  • FIG. 10 is an image diagram showing a display image when displaying an endoscopic image, a simple schematic diagram, and a multi-region part evaluation result;
  • FIG. 10 is an image diagram showing a display image when displaying an endoscopic image, a part currently being observed, a multi-region frame evaluation result, and a multi-region part evaluation result;
  • FIG. 10 is an image diagram showing a display image when displaying an endoscopic image, a part currently being observed, a multi-region frame evaluation result, and a multi-region part evaluation result;
  • FIG. 10 is an image
  • FIG. 10 is an image diagram showing a display image when displaying an endoscopic image, a schematic diagram, a multi-region frame evaluation result, and a multi-region part evaluation result;
  • FIG. 10 is an image diagram showing a display image when displaying an endoscopic image, a three-dimensional structure diagram, a multi-region frame evaluation result, and a multi-region part evaluation result;
  • FIG. 4 is an image diagram showing a display image when displaying an endoscopic image, a simple schematic diagram, a multi-region frame evaluation result, and a multi-region part evaluation result;
  • FIG. 10 is an image diagram showing a display image when the entire endoscopic image is surrounded by a warning display frame;
  • FIG. 10 is an image diagram showing a display image in a case where an area obtained by dividing an endoscopic image is surrounded by a warning display frame; 4 is a flow chart showing the flow of functions of the endoscope system in the first embodiment; It is explanatory drawing which shows a 1st light emission pattern. It is explanatory drawing which shows a 2nd light emission pattern.
  • FIG. 10 is an image diagram showing a display image for displaying a first illumination light image, a second illumination light image, a schematic diagram, and an evaluation result;
  • FIG. 10 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, a part currently being observed, a multi-region frame evaluation result, and a multi-region part evaluation result;
  • FIG. 10 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, a part currently being observed, a multi-region frame evaluation result, and a multi-region part evaluation result;
  • FIG. 10 is an
  • FIG. 10 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, a schematic diagram, a multi-region frame evaluation result, and a multi-region part evaluation result;
  • FIG. 10 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, a three-dimensional structure diagram, a multi-region frame evaluation result, and a multi-region part evaluation result;
  • FIG. 10 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, a simple schematic diagram, a multi-region frame evaluation result, and a multi-region site evaluation result;
  • FIG. 10 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, a simple schematic diagram, a multi-region frame evaluation result, and a multi-region site evaluation result;
  • FIG. 4 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, and a site currently being observed;
  • FIG. 4 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, and a schematic diagram;
  • FIG. 4 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, and a three-dimensional structure diagram;
  • FIG. 4 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, and a simplified schematic diagram;
  • FIG. 10 is an image diagram showing a display image when displaying a first illumination light image and a site currently being observed;
  • FIG. 10 is an image diagram showing a display image when displaying a first illumination light image and a site currently being observed;
  • FIG. 4 is an image diagram showing a display image when displaying a first illumination light image and a schematic diagram;
  • FIG. 4 is an image diagram showing a display image when displaying a first illumination light image and a three-dimensional structure diagram;
  • FIG. 10 is an image diagram showing a display image when displaying a first illumination light image and a simplified schematic diagram;
  • FIG. 10 is an image diagram showing a display image when the entire first illumination light image and the second illumination light image are surrounded by a warning display frame;
  • FIG. 10 is an image diagram showing a display image when the corresponding first illumination area and second illumination area are surrounded by a warning display frame;
  • the endoscope system 10 includes an endoscope 12 , a light source device 13 , a medical image processing device (processor device) 14 , a display 15 and a user interface 16 .
  • the endoscope 12 is optically connected to the light source device 13 and electrically connected to the processor device 14 .
  • the endoscope 12 has an insertion section 12a, an operation section 12b, a bending section 12c and a distal end section 12d.
  • the insertion portion 12a is inserted into the body of the subject.
  • the operation portion 12b is provided at the proximal end portion of the insertion portion 12a.
  • the curved portion 12c and the distal end portion 12d are provided on the distal end side of the insertion portion 12a.
  • the bending portion 12c is bent by operating the angle knob 12e of the operation portion 12b.
  • the distal end portion 12d is directed in a desired direction by the bending motion of the bending portion 12c.
  • a forceps channel (not shown) for inserting a treatment tool or the like is provided from the insertion portion 12a to the distal end portion 12d.
  • the treatment instrument is inserted into the forceps channel from the forceps port 12j.
  • An optical system for forming a subject image and an optical system for illuminating the subject with illumination light are provided inside the endoscope 12 .
  • the operation unit 12b is provided with an angle knob 12e, a mode changeover switch 12f, a still image acquisition instruction switch 12h, and a zoom operation unit 12i.
  • the mode changeover switch 12f is used for an observation mode changeover operation.
  • a still image acquisition instruction switch 12h is used to instruct acquisition of a still image of a subject.
  • a zoom operation unit 12 i is used for operating the zoom lens 42 .
  • the light source device 13 generates illumination light.
  • the display 15 displays an endoscopic image and display images.
  • the endoscopic image includes at least one of a first illumination light image and a second illumination light image that use different illumination light for imaging.
  • the display image is an image generated to display an endoscopic image, the disease evaluation result output from the endoscopic image, and information accompanying the evaluation result.
  • the user interface 16 has a keyboard, mouse, microphone, tablet, touch pen, and the like, and receives input operations such as function settings.
  • the processor device 14 controls the light source device 13 and controls image processing, analysis, and display of image signals transmitted from the endoscope 12 .
  • the light source device 13 includes a light source unit 20 , a light source processor 21 that controls the light source unit 20 , and an optical path coupling unit 22 .
  • the light source unit 20 has a plurality of semiconductor light sources, which are turned on or off. When lighting a plurality of semiconductor light sources, the amount of light emitted from each semiconductor light source is controlled to emit illumination light for illuminating the subject.
  • the light source unit 20 includes a V-LED (Violet Light Emitting Diode) 20a, a B-LED (Blue Light Emitting Diode) 20b, a G-LED (Green Light Emitting Diode) 20c, and an R-LED (Red Light Emitting Diode) 20d. It has four color LEDs.
  • the light source unit 20 and/or the light source processor 21 may be built into the endoscope 12 . Also, the light source processor 21 may be incorporated in the processor device 14 .
  • the endoscope system 10 has a first illumination light mode and a second illumination light mode.
  • the first illumination light mode and the second illumination light mode are switched via the central control unit 50 by operating the mode switch 12f.
  • the illumination light includes first illumination light and second illumination light having a spectrum different from that of the first illumination light.
  • the first illumination light is normal light used for screening observation by giving brightness to the entire subject, and is preferably broadband light such as white light.
  • the second illumination light is at least one type of special light used for emphasizing specific structures such as ducts and blood vessels of the mucous membrane of the gastrointestinal tract, which is the subject.
  • the second illumination light preferably has a narrower wavelength band than the first illumination light.
  • the second illumination light contains more specific blue or specific purple than the first illumination light.
  • the second illumination light is preferably light with a central wavelength of 410 nm.
  • the V-LED 20a When emitting the first illumination light, as shown in FIG. 3, the V-LED 20a emits violet light V with a central wavelength of 405 ⁇ 10 nm and a wavelength range of 380-420 nm.
  • the B-LED 20b generates blue light B with a central wavelength of 450 ⁇ 10 nm and a wavelength range of 420-500 nm.
  • the G-LED 20c generates green light G with a wavelength range of 480-600 nm.
  • the R-LED 20d emits red light R with a central wavelength of 620-630 nm and a wavelength range of 600-650 nm.
  • the second illumination light for example, light with a center wavelength of 410 nm as shown in FIG. 4 is generated.
  • the light source processor 21 independently controls the light amounts of the four colors of violet light V, blue light B, green light G, and red light R, and changes the light amounts to emit the first illumination light or the second illumination light.
  • the first illumination light image is displayed on the display 15 in natural colors by illuminating the subject with the first illumination light for each frame and capturing the image.
  • the second illumination light image is displayed on the display 15 by illuminating the subject with the second illumination light for each frame and capturing an image, thereby emphasizing a specific structure.
  • the term “frame” refers to a unit of period including at least the period from the timing of light emission to the completion of readout of the image signal by the imaging sensor 43 .
  • the still image can be obtained by operating the still image acquisition instruction switch 12h.
  • Signals relating to image acquisition instructions are sent to the endoscope 12 , the light source device 13 and the processor device 14 .
  • each of the LEDs 20a to 20d (see FIG. 2) is incident on the light guide 23 via the optical path coupling section 22 composed of mirrors, lenses, and the like.
  • the light guide 23 propagates the light from the optical path coupling portion 22 to the distal end portion 12 d of the endoscope 12 .
  • the distal end portion 12d of the endoscope 12 is provided with an illumination optical system 30a and an imaging optical system 30b.
  • the illumination optical system 30 a has an illumination lens 31 , and the illumination light propagated by the light guide 23 is applied to the subject through the illumination lens 31 .
  • the imaging optical system 30 b has an objective lens 41 and an imaging sensor 43 . Light from a subject irradiated with illumination light enters an imaging sensor 43 via an objective lens 41 and a zoom lens 42 . As a result, an image of the subject is formed on the imaging sensor 43 .
  • the zoom lens 42 is a lens for enlarging a subject, and is moved between the tele end and the wide end by operating the zoom operation section 12i.
  • the imaging sensor 43 is a primary color sensor, and includes B pixels (blue pixels) having blue color filters, G pixels (green pixels) having green color filters, and R pixels (red pixels) having red color filters. and three types of pixels.
  • the imaging sensor 43 is preferably a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).
  • the imaging processor 44 controls the imaging sensor 43 . Specifically, an image signal is output from the imaging sensor 43 by reading the signal of the imaging sensor 43 by the imaging processor 44 . The output image signal is transmitted to the medical image acquisition unit 60 of the processor device 14 .
  • the medical image acquisition unit 60 performs various signal processing such as defect correction processing, offset processing, demosaicing processing, matrix processing, white balance adjustment, gamma conversion processing, and YC conversion processing on the received color image. Next, by performing image processing including 3 ⁇ 3 matrix processing, gradation conversion processing, color conversion processing such as three-dimensional LUT (Look Up Table) processing, color enhancement processing, and structural enhancement processing such as spatial frequency enhancement, A first illumination light image for the first illumination light and a second illumination light image for the second illumination light are acquired.
  • image processing including 3 ⁇ 3 matrix processing, gradation conversion processing, color conversion processing such as three-dimensional LUT (Look Up Table) processing, color enhancement processing, and structural enhancement processing such as spatial frequency enhancement.
  • the processor device 14 includes a central control unit 50, a medical image acquisition unit 60, an image determination unit 70, a comprehensive evaluation result calculation unit 71, a display control unit 130, an image division unit 160, an area evaluation unit 170, and a notification control unit 200 ( See Figure 2).
  • the programs in the program memory are operated by the central control unit 50 composed of the light source and image processing processors, thereby operating the medical image acquisition unit 60, the image determination unit 70, and the comprehensive evaluation result calculation. Functions of the unit 71, the display control unit 130, the image division unit 160, and the region evaluation unit 170 are realized.
  • the first illumination light image or the second illumination light image acquired by the medical image acquisition unit 60 is input to the image determination unit 70 .
  • the image determination unit 70 outputs at least one of a first evaluation result based on a first evaluation criterion, which is an evaluation criterion for the degree of disease, and a second evaluation result based on a second evaluation criterion different from the first evaluation criterion.
  • the evaluation criteria hereinafter used as a term collectively describing the first evaluation criteria and the second evaluation criteria), which will be described in detail later, are scales for determining the degree of disease.
  • An evaluation result (hereinafter used as a term collectively describing the first evaluation result and the second evaluation result), which will be described later in detail, is the result output by the image determination unit 70, and is, for example, “remission”. , “non-remission”, “mucosal hemorrhage” and “severe vascular irregularity”, and may use existing scales such as “Mayo0” and “Geboes2B", " Prestored scales such as morphology 1 and morphology 2 may be used, or values such as '0' and '1' may be used.
  • Judgment means that the image judgment unit 70 analyzes the endoscopic image in order to output the evaluation result.
  • Output means that the image determination section 70 outputs the first evaluation result and/or the second evaluation result.
  • the image determination unit 70 has, as shown in FIG. 5, a first determiner 80, a second determiner 90, and a third determiner 100 that perform determination using endoscopic images.
  • the first determiner 80 is an image processing unit that stores surface morphology classification, which will be described later, or a trained model.
  • the second determiner 90 and the third determiner 100 are different trained models.
  • a trained model is a model that has been trained using learning data.
  • the image determination unit 70 further includes a frequency domain image generation unit 110 and a structure-enhanced image generation unit 120, which will be described later.
  • deep learning for machine learning to generate learning models
  • machine learning includes decision trees, support vector machines, random forests, regression analysis, supervised learning, semi-unsupervised learning, unsupervised learning, reinforcement learning, deep reinforcement learning, learning using neural networks, Includes generative adversarial networks and the like.
  • the first determiner 80 as a trained model is trained using the first learning data.
  • the second determiner 90 is a trained model that has been trained using the second learning data.
  • a first evaluation criterion is used when generating the dataset of the first training data and the dataset of the second training data. Details of the first learning data and the second learning data will be described later.
  • the third determiner 100 is a trained model that has been trained using the third learning data. A second evaluation criterion is used when generating the third learning data. Details of the third learning data will be described later.
  • the image determination unit 70 having the first determiner 80, the second determiner 90, and the third determiner 100 determines at least the first evaluation result and the second evaluation result based on different first evaluation criteria and second evaluation criteria, respectively.
  • a goal is, for example, a therapeutic goal.
  • Clinical remission is a state in which the patient's symptoms such as diarrhea and abdominal pain are mild or absent.
  • clinical remission with the prescription of therapeutic agents has been the goal of treatment for ulcerative colitis.
  • endoscopic examinations of relapsed patients have occasionally revealed endoscopic non-remission (reddening of the mucous membrane, etc., which has not healed) in the gastrointestinal tract.
  • the first evaluation criterion which is the evaluation criterion for the degree of disease, is preferably the evaluation criterion for the degree of disease based on an image captured by an imaging device different from the endoscope. With this configuration, it is possible to obtain an evaluation result that is different from an evaluation result obtained only from an endoscopic image.
  • the image captured by an imaging device different from the endoscope is preferably a pathological image of biopsy tissue collected from the subject.
  • a pathological image is an image captured after a biopsy tissue, such as a pathological tissue specimen that has been subjected to various staining such as hematoxylin and eosin staining, is observed under a microscope and made ready for diagnosis. .
  • steroids At the stage where pathological remission is the goal of treatment, steroids, immunosuppressants, biological agents, etc. are used as therapeutic agents. These therapeutic agents are highly effective during the induction phase of remission to suppress the symptoms of ulcerative colitis. On the other hand, these therapeutic agents have a large systemic effect on the living body, and the risk of side effects increases when the amount used increases. In order to prevent excessive administration of therapeutic drugs with large side effects, it is necessary to switch the therapeutic drug to a therapeutic drug with relatively small side effects (5-aminosalicylic acid preparation) when a sufficient therapeutic effect is obtained. .
  • the above configuration distinguishes between pathological non-remission and pathological remission based on evaluation criteria based on pathological images, and outputs the evaluation results. allows users to know when to switch medications.
  • the evaluation result based on the evaluation criteria based on the pathological image can be output based only on the endoscopic image, there is no need to collect a biopsy, and the burden on the operator and the subject can be greatly reduced.
  • the first evaluation criteria may be criteria for evaluating the severity of the disease in two stages, or may be a multi-stage evaluation of three or more stages that divides the subclasses into remission or non-remission more finely.
  • the disease is ulcerative colitis.
  • the first evaluation criterion is two-stage, the degree of disease has two stages of remission or non-remission, and the first evaluation result is output as "remission” or "non-remission.” In this case, since the output of the first evaluation result is a relatively simple process, the determination can be made quickly without imposing a burden on the processor.
  • the first evaluation criteria are multiple stages of 3 stages or more, more detailed determination can be made than when there are two stages of remission or non-remission.
  • Geboes histopathology score (Table 2) as the first evaluation criterion.
  • GHS Geboes histopathology score
  • Table 2 Geboes histopathology score
  • Geboes 0-2A are in remission (pathological remission)
  • Geboes 2B-5 are non-remission (pathological non-remission).
  • the image determination unit 70 determines that the endoscopic image is "Geboes2B”
  • it may output "Geboes2B” or "non-remission” as the first evaluation result.
  • pathological image-based evaluation criteria for ulcerative colitis such as Riley score, Modified Riley score, or Roberts histopathology index may be used as the first evaluation criteria.
  • evaluation results can be output based on evaluation criteria used clinically.
  • the first evaluation criterion may be the morphology of superficial blood vessels and/or the degree of bleeding.
  • the first evaluation result for example, "surface blood vessel: honeycomb pattern”, “surface blood vessel: mild irregularity”, and “bleeding: mild” can be output.
  • the surface layer morphology classification is shown in Table 3, the pattern of the blood vessel structure seen in the endoscopic image obtained by irradiating the subject with illumination light with a central wavelength of 410 nm, and the degree of disease (GHS) based on the pathological image. This is the correspondence relationship discovered by the inventors.
  • ulcerative colitis as shown in FIGS. 6A to 6E, the vascular structure pattern changes, and there is a correspondence between the vascular structure pattern and the pathological state.
  • ulcerative colitis is in very mild remission and the mucosa is normal and superficial blood vessels cannot be seen, it is classified as "Form 1" in Table 1.
  • the pattern of superficial blood vessels is regular (Fig. 6(A); Table 3, "Morphology 2")
  • the regularity is only slightly disturbed (Fig. 6(B); Table 3 "Form 3”).
  • the ulcerative colitis is in non-remission and the severity is mild, the superficial blood vessels become dense, with a high-density superficial blood vessel region in which the superficial blood vessels are concentrated (Fig. 6 ( C), Table 3 "Form 4").
  • Intramucosal hemorrhage 81 occurs (Fig. 6(D), Table 3 "Form 5").
  • Extramucosal hemorrhage 82 occurs when the ulcerative colitis is non-remission and moderate to severe in severity (FIG. 6(E), Table 3 "Morph 6").
  • the “dense superficial blood vessel region” is a region in which superficial blood vessels are gathered in a meandering manner, and in appearance on the image, many superficial blood vessels surround intestinal crypts (see FIG. 7).
  • “Intramucosal hemorrhage” is bleeding within the mucosa (see Figure 7) and is different from bleeding in the gastrointestinal lumen (lumen).
  • “Extramucosal hemorrhage” means a small amount of blood into the lumen, blood visible in the lumen ahead of the endoscope distal end 12d after washing the lumen or in the lumen side of the mucosal epithelium , or intraluminal blood with oozing on the mucosa.
  • the first determiner 80 transmits the first frequency domain image or the second frequency domain image generated by the frequency domain image generator 110 shown in FIG. 8 (see also FIG. 2) to the first determiner 80. do.
  • the frequency domain image generator 110 includes a mask processor 112, a frequency component spatial distribution calculator 113, and a frequency domain extractor 114.
  • the first determiner 80 includes a blood vessel density calculator 115, a luminance A calculation unit 116 and a first evaluation result output unit 117 are provided.
  • the first evaluation result is output along the flow shown in FIG. 9A.
  • processing proceeds from top to bottom.
  • the mask processing unit 112 performs mask processing on the marginal portion with large distortion in the second illumination light image 111a, which is the endoscopic image, and generates a masked second illumination light image 112a by extracting only the central portion.
  • the frequency component space distribution calculator 113 calculates the frequency component space distribution and performs frequency processing for extracting the frequency domain.
  • the frequency component spatial distribution calculator 113 calculates the frequency component spatial distribution by applying a Laplacian filter to the masked second illumination light image 112a.
  • the frequency domain extraction unit 114 extracts a first frequency domain 114b (Fig. 9A, shaded area of the first frequency domain 114a) from the masked second illumination light image 112a to generate a first frequency domain image 114a. Specifically, a first frequency region (high frequency region) 114b having a first frequency (high frequency) is extracted by Hessian analysis on the frequency component spatial distribution.
  • the blood vessel density calculation unit 115 performs an averaging process on the first frequency domain image 114a to generate a density distribution image 115a in which the density distribution of superficial blood vessels is calculated.
  • FIG. 9A shows a region 115b with a high density distribution of superficial blood vessels, a region 115c with an intermediate density distribution of superficial blood vessels, and a region 115d with a low density distribution of superficial blood vessels.
  • the first evaluation result output unit 117 performs threshold processing using the density distribution threshold, and the density distribution of the superficial blood vessels is divided into the region 117a above the density distribution threshold and the region 117b below the density distribution threshold.
  • a region 117a in which the density distribution of superficial blood vessels is equal to or greater than the density distribution threshold corresponds to the "high-density superficial blood vessel region".
  • the frequency domain extraction unit 114 further extracts a second frequency domain 114d having a second frequency (low frequency) that is lower than the first frequency from the masked second illumination light image 112a (low frequency domain; The shaded portion of the two-frequency-domain image 114c) is extracted to generate the second frequency-domain image 114c.
  • the frequency statistic for example, maximum value, minimum value, or standard deviation
  • the specific pixel is set as a pixel belonging to the second frequency.
  • the brightness calculator 116 calculates the brightness of the second frequency domain image 114c and generates a second frequency brightness distribution image 116a.
  • FIG. 9A shows a region 116b with a low luminance value and a region 116c with a high luminance value.
  • the first evaluation result output unit 117 performs threshold processing using the first luminance threshold, and determines the area 117c whose luminance is equal to or less than the first luminance threshold and the area 117d whose luminance is greater than the first luminance threshold. Divide into A region 117c whose brightness is equal to or lower than the first brightness threshold corresponds to "extramucosal bleeding".
  • the frequency domain extraction unit 114 extracts a third frequency domain 114f (intermediate frequency domain; FIG. 9A, third frequency domain The shaded portion of the image 114e) is extracted to generate a third frequency domain image 114e. Specifically, from the masked second illumination light image 112a, a region excluding the first frequency region 114b and the second frequency region 114d is extracted as a third frequency region 114f, and a third frequency region image 114e is obtained. Generate. Next, the brightness calculation unit 116 calculates the brightness of the third frequency domain image 114e to generate a third frequency brightness distribution image 116d, and divides it into a high brightness value region 116e and a low brightness value region 116f.
  • the first evaluation result output unit 117 performs threshold processing using the second luminance threshold, and divides the area into an area 117e whose luminance is equal to or less than the second luminance threshold and an area 117f whose luminance is greater than the first luminance threshold. .
  • a region 117e whose brightness is equal to or lower than the second threshold for brightness corresponds to "bleeding in the mucous membrane".
  • the number of pixels of "bleeding” is calculated by adding the number of pixels of "bleeding”, and when the number of pixels of "bleeding" is equal to or greater than the threshold for bleeding, the second illumination light image 111a is "morphology 5" or “morphology 6 , and pathological non-remission is output as the first evaluation result.
  • the second illumination light image 111a corresponds to "form 4", and pathological non-remission is output as the first evaluation result.
  • the second illumination light image 111a is “morphology 1”. , “form 2” or “form 3”, and pathological remission is output as the first evaluation result.
  • the first determiner 80 as an image processing unit preferably outputs the first evaluation result using the first illumination light image.
  • the structure-enhanced image generated by the structure-enhanced image generator 120 from the first illumination light image is transmitted to the first determiner 80 .
  • the structure-enhanced image generation unit 120 extracts superficial blood vessels, intramucosal bleeding, extramucosal bleeding, and the like from the first illumination light image, which is a learning endoscopic image, using the pixel value difference of the bilateral filter ( A structure-enhanced image that is emphasized using the difference in frequency characteristics associated with the amplitude value) is generated. Specifically, like the texture image generation method disclosed in Japanese Unexamined Patent Application Publication No. 2012-135345, filter coefficient generation obtained from the absolute value of the difference between the target pixel extracted from the first illumination light image and the surrounding pixel region is performed.
  • a noise amount estimation model obtained by calculating the first filter coefficients based on the parameters for generating the first illumination light and the average value and the standard deviation of the pixel values of a predetermined region in the gray chart of the first illumination light image is used as the second filter coefficient generation parameters.
  • the first determiner 80 as an image processing unit outputs the first evaluation result using the surface layer morphology classification, so that the pathology can be determined from the first illumination light image and the second illumination light image without collecting a biopsy tissue.
  • Pathological assessment results can be obtained using image-based criteria. Further, since the surface layer morphology classification stored in advance in the first determiner 80 has a clear relationship between the endoscopic image and the pathological image, the interpretability of the evaluation result is high.
  • the first determiner 80 as a trained model preferably outputs the first evaluation result when the first illumination light image or the second illumination light image is input.
  • the first determiner 80 is trained with the first training data.
  • the associating unit 118 as the first learning data generating unit assigns surface morphology classification to the first illumination light image 118a or the second illumination light image 118b, which are endoscopic images for learning. It is preferable to generate the first learning data 118c as a data set of the image data obtained.
  • the first illumination light image 118a or the second illumination light image 118b as the learning endoscope image may be an endoscope image acquired in advance by an endoscope other than the endoscope system 10. It may be an endoscopic image acquired by an endoscope.
  • the associating unit 118 combines the second illumination light image from which the first frequency domain image, the second frequency domain image, and the third frequency domain image are generated, or the first illumination light image from which the structure-enhanced image is generated.
  • a surface layer morphology classification can be associated with an image and used as first learning data.
  • the association unit 118 may be provided in the processor device 14 or may be provided in a device other than the endoscope system 10 .
  • the first determiner 80 can be configured by generating the first learning data by adding the disease degree based on the pathological image to the learning endoscopic image using the surface layer morphology classification.
  • the second determiner 90 which is a trained model, preferably outputs the first evaluation result when the first illumination light image or the second illumination light image is input.
  • the second learning data used for learning of the second determiner 90 will be described.
  • the second learning data is preferably a data set of image data obtained by adding the first evaluation result obtained from the pathological image to the learning endoscopic image.
  • a second learning data generator that generates the second learning data may be provided in the processor device 14 or may be provided in a device other than the endoscope system 10 . It is preferable that the second determiner 90 performs learning using the second learning data, determines the endoscopic image, and outputs the first evaluation result.
  • the first evaluation result attached to the learning endoscopic image used for the second learning data is obtained from the pathological image obtained from the biopsy tissue taken at the same position as the learning endoscopic image was captured by the doctor.
  • This is the evaluation result diagnosed by Diagnosis of pathological images may be performed by an apparatus that diagnoses pathological images.
  • the learning endoscope image may be the first illumination light image or the second illumination light image.
  • pathological evaluation can be obtained from the endoscopic image without taking a biopsy, as in the case of using the first determiner 80.
  • the second training data set is generated based on the established pathology, the reliability of the data set is high.
  • the first evaluation results are output by artificial intelligence, multidimensional features including not only findings that humans can understand, such as the morphology of blood vessels, pixel values, and brightness of endoscopic images, but also other findings are included. Since the first evaluation result is output based on the evaluation result, high accuracy can be expected from the evaluation result.
  • the methods by which the image determination unit 70 outputs the first evaluation result based on the first evaluation criteria are the following (1) to (5).
  • the frequency domain image generator 110 generates a first frequency domain image, a second frequency domain image, and a third frequency domain image ("frequency domain image 110a" in FIG. 12) from the second illumination light image 60a. ”), inputs the first frequency domain image, the second frequency domain image, and the third frequency domain image to the first determiner 80 as an image processing unit, and outputs the first evaluation result 80a.
  • the structure-enhanced image generator 120 generates a structure-enhanced image 120a from the first illumination light image 60b, and inputs the structure-enhanced image 120a to the first determiner 80 as an image processing unit. , to output a first evaluation result 80a.
  • the second illumination light image 60a is input to the first determiner 80 as a learned model, and the first determiner 80 outputs a first evaluation result 80a.
  • the second illumination light image 60a is input to the second determiner 90, and the second determiner 90, which is a trained model, outputs the first evaluation result 90a.
  • the first illumination light image 60b is input to the second determiner 90, and the second determiner 90, which is a trained model, outputs the first evaluation result 90a.
  • the first evaluation result can be output based on the first evaluation criteria.
  • the second evaluation criteria are preferably evaluation criteria for the degree of disease based on endoscopic images.
  • the second evaluation criteria may be criteria for evaluating the severity of the disease in two stages, or may be a multi-stage evaluation of three or more stages in which the disease is further subclassed into remission or non-remission.
  • the disease is ulcerative colitis.
  • the second evaluation criterion is two-stage, the severity of the disease has two stages of remission or non-remission, and the second evaluation result is output as "remission" or "non-remission.” In this case, the output of the second evaluation result is a relatively simple process, so the determination can be made quickly without imposing a burden on the processor.
  • the second evaluation criteria are multiple stages of three or more stages, more detailed determination can be made than two stages of remission or non-remission.
  • Mayo endoscopic subscore (MES) (Table 4) is preferably used as the second evaluation criterion.
  • the output of the second evaluation result as the MES will be described as "Mayo0" when the MES is "0", for example.
  • Mayo 0-1 are in remission (endoscopic remission) and Mayo 2-3 are non-remission (endoscopic non-remission).
  • the image determination unit 70 determines that the endoscopic image is "Mayo2”
  • it may output "Mayo2" or "non-remission” as the second evaluation result.
  • evaluation criteria for ulcerative colitis such as UCEIS (Ulcerative colitis endoscopic index of severity) score may be used as the second evaluation criteria.
  • UCEIS User colitis endoscopic index of severity
  • the second evaluation criteria may be the degree of redness of the mucous membrane, the degree of see-through of deep blood vessels, and/or the degree of erosion or ulcer.
  • the second evaluation result for example, “redness: mild”, “vascular fluoroscopic image: normal”, and “hemorrhage: moderate” can be output.
  • the first evaluation criterion and the second evaluation criterion have a range of disease remission and a non-remission range, and the range of remission by the first evaluation criterion is the range of remission by the second evaluation criterion. Less than the range that would put the disease in remission. That is, the criteria for pathological remission are stricter than those for endoscopic remission.
  • the first non-remission boundary value that is the lowest value for non-remission of the disease is different from the second non-remission boundary value that is the lowest value for non-remission of the disease among the second evaluation criteria.
  • the first non-remission boundary value is preferably set to a range in which the disease is in remission according to the second evaluation criteria.
  • the lowest value for non-remission of ulcerative colitis in the first criterion is "Geboes 2B”.
  • the lowest value for non-remission of ulcerative colitis in the second evaluation criterion is "Mayo2”.
  • “Geboes 2B” falls within the range of "Mayo 1", which is generally regarded as endoscopic remission, when applied to the second criterion of MES.
  • the third learning data used for learning of the third determiner 100 which outputs the second evaluation result by inputting the endoscopic image, will be explained.
  • the third learning data is preferably a data set of image data obtained by attaching a second evaluation result obtained from an endoscopic image to a learning endoscopic image.
  • a third learning data generator that generates the third learning data may be provided in the processor device 14 or may be provided in a device other than the endoscope system 10 . It is preferable that the third determiner 100 performs learning using the third learning data, determines the endoscopic image, and outputs a second evaluation result.
  • the second evaluation result attached to the learning endoscopic image used for the third learning data is the evaluation result of the doctor diagnosing the learning endoscopic image.
  • the diagnosis of the learning endoscopic image may be performed by a system other than the third determiner 100 that diagnoses the endoscopic image.
  • the learning endoscope image may be the first illumination light image or the second illumination light image. That is, the third determiner 100 can output the second evaluation result based on the second evaluation criteria from both the first illumination light image and the second illumination light image. Since the second evaluation result is output by artificial intelligence, it is based on multidimensional features including not only findings that humans can understand, such as blood vessel morphology, pixel values and brightness, but also other findings in the endoscopic image. Since the second evaluation result is output, the evaluation result can be expected to be highly accurate.
  • the learning endoscope image is acquired at a position near the position where the first illumination light image was captured. It may be a second illumination light image.
  • the third learning data is preferably a data set of image data obtained by adding the second evaluation result to the first frequency domain image, the second frequency domain image, and the third frequency domain image.
  • the image determination unit 70 inputs an endoscopic image and outputs both the first evaluation result and the second evaluation result. By outputting both the first evaluation result and the second evaluation result, the endoscopic image can be evaluated from various aspects.
  • the methods by which the image determination unit 70 outputs the second evaluation result based on the second evaluation criteria are the following (1) to (3).
  • (1) As shown in FIG. 17, the first illumination light image 60b is input to the third determiner 100, and the third determiner 100 outputs the second evaluation result 101.
  • FIG. (2) As shown in FIG. 18, the second illumination light image 60a is input to the third determiner 100, and the third determiner 100 outputs the second evaluation result 101.
  • FIG. (3) As shown in FIG. 19, the frequency domain image generator 110 converts the second illumination light image 60a into a first frequency domain image, a second frequency domain image, and a third frequency domain image ("frequency domain image 110a" in FIG. 12). ), and inputs the first, second, and third frequency-domain images to the third determiner 100 , and the third determiner 100 outputs the second evaluation result 101 .
  • the first evaluation result and/or the second evaluation result output by the image determination unit 70 are transmitted to the display control unit 130.
  • the display control unit 130 displays a display image 131 that displays an endoscopic image 132 currently being observed and an evaluation result display field 133 that indicates the first evaluation result and/or the second evaluation result. is preferably generated and displayed on the display 15 .
  • the image determination unit 70 outputs at least one of the first evaluation result and the second evaluation result for one frame of endoscopic image. Thereafter, the first evaluation result output for one frame of endoscopic image is the first frame evaluation result, and the second evaluation result output for one frame of endoscopic image is the second frame evaluation result. call.
  • the evaluation result of the first frame or the evaluation result of the second frame is used as a unit score, and the unit score is used as a total score for comprehensive evaluation. It is preferable to output the result.
  • the comprehensive evaluation result is output by the comprehensive evaluation result calculation unit 71 .
  • the first determiner 80, the second determiner 90, or the third determiner 100 of the image determination unit 70 outputs the first frame evaluation result or the second frame evaluation result, which is a unit score, and outputs the result to the comprehensive evaluation result calculation unit 71.
  • Send see Figure 2.
  • the comprehensive evaluation result calculation unit 71 has a multi-frame evaluation result calculation unit 72 shown in FIG. 21 and a multi-region evaluation result calculation unit 73, which will be described later.
  • the multiple-frame evaluation result calculation unit 72 uses the first-frame evaluation result or the second-frame evaluation result as a unit score, and outputs the first multiple-frame evaluation result or the second multiple-frame evaluation result as a total score based on the unit score.
  • the multiple-frame evaluation result calculation unit 72 calculates a first multiple-frame evaluation result calculated using a first evaluation result related to at least two frames of endoscopic images, and a first multiple-frame evaluation result related to at least two frames or more of endoscopic images. It is preferable to output at least one of the second multi-frame evaluation results calculated using the two evaluation results.
  • the first multi-frame evaluation result and/or the second multi-frame evaluation result are sent to the display control unit 130 .
  • the percentage indicating the ratio of pathological remission and the ratio of pathological non-remission Outputs a percentage indicating .
  • one of the pathological states with a high ratio of pathological remission and pathological non-remission may be output, and the pathological state in which the ratio of pathological remission and pathological non-remission is equal to or greater than a certain value may be output. good too.
  • the percentage indicating the ratio of Geboes0, the percentage indicating the ratio of Geboes1, the percentage indicating the ratio of Geboes2A, the percentage indicating the ratio of Geboes2B, the ratio of Geboes3 , the percentage of Geboes4, and the percentage of Geboes5 are output.
  • the GHS with the highest proportion among Geboes0 to Geboes5 may be output, or the GHS with a proportion equal to or greater than a certain value among Geboes0 to Geboes5 may be output.
  • the percentage indicating the percentage of endoscopic remission and the percentage indicating the percentage of endoscopic non-remission When outputting the second multiple-frame evaluation results in two stages of remission or non-remission, output the percentage indicating the percentage of endoscopic remission and the percentage indicating the percentage of endoscopic non-remission.
  • one of the endoscopic states with a high ratio of endoscopic remission and endoscopic non-remission may be output, and the ratio of endoscopic remission and endoscopic non-remission is a constant value.
  • the above endoscopic states may be output.
  • the percentage indicating the ratio of Mayo 0 the percentage indicating the ratio of Mayo 1
  • the percentage indicating the ratio of Mayo 2 When outputting the first multi-frame part evaluation result in multiple stages like MES, the percentage indicating the ratio of Mayo 0, the percentage indicating the ratio of Mayo 1, the percentage indicating the ratio of Mayo 2, and the ratio of Mayo 3 are indicated. Print percentages.
  • the MES with the highest ratio among Mayo0 to Mayo3
  • the multi-frame evaluation result is any section in the body or a plurality of endoscopic images obtained using the first frame evaluation result or the second frame evaluation result of at least two frames of endoscopic images. is preferably output in a temporal period of .
  • the multi-frame part evaluation result calculator 72 has a multi-frame part evaluation result calculator 140 .
  • the multi-frame part evaluation result calculation unit 140 uses the first frame part evaluation result or the second frame part evaluation result as a unit score, and calculates the first multi-frame part evaluation result or the second multi-frame part evaluation result as a total score based on the unit score. Output.
  • a multi-frame part evaluation result calculation unit 140 calculates a first multi-frame part evaluation result calculated using a first frame evaluation result related to at least two frames of endoscopic images, and at least two frames of endoscopic images. It is preferable to output at least one of the second multi-frame part evaluation results calculated using the second frame evaluation results.
  • the first multi-frame part evaluation result and/or the second multi-frame part evaluation result are transmitted to the display control unit 130 and displayed on the display image 131 as shown in FIGS. 22 to 25.
  • the first multi-frame site evaluation result and/or the second multi-frame site evaluation result may be obtained from an anatomical site of the gastrointestinal tract (esophagus, stomach, duodenum, jejunum, ileum, cecum, ascending colon, transverse colon, descending colon, sigmoid colon). It is preferable to output for each colon, rectum, etc.). It may be a majority decision on the number of remission and non-remission frames in the first frame evaluation results in the anatomical region, or a majority decision on the remission and non-remission frames in the second frame evaluation results.
  • the multi-frame site evaluation results are calculated based on the ratio of the number of frames for "remission” to A%, "non-remission” to B%, Based on the ratio of the number of frames of remission and non-remission in the second frame evaluation results, C% for "remission” and D% for "non-remission” may be output.
  • the multi-frame site evaluation results are based on the ratio of each GHS to the first frame evaluation results obtained for each anatomical site, such as "Geboes0" is E%, “Geboes1" is F%, and so on.
  • the percentage may be obtained and output in the multi-level evaluation.
  • the multi-frame site evaluation results may be statistics such as averages, maximum values, and minimum values. It is preferable that the multi-frame site evaluation result is output in an arbitrary section such as dividing the descending colon into three sections, such as proximal, middle, and distal sections. Moreover, it is preferable that the multi-frame part evaluation result is output at an arbitrary timing. For example, the user inputs the timing at which imaging of the descending colon is started and the timing at which imaging is finished, and at the end timing, the evaluation results obtained from the start to the end of imaging are used to output the multi-frame site evaluation results. good too.
  • the first multi-frame part evaluation result and the second multi-frame part evaluation result are output, as shown in FIG. 151 is displayed.
  • the first multiple-frame site evaluation result of "remission” is A%
  • the result of evaluation of one multiple-frame site is B%
  • the result of second multiple-frame site evaluation of "remission” is indicated by C%
  • the result of second multiple-frame site evaluation of "non-remission” is indicated by D%. Evaluation results and corresponding bar charts are displayed.
  • a schematic diagram 152 may be shown, and the part currently being observed may be highlighted with a frame 153 .
  • the contents of the multi-frame part evaluation result display column 151 are the same as the specific example shown in FIG.
  • a three-dimensional structural diagram 154 of the large intestine is shown, and the first multi-frame site evaluation result or the second multi-frame site evaluation result reflects a "remission" site 156 and a "non-remission” site 157, respectively.
  • the contents of the multi-frame part evaluation result display column 151 are the same as the specific example shown in FIG.
  • the three-dimensional structure drawing 154 may be stored in advance, or may be generated from an endoscopic image using a method such as Structure from Motion.
  • a simplified schematic diagram 158 is shown in which the position of the large intestine is represented by bars.
  • the parts 157 may be reflected and displayed.
  • the contents of the multi-frame part evaluation result display column 151 are the same as the specific example shown in FIG. R indicates rectum, SC sigmoid colon, DC descending colon, TC transverse colon, AC ascending colon, C cecum.
  • the evaluation result can be visually recognized by showing the evaluation result that evaluates a relatively wide range as in the above configuration. can improve sexuality.
  • the medical image acquisition unit 60 transmits the endoscopic image to the image dividing unit 160 (see FIG. 2), and the image dividing unit 160 divides the endoscopic image into a plurality of regions, and divides the endoscopic image into a plurality of regions.
  • the resulting endoscopic image is input to the image determination section 70, and the image determination section 70 outputs at least one of the first evaluation result and the second evaluation result.
  • the first evaluation result output for one area will be referred to as the first area evaluation result
  • the second evaluation result output for one area will be referred to as the second area evaluation result.
  • the image dividing unit 160 divides the endoscopic image into at least two or more regions.
  • the endoscopic image 60c is divided into a plurality of grid-like regions as shown in FIG.
  • an endoscopic image 60c is divided into a plurality of 16 areas from area A to area P.
  • the shape of division is not limited to a lattice shape, and may be any shape that can subdivide an endoscopic image, such as a polygonal shape such as a hexagon, or a curved shape.
  • the plurality of regions may be cut out in random shapes having different shapes and sizes depending on the location.
  • the image determination unit 70 outputs first evaluation results and/or second evaluation results for a plurality of regions, as shown in FIG.
  • the first evaluation result of areas A to F, H, and L to P is output as "remission”
  • the first evaluation result of areas G and I to K is "non-remission.” output.
  • the image determination unit 70 preferably determines the quality of the image itself for each of the plurality of regions before evaluating the lesion, and then outputs at least one of the first evaluation result and the second evaluation result.
  • the medical image acquisition unit 60 transmits the endoscopic image to the image dividing unit 160, and the image dividing unit 160 divides the endoscopic image into a plurality of regions.
  • the endoscopic image divided into a plurality of regions is input to the region evaluation section 170, and the region evaluation section 170 outputs region evaluation indices for the plurality of divided regions.
  • the area evaluation index is an evaluation value for the quality of the segmented endoscopic image itself. That is, the higher the area evaluation index, the higher the quality of the original endoscopic image, and the higher the reliability of the first evaluation result and the second evaluation result.
  • An endoscopic image in which a region evaluation index is assigned to a plurality of divided regions is input to the image determination unit 70, and the image determination unit 70 determines at least one of the first evaluation result and the second evaluation result for each divided region. to output
  • the image dividing unit 160 divides the image into a size that allows determination of the lesion area or the evaluation inhibition target that is not the lesion area.
  • An evaluation obstruction target refers to a structure or artifact that is unsuitable for outputting the first evaluation result or the second evaluation result and causes a decrease in the accuracy of determination.
  • it includes a specific pool 161 such as a pool of water, a pool of blood, or a pool of excess medical fluid covering the observation target as shown in FIG. 28, and bubbles 162 as shown in FIG.
  • a specific pool 161 such as a pool of water, a pool of blood, or a pool of excess medical fluid covering the observation target as shown in FIG. 28, and bubbles 162 as shown in FIG.
  • the evaluation inhibited objects include, as shown in FIG. A cap rim 165a or, as shown in FIG. 32, a cap rim 165a and a cap outer side 165b are also included.
  • the area evaluation unit 170 calculates a high area evaluation index. On the other hand, if the area includes an evaluation inhibition target, the area evaluation unit 170 calculates a low area evaluation index.
  • area evaluation indices are assigned to areas A to P, respectively.
  • area B area evaluation index 0.9
  • area E area evaluation index 0.8
  • area F area evaluation index 0.95
  • area G area evaluation index 0.95
  • area H area evaluation index 0.8
  • area I area evaluation index 0.8
  • area J area evaluation index 0.95
  • area K area evaluation index 0.95
  • area L area evaluation index 0.8
  • area N area evaluation index 0.9
  • area O area evaluation index 0.9
  • regions A, D, M, and P are at the edges of the field of view of the endoscope, and therefore distortion occurs. Therefore, this is an area in which the output accuracy of the first evaluation result or the second evaluation result is low. Therefore, the areas A, C, D, M, and P that contain evaluation inhibition targets are divided into areas B, E, F, G, H, I, and J that do not contain evaluation inhibition targets. , area K, area L, area N, and area O are outputted.
  • the region evaluation unit 170 may generate an extracted image by extracting only regions equal to or larger than the threshold for region evaluation out of the endoscopic image divided into a plurality of regions, and input the extracted image to the image determination unit 70. . With the above configuration, it is possible to preliminarily exclude regions with low quality and output the first evaluation result and/or the second evaluation result only to regions where a certain quality is ensured.
  • an area whose area evaluation index is less than the area evaluation threshold may be highlighted with a frame 166 to call attention to it.
  • the image determination unit 70 outputs at least one of the first evaluation result and the second evaluation result for the divided regions of the endoscopic image of one frame, and further obtains the comprehensive evaluation result for the divided regions. Therefore, it is possible to use the first area evaluation result or the second area evaluation result output for the divided area as a unit score, and output the multi-area evaluation result as a total score based on the unit score. preferable.
  • the multi-region evaluation result is output by the multi-region evaluation result calculation unit 73 of the comprehensive evaluation result calculation unit 71 shown in FIG.
  • the first determiner 80, the second determiner 90, or the third determiner 100 of the image determination unit 70 outputs the first region evaluation result or the second region evaluation result, which is a unit score, and the comprehensive evaluation result calculation unit 71 It is transmitted to the multi-region evaluation result calculation unit 73 (see FIG. 2).
  • a multi-region evaluation result calculation unit 73 calculates a first multi-region evaluation result calculated using a first region evaluation result for at least two or more divided regions, and a second multi-region evaluation result for at least two or more divided regions. It is preferable to output at least one of the second multi-region evaluation results calculated using the evaluation results.
  • the first multi-region evaluation result and/or the second multi-region evaluation result are sent to the display control unit 130 .
  • the percentage indicating the ratio of pathological remission and the ratio of pathological non-remission Outputs a percentage indicating .
  • one of the pathological states with a high ratio of pathological remission and pathological non-remission may be output, and the pathological state in which the ratio of pathological remission and pathological non-remission is equal to or greater than a certain value may be output. good too.
  • the percentage indicating the ratio of Geboes0, the percentage indicating the ratio of Geboes1, the percentage indicating the ratio of Geboes2A, the percentage indicating the ratio of Geboes2B, the ratio of Geboes3 , the percentage of Geboes4, and the percentage of Geboes5 are output.
  • the GHS with the highest proportion among Geboes0 to Geboes5 may be output, or the GHS with a proportion equal to or greater than a certain value among Geboes0 to Geboes5 may be output.
  • the percentage indicating the percentage of endoscopic remission and the percentage indicating the percentage of endoscopic non-remission When outputting the second multiple area evaluation results in two stages of remission or non-remission, output the percentage indicating the percentage of endoscopic remission and the percentage indicating the percentage of endoscopic non-remission.
  • one of the endoscopic states with a high ratio of endoscopic remission and endoscopic non-remission may be output, and the ratio of endoscopic remission and endoscopic non-remission is a constant value.
  • the above endoscopic states may be output.
  • the percentage indicating the ratio of Mayo 0 the percentage indicating the ratio of Mayo 1
  • the percentage indicating the ratio of Mayo 2 When outputting the second multi-frame part evaluation result in multiple stages like MES, the percentage indicating the ratio of Mayo 0, the percentage indicating the ratio of Mayo 1, the percentage indicating the ratio of Mayo 2, and the ratio of Mayo 3 are indicated. Print percentages.
  • the MES with the highest ratio among Mayo0 to Mayo3 may
  • the multi-region evaluation result is an arbitrary temporal section in which a plurality of endoscopic images are acquired using the first region evaluation result or the second region evaluation result related to at least two or more regions. It is preferable to output in a middle region including a period, an arbitrary number of regions within one frame of endoscopic image, and an arbitrary region within one frame of endoscopic image.
  • the multi-region evaluation result calculator 73 has a multi-region frame evaluation result calculator 180 .
  • the multi-region frame evaluation result calculation unit 180 uses the first region evaluation result or the second region evaluation result as a unit score, and calculates the first region evaluation result as a total score for each endoscopic image of one frame based on the unit score. and at least one of a first multi-region frame evaluation result calculated using and a second multi-region frame evaluation result calculated using the second region evaluation result.
  • the first multi-region frame evaluation result and/or the second multi-region frame evaluation result are preferably output for each frame of endoscopic image.
  • a majority decision on the number of remission and non-remission regions in the first region evaluation results in one frame of endoscopic image, or a majority decision on the number of remission and non-remission regions in the second region evaluation results may be used.
  • the multi-region frame evaluation results are based on the ratio of “remission” to A% and “non-remission” to 1%.
  • B % based on the ratio of the number of remission and non-remission regions in the second region evaluation results obtained for each frame of endoscopic image, output as C% for “remission” and D% for “non-remission” may be
  • the multi-region frame evaluation result is based on the ratio of each GHS to the first region evaluation result obtained for each endoscopic image of one frame, and "Geboes0" is E%, and "Geboes1" is F%. Percentages may be obtained and output in multi-level evaluation, as in the following.
  • the multi-region frame evaluation results may be statistics such as average, maximum and minimum values.
  • the first multi-region frame evaluation result and/or the second multi-region frame evaluation result are transmitted to the display control unit 130 and displayed on the display image 131 .
  • an endoscopic image 150 currently being observed and a multi-region frame evaluation result display column 181 are displayed on the display image 131.
  • FIG. Grid lines 182 may be displayed on the currently observed endoscopic image 150 to indicate how the regions are divided.
  • the multi-region frame evaluation result display column 181 displays the first multi-region evaluation result of “remission” of A % and the first multi-region evaluation result of “non-remission” for the endoscopic image 150 currently being observed.
  • the multi-domain evaluation result is B%
  • the second multi-domain evaluation result of "remission” is C%
  • the second multi-domain evaluation result of "non-remission” is D%
  • the multi-domain evaluation result corresponds to the vicinity thereof.
  • a bar graph is displayed.
  • the image determination unit 70 uses the first region evaluation result or the second region evaluation result as a unit score, and based on the unit score, determines the anatomical It is preferable to output a multi-region part evaluation result as a total score for each target part.
  • the multi-part area evaluation result calculation unit 190 shown in FIG. 21 calculates a first multi-part area part evaluation result calculated using the first area evaluation result, and a second It is preferable to output at least one of the multi-region part evaluation results.
  • the first multi-region site evaluation result and/or the second multi-region site evaluation result are preferably output for each anatomical site of the gastrointestinal tract. It may be a majority decision of the number of remission and non-remission regions in the first region evaluation results at the anatomical site, or a majority decision of the remission and non-remission region numbers in the second region evaluation results at the anatomical site.
  • the multi-region evaluation results were calculated as follows: “remission” was A%, “non-remission” was B%, “Remission” may be output as C% and “Non-remission” as D% based on the ratio of the number of regions in remission and non-remission in the second region evaluation results obtained for each medical site.
  • the multiple region site evaluation result is based on the ratio of each GHS to the first region evaluation result obtained for each anatomical site, such as "Geboes0" is E%, “Geboes1" is F%, and so on.
  • the percentage may be obtained and output in the multi-level evaluation.
  • the multi-region site evaluation result may be a statistic such as an average, maximum value, and minimum value.
  • it is preferable that the multi-region part evaluation result is output at an arbitrary timing. For example, the user inputs the timing at which imaging of the descending colon is started and the timing at which imaging is finished, and at the end timing, the evaluation results obtained from the start to the end of imaging are used to output the evaluation results of multiple regions. good too.
  • the first multi-region part evaluation result and/or the second multi-region part evaluation result are transmitted to the display control unit 130 and displayed on the display image 131 .
  • an endoscopic image 150 currently being observed and a multi-region part evaluation result display column 191 are displayed on the display image 131.
  • the site evaluation result is indicated as C%
  • the second multiple-region site evaluation result of "non-remission” is indicated as D%
  • a bar graph corresponding to the multiple-region site evaluation result is displayed in the vicinity thereof.
  • FIG. 37 shows a schematic diagram 152 in which the currently observed part is highlighted with a frame 153 and the multi-region/part evaluation result for that part is displayed in the multi-region/part evaluation result display column 191 .
  • FIG. 38 a three-dimensional structural diagram 154 of the large intestine is displayed, and the first multi-region site evaluation result or the second multi-region site evaluation result reflects the "remission" site 156 and the "non-remission" site 157, respectively. is displayed.
  • the multiple-area-part display column 191 may display the multiple-area-part evaluation result for an arbitrary part or all parts photographed so far.
  • FIG. 39 shows a simplified schematic diagram 158 in which the positions of the large intestine are represented by bars, and the “remission” site 156 and the “non-remission” site 157 of the first multi-region site evaluation result or the second multiple-region site evaluation result are shown. Each is reflected and displayed.
  • the multi-region part evaluation result display column 191 the multi-region part evaluation results for any part or all parts photographed so far may be displayed.
  • the multi-region frame evaluation result and the multi-region part evaluation result may be output and displayed on the display image.
  • FIG. The display image 131 displays an endoscopic image 150 currently being observed, a multi-region frame evaluation result display column 181, and a multi-region site evaluation result display column 191.
  • the multi-region frame evaluation result is indicated by B%
  • the second multi-region frame evaluation result of “remission” is indicated by C%
  • the second multi-region frame evaluation result of “non-remission” is indicated by D%.
  • a corresponding bar graph is displayed.
  • the first multiple-region site evaluation result of “remission” and the first multiple-region site evaluation result of “non-remission” in the currently observed site (descending colon) are displayed.
  • F% the second multiple-region site evaluation result of “remission”
  • the second multiple-region site evaluation result of “non-remission” is H%
  • a bar graph corresponding to the multiple-region site evaluation result is shown in the vicinity. is displayed.
  • FIG. 40 the part (descending colon) currently being observed is displayed, and the multi-region part evaluation results for that part are displayed in the multi-region part evaluation result display column 191 .
  • FIG. 41 shows a schematic diagram 152 in which the currently observed part is highlighted with a frame 153 and the multi-region/part evaluation result for that part is displayed in the multi-region/part evaluation result display column 191 .
  • FIG. 42 a three-dimensional structural diagram 154 of the large intestine is displayed, and the first multi-region site evaluation result or the second multi-region site evaluation result reflects the "remission" site 156 and the "non-remission" site 157, respectively. I am displaying.
  • FIG. 43 shows a simplified schematic diagram 158 in which the positions of the large intestine are represented by bars, and the “remission” site 156 and the “non-remission” site 157 of the first multiple-region site evaluation result or the second multiple-region site evaluation result are shown. Each is reflected and displayed.
  • the multi-region part evaluation result display column 191 the multi-region part evaluation results for any part or all parts photographed so far may be displayed.
  • the notification control unit 200 (see FIG. 2) preferably performs notification in accordance with the first evaluation result and the second evaluation result output by the image determination unit 70.
  • the notification is preferably performed by displaying a warning on the display image 131 when a specific condition to be described later is satisfied.
  • the warning display is preferably performed by providing a warning display frame 201 on the endoscopic image 150 currently being observed.
  • the display form of the warning display is not limited to this, and may be a warning mark or a warning message.
  • the notification is not limited to the warning display, and may be performed by sound or light.
  • the first region comprehensive notification condition means that when the first multiple region frame evaluation result expresses (a) non-remission or Geboes 2B or more and (b) non-remission or Geboes 2B or more in percentage, the percentage is More than the threshold for general notification of the first area, or (c) non-remission or more than Geboes 2B by majority vote.
  • the first area comprehensive notification condition is not limited to this, and may be set arbitrarily.
  • the second region comprehensive notification condition means that when the second multiple region frame evaluation result expresses (a) remission or Mayo 1 or less, and (b) remission or Mayo 1 or less region as a percentage, the percentage is the second It indicates that there are many remissions or Mayo 1 or less by majority votes, or (c) more than the threshold for general area notification.
  • the second area comprehensive notification condition is not limited to this, and may be set arbitrarily.
  • the warning display frame 201 surrounds the entire endoscopic image 150 currently being observed to give the notification.
  • one area (warning display frame 201) is surrounded by a warning display frame 201 to notify.
  • a schema 152 may be displayed on the display image 131 in order to inform the user of the current observation position.
  • the medical image acquisition unit 60 acquires an endoscopic image (S101).
  • the image determination unit 70 outputs at least one of the first evaluation result and the second evaluation result (referred to as "evaluation result” in FIG. 46) (S102).
  • the comprehensive evaluation result calculation unit 71 calculates the first multi-frame part evaluation result and/or the second multi-frame part evaluation result, the first multi-region frame evaluation result and/or the second multi-region frame evaluation result, and/or , the first multi-region part evaluation result and/or the second multi-region part evaluation result (referred to as "comprehensive evaluation result" in FIG. 46) is output (S103).
  • the display control unit 130 generates a display image 131 (S104), and if a specific condition is satisfied (S105), the notification control unit transmits an instruction to the display control unit 130 to perform notification. (S106), the display image 131 is displayed on the display 15 (S107).
  • At least one of the first evaluation result and the second evaluation result can be output from one type of endoscopic image.
  • the type of evaluation result to be output can be selected according to the user's purpose.
  • both the first evaluation result and the second evaluation result can be output from a plurality of types of endoscopic images.
  • the endoscopic images obtained by the examination can be evaluated from various aspects. Therefore, it is possible to prevent overlooking of "hidden non-remission" in one of the evaluation criteria.
  • the first evaluation result and the second evaluation result are unit scores, and the overall evaluation result as a total score obtained by integrating the unit scores.
  • the frame evaluation result is output as the unit score
  • the multi-frame evaluation result which is the overall evaluation result in an arbitrary section or period
  • the total score can be output as the total score.
  • the multiple-frame evaluation result (3) multiple-frame evaluation result, which is a comprehensive evaluation result for each anatomical region, in which a specific interval is set can be output.
  • the region evaluation result is output as a unit score, and (5) a comprehensive evaluation in an arbitrary section, period, or middle region is output as a total score. It is possible to output the multi-region evaluation result, which is the result.
  • the multi-region evaluation results include (6) a multi-region frame evaluation result that is a comprehensive evaluation result for each frame, and (7) a multi-region frame part evaluation result that is a comprehensive evaluation result for each anatomical region. can be output.
  • the comprehensive evaluation result it is possible to obtain the evaluation result of a range having a certain degree of spread within the body.
  • ulcerative colitis which is a disease in which the lesion spreads in a plane rather than in a spot-like manner, it is possible to recognize the extent of the disease three-dimensionally and facilitate follow-up by presenting evaluation results over a wide range.
  • the first illumination light mode and the second illumination light mode in the first embodiment are changed to a mono light emission mode, and in addition to the mono light emission mode, a multi light emission mode in which the illumination light is automatically switched is provided. is.
  • the mono light emission mode and the multi light emission mode are switched via the central control unit 50 by operating the mode switch 12f.
  • the mono light emission mode is a mode in which illumination light of the same spectrum (first illumination light or second illumination light) is continuously emitted.
  • the multi-emission mode is a mode in which a plurality of illumination lights (first illumination light and second illumination light) with different spectra are emitted while being switched according to a specific pattern to illuminate the subject.
  • it is preferable that the first illumination light and the second illumination light can be switched in the mono light emission mode.
  • control is performed to change the light amounts of violet light V, blue light B, green light G, and red light R for each frame F according to a specific light emission pattern.
  • emission patterns are given below.
  • the first illumination light L1 for two frames is emitted during the first illumination period Pe1 in which the subject is illuminated with the first illumination light L1, and the second illumination light L2 is emitted.
  • the pattern of sequentially emitting the second illumination light for one frame is repeated in the second illumination period Pe2 in which the subject is illuminated by the .
  • the first illumination light L1 is emitted for one frame F during the first illumination period Pe1
  • the second illumination light L2 is emitted for four frames during the second illumination period Pe2. repeat.
  • the second illumination first spectral light L2SP, the second illumination second spectral light L2SQ, the second illumination third spectral light L2SR, and the second illumination fourth spectrum which have different emission spectra It is automatically switched so that the light L2SS is emitted every frame F.
  • the light emission pattern is not limited to this, and can be set arbitrarily.
  • the light source processor 21 adjusts the light intensity of each light source in accordance with the set light emission pattern for photographing. Note that specific examples of the first illumination light and the second illumination light are the same as those in the first embodiment, and therefore description thereof is omitted. It is preferable that the second illumination light is at least one kind.
  • the first illumination light image which users such as doctors are accustomed to seeing
  • the second illumination light image which is unfamiliar to the user but suitable for analyzing a specific structure such as a blood vessel, is used for image analysis and determination of notification conditions, and is displayed when necessary without being displayed normally. is preferred.
  • the display image 231 is provided with a main section 210 and sub-sections 211.
  • the main section 210 contains the first illumination light image 212 currently being observed, the sub-sections the second illumination light image 213, It is preferable to display the schema 252, the character information 215 indicating the first evaluation result and the second evaluation result, and the like.
  • the medical image acquisition unit 60 acquires the first illumination light image and the second illumination light image.
  • the medical image acquisition unit 60 inputs the first illumination light image and the second illumination light image to the image determination unit 70 .
  • the second illumination light image is input to the first determiner 80 or the second determiner 90
  • the first evaluation result based on the first evaluation criteria is output
  • the first illumination light image is output to the third determiner 100. It is preferable to input and output a second evaluation result based on a second evaluation criterion. Description of the first determiner 80, the second determiner 90, and the third determiner 100 is omitted because they are common to the first embodiment.
  • the medical image acquisition unit 60 inputs the first illumination light image and the second illumination light image to the image dividing unit 160 and divides each into a plurality of regions.
  • Each area obtained by dividing the first illumination light image into a plurality of areas is called a first illumination division
  • each area obtained by dividing the second illumination light image into a plurality of areas is called a second illumination division.
  • the second illumination light image as the second illumination division input to the image determination unit 70 is input to the first determiner 80 or the second determiner 90 as the image processing unit or the trained model, and is used as the first evaluation criterion.
  • a first evaluation result based on is output.
  • the first illumination light image as the first illumination division input to the image determination section 70 is input to the third determiner 100 and the second evaluation result based on the second evaluation criteria is output.
  • the first illumination light image as the first illumination split and the second illumination light image as the second illumination split are input to the area evaluation unit 170, and the area evaluation is performed for each of the first illumination split and the second illumination split. It is preferable to input to the image determination section 70 after attaching an index.
  • the second illumination light image is input to the image determination unit 70 to output the first evaluation result
  • the first illumination light image is input to the image determination unit 70 to output the second evaluation result.
  • the first evaluation result and the second evaluation result are output, and the display image 131 on which the first evaluation result and the second evaluation result are displayed is preferably generated.
  • the first evaluation result output using the second illumination light image of one frame or the second illumination light image as the second illumination division is transmitted to the comprehensive evaluation result calculation unit 71, and the first multi-frame evaluation result, Preferably, a first multi-frame part evaluation result, a first multi-region evaluation result, a first multi-region frame evaluation result, and/or a first multi-region part evaluation result are output.
  • the second evaluation result output using the first illumination light image of one frame or the first illumination light image as the first illumination division is transmitted to the comprehensive evaluation result calculation unit 71, and is used for the second multi-frame evaluation.
  • the result, the second multi-frame part evaluation result, the second multi-region evaluation result, the second multi-region frame evaluation result, and/or the second multi-region part evaluation result are output.
  • the output of multi-frame evaluation results, multi-frame part evaluation results, multi-region evaluation results, multi-region frame evaluation results, and multi-region part evaluation results is common to that of the first embodiment, so description thereof will be omitted.
  • FIG. The main section 210 of the display image 231 displays the first illumination light image 212 currently being observed.
  • a currently observed second illumination light image 213, a multi-region frame evaluation result display column 281, and a multi-region part evaluation result display column 291 are displayed.
  • the multi-region frame evaluation result display column 281 displays the second multi-region frame evaluation result of “remission” for the first illumination light image 212 currently being observed,
  • the second multi-region frame evaluation result of “remission” is displayed as X%
  • the first multi-region frame evaluation result of “remission” for the currently observed second illumination light image 213 is displayed as Y%
  • the second multi-region frame evaluation result of “non-remission” is displayed as Y%.
  • One multi-region frame evaluation result Z% is indicated, and a bar graph corresponding to the multi-region frame evaluation result is displayed near it.
  • the second multiple-region site evaluation result of “remission” and the second multiple-region site evaluation result of “non-remission” in the currently observed site (descending colon) are displayed.
  • Q% the first multiple-region site evaluation result of “remission” is R%
  • the first multiple-region site evaluation result of “non-remission” is S%
  • the multiple-region site evaluation result display column is shown in the vicinity of it. A bar graph is displayed.
  • FIG. 50 the part (descending colon) currently being observed is displayed, and the multi-region part evaluation result for that part is displayed in the multi-region part evaluation result display column 291 .
  • FIG. 51 shows a schematic diagram 252 in which the currently observed part is highlighted with a frame 253 and the multi-region/part evaluation result for that part is displayed in the multi-region/part evaluation result display column 291 .
  • FIG. 52 a three-dimensional structural diagram 254 of the large intestine is displayed, and the “remission” site 256 and the “non-remission” site 257 of the first multiple-region site evaluation result or the second multiple-region site evaluation result are reflected. is displayed.
  • the multi-region/part evaluation result display column 291 may display the multi-region/part evaluation results for an arbitrary part or all parts photographed so far.
  • FIG. 53 shows a simple schematic diagram 258 representing the position of the large intestine in a bar shape, and shows the “remission” site 256 and the “non-remission” site 257 of the first multiple-region site evaluation result or the second multiple-region site evaluation result. Each is reflected and displayed.
  • the multi-region/part evaluation result display column 291 may display the multi-region/part evaluation results for an arbitrary part or all parts photographed so far.
  • the multi-region frame evaluation result display column 281 and the multi-region part evaluation result display column 291 may not be displayed.
  • 54 to 57 show the currently observed first illumination light image 212 and the currently observed second illumination light image 213.
  • the currently observed site is displayed, and in FIG. A schema 252 highlighting a site with a frame 253 reflects a site 256 of "remission” and a site 257 of "non-remission" of the first multi-region site evaluation result or the second multiple-region site evaluation result in FIG.
  • the three-dimensional structure diagram 254 of the large intestine that was made reflects the "remission" site 256 and the "non-remission" site 257 of the first multiple-region site evaluation result or the second multiple-region site evaluation result in FIG.
  • a simple schematic diagram 258 representing the position of the large intestine in a bar shape is displayed.
  • the display image 231 includes a multi-region frame evaluation result display column 281, a multi-region part evaluation result display column 291, and the second illumination light image 213 currently being observed. May not be displayed.
  • 58, 59, 55, 60, and 57, respectively, are display forms in which the second illumination light image 214 currently being observed is not displayed.
  • the notification control unit 200 preferably performs notification in accordance with the first evaluation result and the second evaluation result output by the image determination unit 70 . Since specific conditions for notification are common to the first embodiment, description thereof is omitted.
  • the first illumination light image and the second illumination light image in units of one frame are finally obtained. determines whether a "certain condition" is met. For this reason, if (1), (2), and (4) are satisfied among the specific conditions, as shown in FIG. Notify.
  • the "specific condition" is finally satisfied in one first illumination area or second illumination area unit. to decide whether Therefore, when the specific condition (3) is satisfied, as shown in FIG.
  • a warning display frame 301 is enclosed to give notification.
  • the corresponding first illumination area and second illumination area are the first illumination area associated with the first illumination light image and the second illumination area associated with the second illumination light image of consecutive or close frames in time series. It refers to the positionally corresponding part of .
  • the first evaluation result output using the second illumination light image and the second evaluation result output using the first illumination light image are preferably associated with each other and stored in a memory (not shown). Further, the second illumination light image with the first evaluation result and the first illumination light image with the second evaluation result are the anatomical parts where the first illumination light image or the second illumination light image was captured. Preferably, the information is further associated and stored.
  • an evaluation result based on an endoscopic image is output from a white-light image that doctors are accustomed to seeing, and a special light that can analyze specific structures such as blood vessels as information that cannot be grasped from the white-light image.
  • An evaluation result based on the pathological image can be output from the image. In this case, while observing the familiar white light image, the doctor can recognize "hidden non-remission" that is overlooked in the white light image with the assistance of the medical image processing apparatus.
  • the present invention is not limited to this, and other medical devices such as an ultrasonic imaging device and a radiation imaging device can be used.
  • a device may be used.
  • the endoscope 12 may be a rigid scope or a flexible scope.
  • the central control unit 50, the medical image acquisition unit 60, the image determination unit 70, the comprehensive evaluation result calculation unit 71, the display control unit 130, the image division unit 160, the region evaluation unit 170, and the notification control unit Part or all of 200 may be provided, for example, in an image processing device that communicates with processor device 14 and cooperates with endoscopic system 10 .
  • the endoscope system 10 can be provided in a diagnosis support device that acquires an image captured by the endoscope 12 directly from the endoscope system 10 or indirectly from the PACS.
  • the endoscope system 10 is connected to various inspection devices such as the first inspection device, the second inspection device, .
  • the central control unit 50, the medical image acquisition unit 60, the image determination unit 70, the comprehensive evaluation result calculation unit 71, the display control unit 130, the image division unit 160, the region evaluation unit 170, and the notification control unit 200 can be provided.
  • various components such as the central control unit 50, the medical image acquisition unit 60, the image determination unit 70, the comprehensive evaluation result calculation unit 71, the display control unit 130, the image division unit 160, the region evaluation unit 170, and the notification control unit 200
  • the hardware structure of a processing unit that executes processing is various processors as shown below.
  • the circuit configuration is changed after manufacturing such as CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), which is a general-purpose processor that executes software (program) and functions as various processing units.
  • Programmable Logic Devices which are processors, dedicated electric circuits, which are processors with circuit configurations specially designed to perform various types of processing, and the like.
  • One processing unit may be configured by one of these various processors, and may be configured by a combination of two or more processors of the same type or different types (for example, multiple FPGAs or a combination of a CPU and an FPGA).
  • a plurality of processing units may be configured by a single processor.
  • this processor functions as a plurality of processing units.
  • SoC System On Chip
  • SoC System On Chip
  • the hardware structure of these various processors is, more specifically, an electric circuit in the form of a combination of circuit elements such as semiconductor elements.
  • the hardware structure of the storage unit is a storage device such as an HDD (hard disc drive) or an SSD (solid state drive).
  • endoscope system 12 endoscope 12a insertion portion 12b operation portion 12c bending portion 12d tip portion 12e angle knob 12f observation mode switching switch 12h still image acquisition instruction switch 12i zoom operation portion 12j forceps port 13 light source device 14 processor device 15 display 16 user interface 20 light source 20a V-LED 20b B-LED 20c G-LED 20d R-LED 21 light source processor 22 optical path coupling unit 23 light guide 30a illumination optical system 30b imaging optical system 31 illumination lens 41 objective lens 42 zoom lens 43 imaging sensor 44 imaging processor 50 central control unit 60 medical image acquisition unit 60a, 111a, 118b, 213 second illumination light image 60b, 118a, 212 first illumination light image 60c endoscope image 70 image determination unit 71 comprehensive evaluation result calculation unit 72 multiple frame evaluation result calculation unit 73 multiple region evaluation result calculation unit 80 first determiner 80a, 90a First evaluation result 81 Intramucosal bleeding 82 Extramucosal bleeding 90 Second determiner 100 Third determiner 101 Second evaluation result 110 Frequency domain image

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Surgery (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Biomedical Technology (AREA)
  • Optics & Photonics (AREA)
  • Pathology (AREA)
  • Radiology & Medical Imaging (AREA)
  • Biophysics (AREA)
  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Endoscopes (AREA)
PCT/JP2022/035958 2021-10-04 2022-09-27 内視鏡システム、医療画像処理装置及びその作動方法 Ceased WO2023058503A1 (ja)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2023552817A JPWO2023058503A1 (https=) 2021-10-04 2022-09-27

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021-163712 2021-10-04
JP2021163712 2021-10-04

Publications (1)

Publication Number Publication Date
WO2023058503A1 true WO2023058503A1 (ja) 2023-04-13

Family

ID=85804272

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/035958 Ceased WO2023058503A1 (ja) 2021-10-04 2022-09-27 内視鏡システム、医療画像処理装置及びその作動方法

Country Status (2)

Country Link
JP (1) JPWO2023058503A1 (https=)
WO (1) WO2023058503A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2026042270A1 (ja) * 2024-08-23 2026-02-26 日本電気株式会社 内視鏡検査支援装置、内視鏡検査支援方法、及び、記録媒体
WO2026078862A1 (ja) * 2024-10-11 2026-04-16 日本電気株式会社 学習装置、画像処理装置、学習方法、画像処理方法及び記憶媒体

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020121906A1 (ja) * 2018-12-13 2020-06-18 ソニー株式会社 医療支援システム、医療支援装置及び医療支援方法
WO2021006121A1 (ja) * 2019-07-08 2021-01-14 富士フイルム株式会社 画像処理装置、内視鏡システム、及び画像処理装置の作動方法
WO2021157392A1 (ja) * 2020-02-07 2021-08-12 富士フイルム株式会社 画像処理装置、内視鏡システム、及び画像処理方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020121906A1 (ja) * 2018-12-13 2020-06-18 ソニー株式会社 医療支援システム、医療支援装置及び医療支援方法
WO2021006121A1 (ja) * 2019-07-08 2021-01-14 富士フイルム株式会社 画像処理装置、内視鏡システム、及び画像処理装置の作動方法
WO2021157392A1 (ja) * 2020-02-07 2021-08-12 富士フイルム株式会社 画像処理装置、内視鏡システム、及び画像処理方法

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
ANONYMOUS: "Artificial Intelligence (AI) Enables Endoscopic Image Evaluation of Ulcerative Colitis!" -Experimental Development of Computer Image Support System (DNUC)- 「人工知能（ＡＩ）により潰瘍性大腸炎の内視鏡画像評価が可能に！」 －コンピューター画像支援システム（ＤＮＵＣ）を試験的に開発", NATIONAL UNIVERSITY CORPORATION TOKYO MEDICAL AND DENTAL UNIVERSITY, 2 March 2020 (2020-03-02), pages 1 - 6, XP093057005, [retrieved on 20230622] *
LEMMENS, Bart et al. Correlation Between the Endoscopic and Histologic Score in Assessing the Activity of Ulcerative Colitis. INFLAMMATORY BOWEL DISEASES. 01 May 2013, vol. 19, no. 6, pp. 1194-1201 *
OKUSA TOSHIFUMI, TARO NAGATA , TAKESHI TERAI , NOBUHIRO SATO , ISAO OKAYASU: "Evaluation of endoscopic and histological images before and after treatment for ulcerative colitis -Proposal of seven-section colonoscopy and histological score summation method", vol. 48, no. 4, 1 April 2006 (2006-04-01), pages 977 - 986, XP093055869 *
TAKENAKA KENTO; OHTSUKA KAZUO; FUJII TOSHIMITSU; NEGI MARIKO; SUZUKI KOHEI; SHIMIZU HIROMICHI; OSHIMA SHIORI; AKIYAMA SHINTARO; MO: "Development and Validation of a Deep Neural Network for Accurate Evaluation of Endoscopic Images From Patients With Ulcerative Colitis", GASTROENTEROLOGY, ELSEVIER INC., US, vol. 158, no. 8, 12 February 2020 (2020-02-12), US , pages 2150 - 2157, XP086171054, ISSN: 0016-5085, DOI: 10.1053/j.gastro.2020.02.012 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2026042270A1 (ja) * 2024-08-23 2026-02-26 日本電気株式会社 内視鏡検査支援装置、内視鏡検査支援方法、及び、記録媒体
WO2026078862A1 (ja) * 2024-10-11 2026-04-16 日本電気株式会社 学習装置、画像処理装置、学習方法、画像処理方法及び記憶媒体

Also Published As

Publication number Publication date
JPWO2023058503A1 (https=) 2023-04-13

Similar Documents

Publication Publication Date Title
JP7346285B2 (ja) 医療画像処理装置、内視鏡システム、医療画像処理装置の作動方法及びプログラム
JP6785941B2 (ja) 内視鏡システム及びその作動方法
US11426054B2 (en) Medical image processing system, endoscope system, diagnosis support apparatus, and medical service support apparatus
JP6831463B2 (ja) 医療画像処理装置、内視鏡システム、診断支援装置、並びに医療業務支援装置
JP7337073B2 (ja) 医用画像処理装置及び内視鏡システム並びに医用画像処理装置の作動方法
US12561955B2 (en) Endoscope system and method of operating the same
WO2020162275A1 (ja) 医療画像処理装置、内視鏡システム、及び医療画像処理方法
JP7125479B2 (ja) 医療画像処理装置、医療画像処理装置の作動方法及び内視鏡システム
JPWO2007119297A1 (ja) 医療用画像処理装置及び医療用画像処理方法
JP7289373B2 (ja) 医療画像処理装置、内視鏡システム、診断支援方法及びプログラム
JP2022189900A (ja) 画像処理装置、内視鏡システム及び画像処理装置の作動方法
CN112752535B (zh) 医用图像处理装置和内窥镜系统以及医用图像处理装置的工作方法
CN116133572B (zh) 图像分析处理装置、内窥镜系统、图像分析处理装置的工作方法及存储器
JP2023026480A (ja) 医療画像処理装置、内視鏡システム、及び医療画像処理装置の作動方法
JP7335399B2 (ja) 医用画像処理装置及び内視鏡システム並びに医用画像処理装置の作動方法
JP7402314B2 (ja) 医用画像処理システム、医用画像処理システムの作動方法
JPWO2020040087A1 (ja) 医療画像処理システム
WO2023058503A1 (ja) 内視鏡システム、医療画像処理装置及びその作動方法
JP7610342B2 (ja) 内視鏡システム
JP7775047B2 (ja) 内視鏡システム、医療画像処理装置及びその作動方法
WO2023153069A1 (ja) 医療画像装置、内視鏡システム、及び、診断書作成システム
CN115607092A (zh) 内窥镜系统、医疗图像处理装置及其工作方法
JP7389257B2 (ja) 内視鏡システム及びその作動方法
US20230237659A1 (en) Image processing apparatus, endoscope system, operation method of image processing apparatus, and non-transitory computer readable medium
JP6866497B2 (ja) 医療画像処理装置、及び、内視鏡装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22878373

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023552817

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22878373

Country of ref document: EP

Kind code of ref document: A1