WO2021220822A1 - 画像診断装置、画像診断方法、画像診断プログラムおよび学習済みモデル - Google Patents

画像診断装置、画像診断方法、画像診断プログラムおよび学習済みモデル Download PDF

Info

Publication number
WO2021220822A1
WO2021220822A1 PCT/JP2021/015555 JP2021015555W WO2021220822A1 WO 2021220822 A1 WO2021220822 A1 WO 2021220822A1 JP 2021015555 W JP2021015555 W JP 2021015555W WO 2021220822 A1 WO2021220822 A1 WO 2021220822A1
Authority
WO
WIPO (PCT)
Prior art keywords
endoscopic
image
esophagus
iodine
unstained
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/015555
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
洋平 池之山
翔 城間
敏之 由雄
智裕 多田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Japanese Foundation for Cancer Research
AI Medical Service Inc
Original Assignee
Japanese Foundation for Cancer Research
AI Medical Service Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Japanese Foundation for Cancer Research, AI Medical Service Inc filed Critical Japanese Foundation for Cancer Research
Priority to US17/997,028 priority Critical patent/US20230255467A1/en
Priority to CN202180030877.3A priority patent/CN115460968A/zh
Priority to JP2022517627A priority patent/JP7550409B2/ja
Publication of WO2021220822A1 publication Critical patent/WO2021220822A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/273—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for the upper alimentary canal, e.g. oesophagoscopes, gastroscopes
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A61B1/000094—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope extracting biological structures
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A61B1/000096—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope using artificial intelligence
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00043—Operational features of endoscopes provided with output arrangements
    • A61B1/00045—Display arrangement
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00043—Operational features of endoscopes provided with output arrangements
    • A61B1/00045—Display arrangement
    • A61B1/0005—Display arrangement combining images e.g. side-by-side, superimposed or tiled
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A61B1/045—Control thereof
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/273—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for the upper alimentary canal, e.g. oesophagoscopes, gastroscopes
    • A61B1/2733—Oesophagoscopes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00043—Operational features of endoscopes provided with output arrangements
    • A61B1/00055—Operational features of endoscopes provided with output arrangements for alerting the user
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30092—Stomach; Gastric
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion

Definitions

  • the present invention relates to a diagnostic imaging apparatus, a diagnostic imaging method, a diagnostic imaging program, and a trained model.
  • Esophageal cancer is the eighth most common cancer in the world among all carcinomas, and has the sixth highest cancer-related mortality rate, with more than 500,000 deaths annually.
  • squamous cell carcinoma of the esophagus is common in South America and Asia (including Japan).
  • Advanced esophageal cancer has a poor prognosis, but superficial esophageal cancer can be treated with minimally invasive treatment such as endoscopic resection if detected early, and the prognosis is also good. Therefore, early detection of superficial esophageal cancer is the most important issue.
  • ESD endoscopic submucosal dissection
  • NBI Narrow Band Imaging, narrow-band imaging
  • esophageal cancer has little change in color tone and develops as a flat lesion with almost no unevenness, and it is difficult to recognize such findings as lesions without skill.
  • the background mucosa is often accompanied by inflammation, inexperienced endoscopists tend to confuse the inflamed mucosa with esophageal cancer, making it even more difficult to determine cancer lesions. In this way, it is still difficult to properly diagnose esophageal cancer endoscopically, even if it is generally called the gastrointestinal tract, even if it is compared with colon cancer, which is characterized by polyps. Diagnostic technology is required in the field of endoscopic diagnosis.
  • iodine-unstained zones are associated with heavy smoking and drinking, and low intake of green and yellow vegetables, and multiple iodine-unstained zones present in the esophagus are caused by mutations in the cancer suppressor gene TP53 in the background epithelium.
  • subjects with multiple iodine-unstained zones are at high risk of esophageal cancer and head and neck cancer. Therefore, observation using iodine staining is performed by endoscopy for esophageal cancer. Suitable for precise screening of head and neck cancer.
  • iodine staining has problems such as chest discomfort (side effects) and prolonged operation time, so it is not realistic to use it in all cases, and cases with a history of esophageal cancer and head and neck cancer. It is desirable to select using a very limited number of high-risk cases, such as cases with complications. Further rapid and useful methods such as a high-precision test method that does not require iodine staining or a test method that combines iodine staining as needed are required for early detection of esophageal cancer.
  • AI Artificial Intelligence
  • CNN convolutional neural network
  • CAD Computer-aided diagnosis
  • Image judgment technology by deep learning in the medical field is various reports that AI supports the diagnosis of specialists such as radiological image diagnosis, skin cancer classification, histological classification of pathological specimens, colon lesion detection by super-magnifying endoscopy, etc. There is. In particular, it has been proved that AI can obtain the same accuracy as a specialist at the microscopic endoscopy level (see Non-Patent Document 1). In addition, in dermatology, it has been announced that AI with a deep learning function exhibits the same diagnostic imaging ability as a specialist (see Non-Patent Document 2), and patent documents using various machine learning methods (patents). (See References 1 and 2) also exists.
  • AI's diagnostic imaging ability in the medical field is comparable to that of specialists in some areas, but AI's diagnostic imaging ability is used to diagnose esophageal cancer with high accuracy in real time.
  • the technology has not yet been introduced in the actual medical field (actual clinical practice), and it is expected that it will be put into practical use at an early stage in the future.
  • cancer diagnostic imaging criteria based on the characteristics of the cancer tissue, such as morphological characteristics and tissue-derived biochemical biomarkers and cell biological reactions, are essential, so endoscopy is used. Even if it is said to be a diagnosis of gastrointestinal cancer, if the organs are different, the AI diagnosis program also requires the design of techniques and criteria optimized for each organ.
  • flat esophageal cancer has a different form from colorectal cancer, which is easy to detect with raised polyps, and is more difficult and requires new ingenuity and technology. Since there is a high possibility that the accuracy and judgment of the results obtained will differ depending on the experience of the operator of medical equipment, some of the ingenuity and technology include not only the functions related to image processing of endoscopes, but also the equipment operator. Methods of optimizing the operation of an endoscopist should also be considered. That is, the extraction of the unique characteristic amount of each gastrointestinal cancer (esophageal cancer, gastric cancer, colorectal cancer, etc.) and the judgment criteria of the pathological level are different, and the design of the AI program that matches the characteristics of each cancer type.
  • gastrointestinal cancer esophageal cancer, gastric cancer, colorectal cancer, etc.
  • An object of the present invention is to provide a diagnostic imaging apparatus, a diagnostic imaging method, and a diagnostic imaging program capable of improving the diagnostic accuracy of esophageal cancer in esophageal endoscopy.
  • the diagnostic imaging apparatus is An endoscopic image acquisition unit that acquires an endoscopic moving image of the subject's esophagus, Estimation to estimate the position of esophageal cancer present in the acquired endoscopic moving image using a convolutional neural network trained as teacher data from an esophageal cancer image that images the esophagus in which esophageal cancer is present Department and A display control unit that superimposes and displays the estimated position of esophageal cancer and the degree of certainty that indicates the possibility that esophageal cancer exists at the position on the endoscopic moving image. To be equipped.
  • the diagnostic imaging method for acquiring endoscopic moving images of the subject's esophagus, Estimation to estimate the position of esophageal cancer present in the acquired endoscopic moving image using a convolutional neural network trained as teacher data from an esophageal cancer image that images the esophagus in which esophageal cancer is present
  • the diagnostic imaging program On the computer Endoscopic image acquisition processing to acquire endoscopic moving images of the subject's esophagus, Estimation to estimate the position of esophageal cancer present in the acquired endoscopic moving image using a convolutional neural network trained as teacher data from an esophageal cancer image that images the esophagus in which esophageal cancer is present Processing and Display control processing that superimposes and displays the estimated position of esophageal cancer and the degree of certainty that indicates the possibility that esophageal cancer exists at that position on the endoscopic moving image. To execute.
  • the trained model according to the present invention A non-iodine-stained image of the esophagus in which multiple iodine-unstained zones are present without iodine staining, and a non-iodine-stained image of the esophagus in which multiple iodine-unstained zones are not present. It is obtained by training a convolutional neural network using a non-multiple iodine-unstained esophageal image as training data. The computer is made to function to estimate the relationship between the endoscopic image of the subject's esophagus and esophageal cancer and output the estimation result.
  • FIGS. 7A, 7B, and 7C are diagrams showing an example of an endoscopic image of the esophagus when iodine solution is sprayed into the lumen of the esophagus in the second embodiment. It is a figure which shows the characteristic of the subject and the lesion (esophageal cancer) about the endoscopic moving image (low velocity) used for the data set for evaluation test. It is a figure which shows the characteristic of the subject and the lesion (esophageal cancer) about the endoscopic moving image (high speed) used for the data set for evaluation test.
  • 14A, 14B, 14C, 14D, 14E, 14F, 14G, 14H, 14I are diagrams showing various endoscopic findings in endoscopic images. It is a figure which shows the sensitivity, specificity, positive predictive value, negative predictive value and correct diagnosis rate of a diagnostic imaging apparatus, an endoscopist. A diagram showing the evaluation result of the presence or absence of endoscopic findings in an endoscopic image having multiple iodine unstained bands and the evaluation result of the presence or absence of endoscopic findings in an endoscopic image without multiple iodine unstained bands. Is. It is a figure which shows the comparison result of the image diagnostic apparatus and endoscopic findings about whether or not it is possible to correctly diagnose the existence of multiple iodine unstained bands in an endoscopic image (sensitivity). Incidence of squamous cell carcinoma of the esophagus and squamous cell carcinoma of the head and neck and the incidence per 100 man-years It is a figure showing.
  • the first embodiment comprises a real-time moving image diagnostic device, a diagnostic imaging method, and a diagnostic imaging program
  • the second embodiment is trained with teacher data relating to multiple iodine-unstained zones by iodine staining of the esophageal lumen. It consists of an image diagnosis device using iodine, an image diagnosis method, and an image diagnosis program.
  • the first embodiment or the second embodiment may be performed alone, or the first embodiment and the second embodiment may be combined.
  • FIG. 1 is a block diagram showing an overall configuration of the diagnostic imaging apparatus 100.
  • FIG. 2 is a diagram showing an example of the hardware configuration of the diagnostic imaging apparatus 100 according to the first embodiment.
  • the diagnostic imaging apparatus 100 is an endoscopic image possessed by a convolutional neural network (CNN) in an endoscopic examination of a digestive organ (esophagus in the present embodiment) by a doctor (for example, an endoscopist). Diagnose esophageal cancer with real-time video using the diagnostic imaging capabilities of.
  • An endoscopic imaging device 200 and a display device 300 are connected to the diagnostic imaging device 100.
  • the endoscope imaging device 200 is, for example, an electronic endoscope (also referred to as a videoscope) having a built-in imaging means, a camera-mounted endoscope in which a camera head having a built-in imaging means is attached to an optical endoscope, or the like. be.
  • the endoscopic imaging device 200 is inserted into the digestive organ, for example, through the mouth or nose of the subject, and images a diagnosis target site in the digestive organ.
  • the endoscope imaging device 200 irradiates the esophagus of the subject with white light or narrow band light (for example, narrow band light for NBI) in response to a doctor's operation (for example, button operation).
  • a doctor's operation for example, button operation.
  • the part to be diagnosed in the esophagus is imaged as an endoscopic moving image.
  • the endoscopic moving image is composed of a plurality of endoscopic images that are continuous in time.
  • the endoscope imaging device 200 outputs the endoscopic image data D1 representing the captured endoscopic moving image to the diagnostic imaging device 100.
  • the display device 300 is, for example, a liquid crystal display, and displays the endoscopic moving image and the determination result image output from the diagnostic imaging device 100 so that the doctor can identify them.
  • the diagnostic imaging apparatus 100 has a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, and an external storage device (for example, a flash memory) as main components. It is a computer equipped with 104, a communication interface 105, a GPU (Graphics Processing Unit) 106, and the like.
  • CPU Central Processing Unit
  • ROM Read Only Memory
  • RAM Random Access Memory
  • an external storage device for example, a flash memory
  • Each function of the diagnostic imaging apparatus 100 includes, for example, a control program (for example, an diagnostic imaging program) in which the CPU 101 and GPU 106 are stored in a ROM 102, a RAM 103, an external storage device 104, and various data (for example, endoscopic image data, learning). It is realized by referring to the training data, the model data of the convolutional neural network (structural data, learned weight parameters, etc.), and the RAM 103 functions as, for example, a data work area or a temporary save area.
  • a control program for example, an diagnostic imaging program
  • the CPU 101 and GPU 106 are stored in a ROM 102, a RAM 103, an external storage device 104, and various data (for example, endoscopic image data, learning). It is realized by referring to the training data, the model data of the convolutional neural network (structural data, learned weight parameters, etc.), and the RAM 103 functions as, for example, a data work area or a temporary save area.
  • diagnostic imaging apparatus 100 may be realized by processing by the DSP (Digital Signal Processor) instead of or in combination with the processing by the CPU 101 and GPU 106.
  • DSP Digital Signal Processor
  • a part or all of each function may be realized by processing by a dedicated hardware circuit in place of or in combination with processing by software.
  • the diagnostic imaging apparatus 100 includes an endoscopic image acquisition unit 10, an estimation unit 20, and a display control unit 30.
  • the learning device 40 has a function of generating model data (corresponding to the "learned model” of the present invention) of the convolutional neural network used in the diagnostic imaging device 100.
  • the display control unit 30 also functions as the "warning output control unit” of the present invention.
  • Endoscopic image acquisition unit The endoscope image acquisition unit 10 acquires the endoscope image data D1 output from the endoscope imaging device 200. Then, the endoscopic image acquisition unit 10 outputs the acquired endoscopic image data D1 to the estimation unit 20. When the endoscope image acquisition unit 10 acquires the endoscope image data D1, the endoscope image acquisition unit 10 may directly acquire the endoscope image data D1 or the endoscope image data stored in the external storage device 104. Endoscopic image data D1 provided via D1 or an internet line may be acquired.
  • the estimation unit 20 uses a convolutional neural network to display a lesion in the endoscopic moving image represented by the endoscopic image data D1 output from the endoscopic image acquisition unit 10 (in the present embodiment, the esophagus is formed. The existence of) is estimated and the estimation result is output. Specifically, the estimation unit 20 determines the lesion name (name) and lesion position (position) of the lesion existing in the endoscopic moving image and the certainty (also referred to as accuracy) of the lesion name and lesion position. presume. Then, the estimation unit 20 displays the endoscopic image data D1 output from the endoscopic image acquisition unit 10 and the estimation result data D2 representing the estimation result of the lesion name, the lesion position, and the certainty in the display control unit 30. Output.
  • the endoscopic image whose certainty is a predetermined value (for example, 0.5) or more in the endoscopic moving image represented by the endoscopic image data D1 is for a predetermined time (for example, 0). If a predetermined number (for example, 3) is present within .5 seconds), it is presumed that a lesion (esophageal cancer) is present in the endoscopic moving image.
  • the predetermined number is set so as to increase as the predetermined value decreases.
  • the estimation unit 20 estimates the probability score as an index showing the certainty of the lesion name and the lesion position.
  • the probability score is represented by a value greater than 0 and less than or equal to 1. The higher the probability score, the higher the certainty of the lesion name and lesion location.
  • the probability score is an example of an index indicating the degree of certainty of the lesion name and the lesion position, and an index of any other aspect may be used.
  • the probability score may be represented by a value of 0% to 100%, or may be represented by any of several levels.
  • a convolutional neural network is a type of feedforward neural network and is based on knowledge in the structure of the visual cortex of the brain. Basically, it has a structure in which a convolution layer responsible for extracting local features of an image and a pooling layer (subsampling layer) that summarizes features for each local area are repeated. According to each layer of the convolutional neural network, it possesses multiple neurons (Neurons), and each neuron is arranged so as to correspond to the visual cortex. The basic function of each neuron consists of signal input and output. However, when transmitting signals between neurons in each layer, instead of outputting the input signal as it is, a coupling load is set for each input, and the sum of the weighted inputs is each.
  • a signal is output to the neuron in the next layer.
  • the connection load between these neurons is calculated from the training data. This makes it possible to estimate the output value by inputting real-time data.
  • convolutional neural network models include GoogLeNet, ResNet, and SENEt, but the algorithm for constructing the convolutional neural network is not particularly limited as long as it is a convolutional neural network suitable for this purpose.
  • FIG. 3 is a diagram showing a configuration of a convolutional neural network according to the present embodiment.
  • the model data (structural data, learned weight parameters, etc.) of the convolutional neural network is stored in the external storage device 104 together with the diagnostic imaging program.
  • the convolutional neural network has, for example, a feature extraction unit Na and an identification unit Nb.
  • the feature extraction unit Na performs a process of extracting image features from an input image (specifically, an endoscopic image constituting an endoscopic moving image represented by the endoscopic image data D1).
  • the identification unit Nb outputs an estimation result related to the image from the image features extracted by the feature extraction unit Na.
  • the feature extraction unit Na is configured by hierarchically connecting a plurality of feature amount extraction layers Na1, Na2, and so on.
  • Each feature amount extraction layer Na1, Na2 ... Provides a convolution layer, an activation layer, and a pooling layer.
  • the feature amount extraction layer Na1 of the first layer scans the input image for each predetermined size by raster scanning. Then, the feature amount extraction layer Na1 extracts the feature amount contained in the input image by performing the feature amount extraction process on the scanned data by the convolutional layer, the activation layer and the pooling layer.
  • the feature amount extraction layer Na1 of the first layer extracts a relatively simple single feature amount such as a linear feature amount extending in the horizontal direction and a linear feature amount extending in the diagonal direction.
  • the feature amount sampling layer Na2 of the second layer scans an image (also referred to as a feature map) input from the feature amount sampling layer Na1 of the previous layer at predetermined size intervals by, for example, raster scanning. Then, the feature amount extraction layer Na2 extracts the feature amount contained in the input image by similarly performing the feature amount extraction process by the convolutional layer, the activation layer and the pooling layer on the scanned data.
  • the feature amount extraction layer Na2 of the second layer is integrated with reference to the positional relationship of a plurality of feature amounts extracted by the feature amount extraction layer Na1 of the first layer, so that it is a higher-dimensional complex. Extract features.
  • the feature amount sampling layers after the second layer perform the same processing as the feature amount extraction layer Na2 of the second layer. do. Then, the output of the feature amount sampling layer of the final layer (each value in the map of the plurality of feature maps) is input to the identification unit Nb.
  • the identification unit Nb is composed of, for example, a multi-layer perceptron in which a plurality of fully connected layers (Fully Connected) are hierarchically connected.
  • the fully connected layer on the input side of the identification unit Nb is fully connected to each value in the map of a plurality of feature maps acquired from the feature extraction unit Na, and the product-sum operation is performed while changing the weighting coefficient for each value. Go and output.
  • the fully connected layer of the next layer of the identification unit Nb is fully coupled to the values output by each element of the fully connected layer of the previous layer, and the product-sum operation is performed while applying different weighting factors to each value. Then, in the final stage of the identification unit Nb, the lesion name and lesion position of the lesion existing in the image (endoscopic image) input to the feature extraction unit Na, and the probability score (confidence) of the lesion name and lesion position.
  • a layer for example, a softmax function, etc. that outputs a degree
  • the convolutional neural network is desired from the input endoscopic image by performing learning processing using reference data (hereinafter referred to as "teacher data") marked in advance by an experienced endoscopist.
  • the estimation function can be possessed so that the estimation result (here, the lesion name, the lesion position and the probability score) can be output.
  • the estimation result here, the lesion name, the lesion position and the probability score
  • over-learning can be prevented and performance generalized for esophageal cancer diagnosis. It is possible to make an AI program having.
  • the convolutional neural network in the present embodiment receives the endoscopic image data D1 as an input (Input in FIG. 3), and is an image of the endoscopic image constituting the endoscopic moving image represented by the endoscopic image data D1. It is configured to output the lesion name, the lesion position, and the probability score according to the characteristics as the estimation result data D2 (Input in FIG. 3).
  • the convolutional neural network has a configuration in which information related to the subject's age, gender, region, or medical history can be input in addition to the endoscopic image data D1 (for example, an input element of the identification unit Nb). It may be provided as). Since the importance of real-world data in clinical practice is particularly recognized, it is possible to develop a more useful system in clinical practice by adding such information on subject attributes. That is, the characteristics of the endoscopic image are considered to have a correlation with information related to the subject's age, gender, region, medical history, family medical history, etc. By referring to the subject attribute information such as age in addition to D1, the lesion name and the lesion position can be estimated with higher accuracy. Since the pathophysiology of the disease may differ depending on the region and race, this method should be adopted especially when the present invention is used internationally.
  • the estimation unit 20 also performs processing for converting the size and aspect ratio of the endoscope image, color division processing for the endoscope image, and color conversion processing for the endoscope image as preprocessing. , Color extraction processing, brightness gradient extraction processing, and the like may be performed. In order to prevent overfitting and improve accuracy, it is also preferable to adjust the weighting.
  • the display control unit 30 has a lesion name and a lesion represented by the estimation result data D2 output from the estimation unit 20 on the endoscopic moving image represented by the endoscopic image data D1 output from the estimation unit 20. A judgment result image for superimposing and displaying the position and probability score is generated. Then, the display control unit 30 outputs the endoscopic image data D1 and the determination result image data D3 representing the generated determination result image to the display device 300.
  • a digital image processing system such as structural enhancement, color enhancement, difference processing, high contrast, and high definition of the lesion part of the endoscopic moving image is connected to understand and judge the observer (for example, a doctor). It can also be displayed with some processing to help.
  • the display device 300 superimposes and displays the determination result image represented by the determination result image data D3 on the endoscopic moving image represented by the endoscope image data D1 output from the display control unit 30.
  • the endoscopic moving image and the determination result image displayed on the display device 300 are used for real-time diagnostic assistance and diagnostic support by a doctor.
  • the display control unit 30 when the probability score is equal to or higher than a certain threshold value (for example, 0.4), the display control unit 30 displays a rectangular frame indicating the lesion position, the lesion name, and the probability score on the endoscopic moving image. Overlay display.
  • a certain threshold for example, 0.4
  • the display control unit 30 sets the probability score on the endoscopic moving image. The rectangular frame indicating the lesion position, the lesion name, and the probability score are not displayed. That is, the display control unit 30 changes the display mode of the determination result image on the endoscopic moving image according to the probability score represented by the estimation result data D2 output from the estimation unit 20.
  • the display control unit 30 controls the display device 300 and emits a screen for displaying the endoscopic moving image.
  • a warning is displayed and output by causing the lesion to blink or blinking the rectangular area of the lesion determination part. This can effectively alert the doctor that the lesion is present in the endoscopic moving image.
  • a warning sound may be sounded (output) from a speaker (not shown) to output a warning. Further, at this time, it is also possible to independently calculate and display the determination probability and the estimated probability.
  • FIG. 4 is a diagram showing an example in which the determination result image is superimposed and displayed on the endoscopic moving image.
  • FIG. 4 is an endoscopic moving image of a diagnosis target site in the esophagus of a subject in a state where the esophagus of the subject is irradiated with narrow-band light.
  • a rectangular frame 50 indicating the lesion position (range) estimated by the estimation unit 20 is displayed as the determination result image.
  • the plurality of (for example, three) endoscopic images displayed on the left side of FIG. 4 capture an endoscopic image having a certainty of a predetermined value (for example, 0.5) or more in the endoscopic moving image.
  • the convolutional neural network of the estimation unit 20 estimates the lesion position, the lesion name, and the probability score from the endoscopic image data D1 (specifically, the endoscopic image constituting the endoscopic moving image).
  • the teacher data D4 stored in an external storage device (not shown) is input, and the convolutional neural network of the learning device 40 is subjected to learning processing.
  • the learning device 40 irradiates the esophagus of a plurality of subjects with white light or narrow band light in the endoscopy of the esophagus performed in the past, and images the esophagus with the endoscopic imaging device 200.
  • Perform processing Specifically, the learning device 40 reduces the error (also referred to as loss) of the output data with respect to the correct answer value (fault name and lesion position) when the endoscopic image is input to the convolutional neural network.
  • the learning device 40 uses an endoscopic image (corresponding to the “esophageal cancer image” of the present invention) in which a lesion (esophageal cancer) is reflected, that is, an existing endoscopic image (corresponding to the “esophageal cancer image” of the present invention) as teacher data D4. And perform learning processing.
  • an endoscopic image corresponding to the “esophageal cancer image” of the present invention
  • an existing endoscopic image corresponding to the “esophageal cancer image” of the present invention
  • Endoscopic images as teacher data D4 in learning processing mainly use the abundant database of Japan's top-class cancer treatment hospitals, and have abundant diagnosis and treatment experience. Examined and selected all the images in detail, and marked the location of the lesion (esophageal cancer) by precise manual treatment.
  • teacher data D4 endoscopic image data
  • an expert endoscopist with abundant experience can directly connect to the diagnostic accuracy of the diagnostic imaging apparatus 100.
  • a sufficient number of cases with image selection, lesion identification, and feature extraction marking is an extremely important process.
  • Such high-precision data cleansing work and use of high-quality reference data provide highly reliable AI program output results.
  • the teacher data D4 of the endoscopic image may be pixel value data or data that has undergone a predetermined color conversion process or the like. Further, as the pretreatment, a texture feature, a shape feature, an uneven state, a spread feature, etc., which are characteristic of the cancerous part, may be extracted from a comparison between an inflammatory image and a non-inflammatory image. Further, the teacher data D4 may perform learning processing in association with information related to the subject's age, gender, region or medical history, family medical history, etc., in addition to the endoscopic image data.
  • the algorithm when the learning device 40 performs the learning process may be a known method.
  • the learning device 40 uses, for example, a known backpropagation (backpropagation method) to perform learning processing on a convolutional neural network and adjust network parameters (weighting coefficient, bias, etc.).
  • backpropagation method backpropagation method
  • the model data (structural data, learned weight parameters, etc.) of the convolutional neural network subjected to the learning process by the learning device 40 is stored in the external storage device 104 together with the diagnostic imaging program, for example.
  • Examples of known convolutional neural network models include GoogleLeNet, ResNet, and SENEt.
  • the diagnostic imaging apparatus 100 includes an endoscopic image acquisition unit 10 that acquires an endoscopic moving image of the esophagus of a subject and an esophagus in which esophageal cancer is present. It is provided with an estimation unit 20 that estimates the presence of esophageal cancer in the acquired endoscopic moving image using a convolutional neural network trained from the captured esophageal cancer image as teacher data, and outputs the estimation result. ..
  • the convolutional neural network is obtained in advance for each of the plurality of subjects and the endoscopic images (esophageal cancer images) of the plurality of esophagus (digestive organs) obtained in advance for each of the plurality of subjects. It is learned based on the lesion name of the lesion (esophageal cancer) and the definite judgment result of the lesion position. Therefore, it is possible to estimate the lesion name and lesion position of the esophagus of a new subject in a short time and with an accuracy comparable to that of a substantially experienced endoscopist. Therefore, in endoscopy of the esophagus, esophageal cancer can be diagnosed in real time by using the diagnostic ability of the endoscopic moving image possessed by the convolutional neural network according to the present embodiment.
  • the diagnostic imaging apparatus 100 can also be used as a diagnostic support tool that directly supports the diagnosis of endoscopic moving images by an endoscopist in a laboratory.
  • the diagnostic imaging apparatus 100 can be used as a central diagnostic support service that supports the diagnosis of endoscopic moving images transmitted from a plurality of laboratories, or can be remotely controlled via an Internet line in a remote institution. It can also be used as a diagnostic support service to support the diagnosis of endoscopic moving images.
  • the diagnostic imaging apparatus 100 can also be operated on the cloud.
  • these endoscopic moving images and AI judgment results can be directly converted into a video library and used as teaching materials and materials for education and training and research.
  • FIG. 5 is a block diagram showing the overall configuration of the diagnostic imaging apparatus 100A.
  • the diagnostic imaging apparatus 100A uses the diagnostic imaging capability of the endoscopic image possessed by the convolutional neural network in the endoscopy of the digestive organs (esophagus in the present embodiment) by a doctor (for example, an endoscopist). , Estimate the presence or absence of multiple iodine-unstained zones in the endoscopic image of the subject's esophagus.
  • the multiple iodine-unstained zone is a portion that shows a yellowish white color without being stained brown when the iodine solution is sprayed into the lumen of the esophagus.
  • An endoscopic imaging device 200A and a display device 300A are connected to the diagnostic imaging device 100A.
  • the endoscope imaging device 200A is, for example, an electronic endoscope (also referred to as a videoscope) having a built-in imaging means, a camera-mounted endoscope in which a camera head having a built-in imaging means is attached to an optical endoscope, or the like. be.
  • the endoscopic imaging device 200A is inserted into the digestive tract from, for example, the mouth or nose of a subject, and images a diagnosis target site in the digestive tract.
  • the endoscope imaging device 200A irradiates the esophagus of the subject with white light or narrow band light (for example, narrow band light for NBI) in response to a doctor's operation (for example, button operation).
  • a doctor's operation for example, button operation.
  • the site to be diagnosed in the esophagus is imaged as an endoscopic image.
  • the endoscope imaging device 200A outputs the endoscopic image data D1 representing the captured endoscopic image to the diagnostic imaging device 100A.
  • the display device 300A is, for example, a liquid crystal display, and displays the endoscopic image and the determination result image output from the diagnostic imaging device 100A so that the doctor can identify them.
  • the diagnostic imaging apparatus 100A has a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, and external storage as main components.
  • a computer including a device (for example, a flash memory) 104, a communication interface 105, a GPU (Graphics Processing Unit) 106, and the like (see FIG. 2).
  • Each function of the diagnostic imaging apparatus 100A includes, for example, a control program (for example, an diagnostic imaging program) in which the CPU 101 and GPU 106 are stored in the ROM 102, RAM 103, an external storage device 104, and various data (for example, endoscopic image data, a teacher). It is realized by referring to data, model data of convolutional neural network (structural data, trained weight parameters, etc.).
  • the RAM 103 functions as, for example, a data work area or a temporary save area.
  • diagnostic imaging apparatus 100A may be realized by processing by the DSP (Digital Signal Processor) instead of or in combination with the processing by the CPU 101 and GPU 106.
  • DSP Digital Signal Processor
  • a part or all of each function may be realized by processing by a dedicated hardware circuit in place of or in combination with processing by software.
  • the diagnostic imaging apparatus 100A includes an endoscopic image acquisition unit 10A, an estimation unit 20A, and a display control unit 30A.
  • the learning device 40A has a function of generating model data (corresponding to the "learned model" of the present invention) of the convolutional neural network used in the diagnostic imaging device 100A.
  • Endoscopic image acquisition unit The endoscope image acquisition unit 10A acquires the endoscope image data D1 output from the endoscope imaging device 200A. Then, the endoscopic image acquisition unit 10A outputs the acquired endoscopic image data D1 to the estimation unit 20A. When the endoscope image acquisition unit 10A acquires the endoscope image data D1, the endoscope image acquisition unit 10A may directly acquire the endoscope image data D1 from the endoscope image pickup device 200A, or the endoscope image data stored in the external storage device 104. Endoscopic image data D1 provided via D1 or an internet line may be acquired.
  • the estimation unit 20A estimates the presence or absence of multiple iodine unstained bands in the endoscopic image represented by the endoscopic image data D1 output from the endoscopic image acquisition unit 10A using a convolutional neural network. , Output the estimation result. Specifically, the estimation unit 20A estimates the certainty (also referred to as accuracy) of the presence or absence of multiple iodine unstained zones in the endoscopic image. Then, the estimation unit 20A displays the endoscopic image data D1 output from the endoscopic image acquisition unit 10A and the estimation result data D2 representing the estimation result relating to the certainty of the presence or absence of the multiple iodine unstained zone. Output to the control unit 30A.
  • the estimation unit 20A estimates the probability score as an index indicating the degree of certainty of the presence or absence of the multiple iodine unstained zone.
  • the probability score is represented by a value greater than 0 and less than or equal to 1. The higher the probability score, the higher the certainty of the presence or absence of multiple iodine unstained zones.
  • the probability score is an example of an index indicating the degree of certainty of the presence or absence of a multiple iodine unstained zone, and an index of any other aspect may be used.
  • the probability score may be represented by a value of 0% to 100%, or may be represented by any of several levels.
  • a convolutional neural network is a type of feedforward neural network and is based on knowledge in the structure of the visual cortex of the brain. Basically, it has a structure in which a convolution layer responsible for extracting local features of an image and a pooling layer (subsampling layer) that summarizes features for each local area are repeated. According to each layer of the convolutional neural network, it possesses multiple neurons (Neurons), and each neuron is arranged so as to correspond to the visual cortex. The basic function of each neuron consists of signal input and output.
  • FIG. 6 is a diagram showing a configuration of a convolutional neural network according to the present embodiment.
  • the model data (structural data, learned weight parameters, etc.) of the convolutional neural network is stored in the external storage device 104 together with the diagnostic imaging program.
  • the convolutional neural network has, for example, a feature extraction unit Na and an identification unit Nb.
  • the feature extraction unit Na performs a process of extracting an image feature from an input image (specifically, an endoscopic image represented by the endoscopic image data D1).
  • the identification unit Nb outputs an estimation result related to the image from the image features extracted by the feature extraction unit Na.
  • the feature extraction unit Na is configured by hierarchically connecting a plurality of feature amount extraction layers Na1, Na2, and so on.
  • Each feature amount extraction layer Na1, Na2 ... Provides a convolution layer, an activation layer, and a pooling layer.
  • the feature amount extraction layer Na1 of the first layer scans the input image for each predetermined size by raster scanning. Then, the feature amount extraction layer Na1 extracts the feature amount contained in the input image by performing the feature amount extraction process on the scanned data by the convolutional layer, the activation layer and the pooling layer.
  • the feature amount extraction layer Na1 of the first layer extracts a relatively simple single feature amount such as a linear feature amount extending in the horizontal direction and a linear feature amount extending in the diagonal direction.
  • the feature amount sampling layer Na2 of the second layer scans an image (also referred to as a feature map) input from the feature amount sampling layer Na1 of the previous layer at predetermined size intervals by, for example, raster scanning. Then, the feature amount extraction layer Na2 extracts the feature amount contained in the input image by similarly performing the feature amount extraction process by the convolutional layer, the activation layer and the pooling layer on the scanned data.
  • the feature amount extraction layer Na2 of the second layer is integrated with reference to the positional relationship of a plurality of feature amounts extracted by the feature amount extraction layer Na1 of the first layer, so that it is a higher-dimensional complex. Extract features.
  • the feature amount sampling layers after the second layer perform the same processing as the feature amount extraction layer Na2 of the second layer. do. Then, the output of the feature amount sampling layer of the final layer (each value in the map of the plurality of feature maps) is input to the identification unit Nb.
  • the identification unit Nb is composed of, for example, a multi-layer perceptron in which a plurality of fully connected layers (Fully Connected) are hierarchically connected.
  • the fully connected layer on the input side of the identification unit Nb is fully connected to each value in the map of a plurality of feature maps acquired from the feature extraction unit Na, and the product-sum operation is performed while changing the weighting coefficient for each value. Go and output.
  • the fully connected layer of the next layer of the identification unit Nb is fully coupled to the values output by each element of the fully connected layer of the previous layer, and the product-sum operation is performed while applying different weighting factors to each value. Then, in the final stage of the identification unit Nb, a layer (for example, a certainty) that outputs a probability score (certainty) of the presence or absence of multiple iodine unstained bands in the image (endoscopic image) input to the feature extraction unit Na (for example). Softmax function, etc.) is provided.
  • the convolutional neural network is desired from the input endoscopic image by performing learning processing using reference data (hereinafter referred to as "teacher data") marked in advance by an experienced endoscopist.
  • the estimation function can be possessed so that the estimation result (here, the probability score of the presence or absence of the multiple iodine unstained zone) can be output.
  • overfitting can be prevented by covering typical pathological conditions, training with a sufficient amount of teacher data adjusted for bias, and adjusting the weight appropriately.
  • an AI program having generalized performance to the diagnosis of the presence or absence of multiple iodine-unstained zones in this implementation, a program having high-speed and high-precision diagnostic performance becomes possible.
  • the convolutional neural network in the present embodiment takes the endoscopic image data D1 as an input (Input in FIG. 6), and multiple iodine unstaining according to the image characteristics of the endoscopic image represented by the endoscopic image data D1. It is configured to output the probability score of the presence or absence of the band as the estimation result data D2 (Auto in FIG. 6).
  • the convolutional neural network has a configuration in which information related to the subject's age, gender, region, or medical history can be input in addition to the endoscopic image data D1 (for example, as an input element of the identification unit Nb). It may be provided). Since the importance of real-world data in clinical practice is particularly recognized, it is possible to develop a more useful system in clinical practice by adding such information on subject attributes. That is, the characteristics of the endoscopic image are considered to have a correlation with information related to the subject's age, gender, region, medical history, family medical history, etc. By referring to subject attribute information such as age in addition to D1, it is possible to estimate the presence or absence of multiple iodine-unstained zones with higher accuracy. Since the pathophysiology of the disease may differ depending on the region and race, this method should be adopted especially when the present invention is used internationally.
  • the estimation unit 20A performs processing for converting the size and aspect ratio of the endoscopic image, color dividing processing for the endoscopic image, and color conversion processing for the endoscopic image as preprocessing. , Color extraction processing, brightness gradient extraction processing, and the like may be performed. In order to prevent overfitting and improve accuracy, it is also preferable to adjust the weighting.
  • the display control unit 30A superimposes and displays the probability score represented by the estimation result data D2 output from the estimation unit 20A on the endoscope image represented by the endoscope image data D1 output from the estimation unit 20A. A judgment result image is generated. Then, the display control unit 30A outputs the endoscopic image data D1 and the determination result image data D3 representing the generated determination result image to the display device 300A.
  • a digital image processing system such as structural enhancement, color enhancement, difference processing, high contrast, and high definition of the endoscopic image is connected, and processing is performed to help the observer (for example, a doctor) understand and judge. Can also be displayed.
  • the display device 300A superimposes and displays the determination result image represented by the determination result image data D3 on the endoscope image represented by the endoscope image data D1 output from the display control unit 30A.
  • the endoscopic image and the determination result image displayed on the display device 300A are used, for example, for real-time diagnostic assistance and diagnostic support by a doctor.
  • the display control unit 30A controls the display device 300A and causes the screen for displaying the endoscopic image to emit light. Display and output a warning that there is a frequent iodine undyed zone. This can effectively alert the doctor to the presence of multiple iodine-unstained zones in the endoscopic image.
  • the diagnostic imaging apparatus 100A may output a warning by sounding (outputting) a warning sound from a speaker (not shown). Further, at this time, it is also possible to independently calculate and display the determination probability and the estimated probability.
  • the learning device 40A is illustrated so that the convolutional neural network of the estimation unit 20A can estimate the probability score of the presence or absence of multiple iodine unstained zones from the endoscopic image data D1 (specifically, the endoscopic image).
  • the teacher data D4 stored in the external storage device is input, and the learning process is performed on the convolutional neural network of the learning device 40A.
  • the learning device 40A is used by the endoscopic imaging device 200A in a state where the esophagus of a plurality of subjects is irradiated with white light or narrow band light in the endoscopy of the esophagus performed in the past.
  • Learning processing is performed using the captured endoscopic image and the presence / absence of multiple iodine-unstained bands in the endoscopic image determined in advance by iodine staining for confirmation as teacher data D4.
  • the learning device 40A reduces the error (also referred to as loss) of the output data with respect to the correct answer value (presence or absence of multiple iodine unstained bands) when the endoscopic image is input to the convolutional neural network.
  • the error also referred to as loss
  • the correct answer value presence or absence of multiple iodine unstained bands
  • the learning device 40A has an endoscopic image (corresponding to the "unstained band image” of the present invention) that images the esophagus in which the multiple iodine unstained zone actually exists, and an actually multiple iodine unstained band.
  • the learning process is performed using an endoscopic image (corresponding to the "non-stained band image” of the present invention) obtained by imaging the esophagus in which no band exists as the teacher data D4.
  • FIG. 7 is a diagram showing an example of an endoscopic image of the esophagus when iodine solution is sprayed into the lumen of the esophagus.
  • the number of multiple iodine unstained zones existing in the esophagus is 0, and the doctor determines that there are no multiple iodine unstained zones in the endoscopic image (grade A). Will be done.
  • the number of multiple iodine-unstained bands existing in the esophagus is 1 or more and 9 or less, and there are no multiple iodine-unstained bands in the endoscopic image (grade B). Determined by the doctor.
  • the endoscopic image shown in FIG. 7C has 10 or more multiple iodine-unstained zones in the esophagus, and multiple iodine-unstained zones are present in the endoscopic image (grade C). It is judged.
  • the endoscopic image processing device (imaging diagnostic device 100A) driven by a program trained by the teacher data of such a multiple iodine unstained zone can estimate the multiple iodine unstained zone without intentionally staining the iodine. Become.
  • Endoscopic images as teacher data D4 in learning processing mainly use the abundant database of Japan's top-class cancer treatment hospitals, and have abundant diagnosis and treatment experience. Examines all endoscopic images in detail to determine the presence or absence of multiple iodine-unstained zones.
  • the teacher data D4 endoscopic image data
  • the reference data which is the reference data
  • a sufficient number of cases for which image selection and determination of the presence or absence of multiple iodine-unstained zones have been performed is an extremely important process.
  • Such high-precision data cleansing work and use of high-quality reference data provide highly reliable AI program output results.
  • the teacher data D4 of the endoscopic image may be pixel value data or data that has undergone a predetermined color conversion process or the like.
  • a texture feature, shape feature, unevenness condition, spread feature, etc. which are characteristic of the presence or absence of multiple iodine undyed bands, are extracted from a comparison between the unstained band image and the non-stained band image. May be done.
  • the teacher data D4 may perform learning processing in association with information related to the subject's age, gender, region, pre-existing medical history, family medical history, etc., in addition to the endoscopic image data.
  • the algorithm when the learning device 40A performs the learning process may be a known method.
  • the learning device 40A uses, for example, known backpropagation (backpropagation) to perform learning processing on a convolutional neural network and adjust network parameters (weighting coefficient, bias, etc.).
  • backpropagation backpropagation
  • the model data (structural data, learned weight parameters, etc.) of the convolutional neural network subjected to the learning process by the learning device 40A is stored in the external storage device 104 together with the diagnostic imaging program, for example.
  • Examples of known convolutional neural network models include GoogleNet, ResNet, and SENEt.
  • the diagnostic imaging apparatus 100A includes an endoscopic image acquisition unit 10A that acquires an endoscopic image of the esophagus of a subject, and an esophagus in which multiple iodine-unstained zones are present.
  • the image of the esophagus in the multiple iodine-unstained zone and the image of the esophagus in the non-multiple iodine-unstained zone, which is an image of the esophagus in which the multiple iodine-unstained zone does not exist, are trained as teacher data.
  • It includes an estimation unit 20A that estimates the presence or absence of multiple iodine-unstained bands in the acquired endoscopic image using a convolutional neural network that detects bands, and outputs an estimation result. Since the presence of multiple iodine-unstained zones leads to a high risk of cancer, the diagnostic imaging apparatus 100A of the present embodiment can be used for diagnosis while having a risk determination function for esophageal cancer as it is.
  • the convolutional neural network is an endoscopic image of a plurality of esophagus (digestive organs) obtained in advance for each of a plurality of subjects (multiple iodine-unstained band esophagus image, non-multiple iodine-unstained band esophagus image). ) And the definite determination result of the presence or absence of the multiple iodine unstained zone obtained in advance for each of the plurality of subjects. Therefore, it is possible to estimate the presence or absence of multiple iodine-unstained zones in the endoscopic image of the esophagus of a new subject.
  • the diagnostic ability of the endoscopic image possessed by the convolutional neural network according to this embodiment is used, and it is an index of high-risk cases of esophageal cancer. Diagnosis can be made while predicting the presence or absence of an iodine-unstained zone. As a result, high-risk cases of esophageal cancer were identified in advance as well as iodine staining in advance, and esophageal cancer was efficiently and accurately detected without imposing the physical burden of iodine staining on the subjects.
  • the esophagus can be detected by real-time moving image. It is possible to efficiently determine the presence or absence of cancer.
  • the diagnostic imaging apparatus 100A can also be used as a diagnostic support tool that directly supports the diagnosis of endoscopic images by an endoscopist in a laboratory.
  • the diagnostic imaging apparatus 100A can be used as a central diagnostic support service that supports the diagnosis of endoscopic images transmitted from a plurality of laboratories, or can be remotely operated via an Internet line for endoscopic viewing at a remote institution. It can also be used as a diagnostic support service to support the diagnosis of mirror images.
  • the diagnostic imaging apparatus 100A can also be operated on the cloud.
  • these endoscopic images and AI judgment results can be directly converted into a video library and used as teaching materials and materials for education and training and research.
  • the technique of observing at low speed at high risk and at high speed at low risk by determining the low-speed mode and high-speed mode when inserting the endoscope.
  • the function that optimizes the operation on the part of the person facilitates more efficient and highly accurate diagnosis. That is, when inserting an endoscope into the esophagus, the magnitude of the risk of esophageal cancer can be determined from the detection status of multiple iodine-unstained zones.
  • the sensitivity can be displayed on the image device display unit, the operating conditions can be reset, and the diagnosis can be made under conditions suitable for observing the esophageal lumen.
  • the endoscope insertion speed during the examination can output a warning so that the difference between the reference insertion speed and the actual insertion speed becomes small. Appropriate observation conditions are maintained. If multiple iodine-unstained zones are not detected and the cancer risk is low, it is possible to pass through the esophageal lumen quickly, but in that case, lesions that are difficult for the endoscopist to notice are sufficiently detected by a real-time diagnostic imaging device. obtain. On the other hand, if multiple iodine-unstained zones are detected and the cancer risk is high, the endoscopist will observe in detail, and the endoscopist and the real-time diagnostic imaging device will miss the minute cancer lesions. No precise diagnosis can be made.
  • the endoscope can be used in the esophagus without imaging a still image or iodine staining.
  • the degree of esophageal cancer risk can be understood immediately by inserting it, and the accuracy of esophageal cancer risk decreases when the affected area is moved quickly, but the accuracy increases when the movement is slow. It is possible to make an efficient judgment at a speed far exceeding the judgment speed of. As a result, the subject can be examined in the shortest time and with the minimum necessary physical load.
  • each subject can be affected.
  • the endoscopic reference insertion speed that enables observation according to the degree of risk, it is possible to assist in the diagnosis of esophageal cancer efficiently and with high accuracy beyond the conventional technology.
  • first and second embodiments are merely examples of embodiment of the present invention, and the technical scope of the present invention is interpreted in a limited manner by these. It must not be. That is, the present invention can be implemented in various forms without departing from its gist or its main features.
  • the endoscopic image as the teacher data set among the endoscopic images of the subject's esophagus captured by the endoscopic imaging device, the endoscopy in which esophageal cancer is observed (exists). Included images. On the other hand, endoscopic images with widespread mucus, blood, out of focus, or poor image quality due to halation were excluded from the teacher dataset.
  • the instructor of the Japan Gastroenterological Endoscopy Society who is an expert in esophageal cancer, examines and selects the prepared endoscopic images in detail, marks the lesion position of the lesion by precise manual processing, and teaches data. I prepared a set.
  • a 22-layer convolutional neural network is constructed by convolving a GoogleNet that has a sufficient number of parameters and expressive power while having the same structure as the previous convolutional neural network. Used as.
  • the Caffe Deep Learning Framework developed at the Berkeley Vision and Learning Center (BVLC) was used for learning and evaluation testing. All layers of the convolutional neural network are fine-tuned with a global learning rate of 0.0001 using stochastic gradient descent. Each endoscopic image was resized to 224 x 224 pixels for compatibility with convolutional neural networks.
  • a total of 40 endoscopic moving images were collected as an evaluation test data set. An endoscopic moving image showing esophageal cancer and an endoscopic moving image not showing esophageal cancer were taken.
  • As the endoscopic imaging device GIF-H240Z, GIF-H260Z, and GIF-H290 manufactured by Olympus Medical Systems Corporation were used as in the preparation of the teacher data set. For structural enhancement during imaging, A mode level 5 was set when irradiating white light, and B mode level 8 was set when irradiating narrow band light.
  • the data set for the evaluation test includes an endoscopic moving image that meets the eligibility criteria, and an endoscopic moving image taken by an endoscopic imaging device for 5 seconds while gazing at the subject's esophagus as a detailed examination video. Including.
  • an endoscopic moving image in which the endoscope is moved at a low speed (for example, 1 cm / s) to observe the lesion. (Low speed) was imaged.
  • an endoscopic moving image in which an endoscope is quickly inserted at a high speed (for example, 2 cm / s) from the esophageal entrance to the esophagogastric junction was taken.
  • a high speed for example, 2 cm / s
  • evaluation test data are used as endoscopic moving images that meet the exclusion criteria. Excluded from the set.
  • the instructor of the Japan Gastroenterological Endoscopy Society who is an expert on esophageal cancer, examined the prepared endoscopic moving images in detail, and the endoscopic moving images with esophageal cancer and the presence of esophageal cancer are present.
  • a data set for evaluation test was prepared by selecting the endoscopic moving images that were not used.
  • FIG. 8 is a diagram showing the characteristics of the subject and the lesion (esophageal cancer) regarding the endoscopic moving image (low velocity) used in the data set for the evaluation test.
  • Median [whole range] is shown for age and tumor diameter. As shown in FIG. 8, for example, the median tumor diameter was 17 mm.
  • the superficial mucosal layer (EP) had 7 lesions
  • the deep mucosal layer (LPM) had 21 lesions
  • the muscularis mucosae infiltration (MM) was 3 cases
  • SM submucosal infiltration
  • rice field In the macroscopic type (classification), 16 lesions were most frequently depressed type (0-llc).
  • FIG. 9 is a diagram showing the characteristics of the subject and the lesion (esophageal cancer) regarding the endoscopic moving image (high velocity) used in the data set for the evaluation test.
  • Median [whole range] is shown for age and tumor diameter. As shown in FIG. 8, for example, the median tumor diameter was 17 mm.
  • the superficial mucosal layer (EP) had 8 lesions
  • the deep mucosal layer (LPM) had 10 lesions
  • the muscularis mucosae infiltration (MM) was 3 cases
  • SM submucosal infiltration
  • rice field In the macroscopic type (classification), 16 lesions were most frequently depressed type (0-llc).
  • evaluation test method In this evaluation test, the data set for the evaluation test is input to the convolutional neural network-based diagnostic imaging device that has been trained using the teacher data set, and each endoscope that constitutes the data set for the evaluation test. We evaluated whether or not it was possible to correctly diagnose whether or not esophageal cancer was present in the moving image.
  • the diagnostic imaging apparatus diagnoses that a lesion exists in the endoscopic moving image when a predetermined number of endoscopic images having a certainty level of a predetermined value or more exist within a predetermined time.
  • the diagnostic imaging apparatus recognizes a 1-second endoscopic moving image as a 30-frame still image.
  • the diagnostic imaging device recognizes esophageal cancer, it goes back for 0.5 seconds (15 frames) and searches, and if there is an endoscopic image containing esophageal cancer for 3 frames or more, it is included in the endoscopic moving image. Diagnose the presence of esophageal cancer.
  • the diagnostic imaging device correctly confirmed that esophageal cancer was present in the endoscopic moving images taken while irradiating the esophagus of the subject with white light and narrow band light. Whether or not the diagnosis can be made (sensitivity) was calculated using the following equation (1).
  • Sensitivity (Number of endoscopic moving images that could correctly diagnose the presence of esophageal cancer in the evaluation test data set) / (Endoscopic moving images in which esophageal cancer actually exists in the evaluation test data set) Number of) ⁇ ⁇ ⁇ (1)
  • Negative predictive value (NPV) (Number of endoscopic images in which esophageal cancer is not actually present among the endoscopic images diagnosed as having no esophageal cancer in the evaluation test data set) / ( Number of endoscopic moving images diagnosed as having esophageal cancer in the evaluation test data set) ... (4)
  • FIG. 10 is a diagram showing the sensitivity of the diagnostic imaging apparatus in an endoscopic moving image taken in a state where the esophagus of a subject is irradiated with white light and narrow band light, respectively.
  • the diagnostic imaging apparatus refers to 75% (95% CI) of the endoscopic moving images taken while irradiating the esophagus of the subject with white light. I was able to correctly diagnose the presence of esophageal cancer.
  • the diagnostic imaging device has esophageal cancer in 55% (95% CI) of the endoscopic moving images taken while irradiating the esophagus of the subject with narrow band light. I was able to correctly diagnose the existence.
  • the diagnostic imaging apparatus uses the esophagus for 85% (95% CI) of the endoscopic moving images taken while irradiating the esophagus of the subject with white light or narrow band light. I was able to correctly diagnose the presence of cancer.
  • FIG. 11 shows the sensitivity, specificity, and positive predictive value of the diagnostic ability of the diagnostic imaging apparatus in the endoscopic moving image captured by irradiating the esophagus of the subject with white light and narrow band light, respectively. It is a figure showing PPV) and negative predictive value (NPV). As shown in FIG. 11, in the endoscopic moving image taken while irradiating the esophagus of the subject with white light, the sensitivity, specificity, positive predictive value and negative predictive value of the diagnostic imaging apparatus are shown, respectively. , 75%, 30%, 52% and 55%.
  • the sensitivity, specificity, positive predictive value and negative predictive value of the diagnostic imaging apparatus were 55%, respectively. It was 80%, 73% and 64%.
  • AI and endoscopists can diagnose almost all esophageal cancers if the endoscopy insertion speed is as slow as about 1.0 cm / s. However, it is very difficult for the endoscopist to recognize the lesion at a high insertion speed of about 2.0 cm / s.
  • the AI displayed a square frame at the location of esophageal cancer, which slightly improved the endoscopist's recognition of lesions. On the other hand, AI can pick up esophageal cancer with a certain degree of accuracy.
  • Non-Patent Document 3 describes a sensitivity of 77% as a result of evaluating the diagnostic ability of a computer-assisted diagnostic (CAD) system for esophageal cancer using an endoscopic image (still image) taken by an NBI combined magnifying endoscope. , The specificity was 79%, the positive predictive value was 39%, and the negative predictive value was 95%. Examples of causes of false positives include severe shadows, normal structures (esophagogastric junction, left main bronchus, vertebral body), and benign lesions (scar, local atrophy, Barrett's esophagus).
  • CAD computer-assisted diagnostic
  • Non-Patent Document 3 the diagnostic ability of the computer-aided diagnostic system is not compared with the diagnostic ability of an endoscopic expert who has acquired the diagnostic technique for esophageal cancer.
  • the diagnostic difficulty of the endoscopic image used was unknown, and there was a limit to the interpretation of the diagnostic ability of the computer-aided diagnosis system.
  • Non-Patent Document 3 a study using a still image (endoscopic image) is carried out, and although it is useful when performing secondary interpretation of the endoscopic image after endoscopy, a moving image is used. It was difficult to introduce it into the actual medical field where esophageal cancer is diagnosed in real time because it has not been examined in. In order to apply it to real-time moving images, it is necessary to redesign and optimize the AI algorithm separately.
  • the diagnostic imaging apparatus of the present invention compares the diagnostic ability with many endoscopists, it is appropriate to set weights and parameters in the convolutional neural network, and it is difficult to evaluate moving images. It is possible to properly evaluate the degree. It is also possible to adjust to reduce the bias that occurs in comparison with a small number of endoscopists by making comparisons with many endoscopists.
  • the CAD system can provide the performance having a diagnostic ability equal to or higher than that of a skilled doctor. It was shown that it can be used not only in clinical practice but also as an education and training system.
  • the diagnostic ability is high, so that it is highly useful in actual clinical practice.
  • a moving image is used instead of a still image, and an endoscopic diagnosis of esophageal cancer can be performed in real time by using an diagnostic imaging apparatus in clinical practice.
  • Endoscopic imaging devices include high-resolution endoscopes (GIF-H290Z, Olympus Medical Systems Co., Ltd., Tokyo) and high-resolution endoscopic video systems (EVIS LUCERA ELITE CV-290 / CLV-290SL, Olympus Medical Systems). Co., Ltd., Tokyo) was used.
  • GIF-H290Z high-resolution endoscopes
  • EVIS LUCERA ELITE CV-290 / CLV-290SL Olympus Medical Systems
  • Endoscopic images taken in cases with a history of esophagectomy and endoscopic images taken in cases receiving chemotherapy or radiotherapy for the esophagus were excluded from the teacher data set.
  • endoscopic images including esophageal cancer and endoscopic images with poor image quality due to poor insufflation, post-biopsy bleeding, halation, blurring, defocusing, mucus, etc. are also available from the teacher dataset.
  • [Learning / Algorithm] It is composed of 22 layers and has the same structure as the previous convolutional neural network in order to construct an image diagnostic device that estimates the presence or absence of multiple iodine unstained bands in the endoscopic image of the subject's esophagus.
  • GoogleNet which has a sufficient number of parameters and expressive power, was used as a convolutional neural network.
  • the Caffe Deep Learning Framework developed at the Berkley Vision and Learning Center (BVLC) was used for learning and evaluation testing. All layers of the convolutional neural network were fine-tuned with a global learning rate of 0.0001 using stochastic gradient descent. Each endoscopic image was resized to 224 x 224 pixels for compatibility with convolutional neural networks.
  • Endoscopic imaging devices include high-resolution endoscopes (GIF-H290Z, Olympus Medical Systems Co., Ltd., Tokyo) and high-resolution endoscopic video systems (EVIS LUCERA ELITE CV-290 / CLV-290SL, Olympus Medical Systems). Co., Ltd., Tokyo) was used.
  • exclusion criteria for endoscopic images are the same as for the teacher dataset, but basically all of the images taken with white light or narrow band light shining on the esophagus to avoid bias. An endoscopic image was used.
  • the instructor of the Japanese Society of Gastroenterological Endoscopy examined the prepared endoscopic images in detail, determined the presence or absence of multiple iodine-unstained zones, and prepared a data set for evaluation tests.
  • FIG. 12 is a diagram showing an example of an endoscopic image used in the evaluation test data set.
  • FIG. 12A is an image taken by an endoscopic imaging device in a state where the esophagus of the subject is irradiated with white light, and there is actually no multiple iodine-unstained zone in the esophagus (degree of staining when iodine staining is performed: It is an endoscopic image judged to be grade A).
  • FIG. 12A is an image taken by an endoscopic imaging device in a state where the esophagus of the subject is irradiated with white light, and there is actually no multiple iodine-unstained zone in the esophagus (degree of staining when iodine staining is performed: It is an endoscopic image judged to be grade A).
  • FIG. 12A is an image taken by an endoscopic imaging device in a state where the esophagus of the subject is irradiated with white
  • 12B is an image taken by an endoscopic imaging device in a state where the esophagus of the subject is irradiated with narrow band light, and there is actually no multiple iodine-unstained zone in the esophagus (degree of staining when iodine staining is performed). : It is an endoscopic image judged to be grade A).
  • FIG. 12C is an image taken by an endoscopic imaging device in a state where the esophagus of the subject is irradiated with white light, and there is actually no multiple iodine-unstained zone in the esophagus (degree of staining when iodine staining is performed: It is an endoscopic image judged as grade B).
  • FIG. 12D is an image taken by an endoscopic imaging device in a state where the esophagus of the subject is irradiated with narrow band light, and there is actually no multiple iodine-unstained zone in the esophagus (degree of staining when iodine staining is performed). : It is an endoscopic image judged to be grade B).
  • FIG. 12E is an image taken by an endoscopic imaging device in a state where the esophagus of the subject is irradiated with white light, and there are actually multiple iodine-unstained zones in the esophagus (degree of staining when iodine staining is performed: It is an endoscopic image judged as grade C).
  • FIG. 12F is an image taken by an endoscopic imaging device in a state where the esophagus of the subject is irradiated with narrow band light, and there are actually multiple iodine-unstained zones in the esophagus (degree of staining when iodine staining is performed). : An endoscopic image determined to be grade C).
  • FIG. 13 is a diagram showing the characteristics of the subject regarding the endoscopic image used in the evaluation test data set.
  • the median age is shown in FIG.
  • Pearson's chi-square test and Fisher's rigorous test are used to compare various characteristics between subjects who do not actually have multiple iodine-unstained zones in the esophagus and subjects who actually have multiple iodine-unstained zones in the esophagus.
  • Wald's test was used for comparison of observer years (see P value in FIG. 13).
  • the statistically significant difference was set to less than 0.05.
  • “EZR version 1.27 (Saitama Medical Center, Autonomous Medical University)" was used to calculate the P value.
  • evaluation test method In this evaluation test, the data set for the evaluation test is input to the convolutional neural network-based image diagnostic device that has been trained using the teacher data set, and each endoscope that constitutes the data set for the evaluation test. It was evaluated whether or not it was possible to correctly diagnose (determine) whether or not there were multiple iodine-unstained bands in the image.
  • the diagnostic imaging apparatus determines that an endoscopic image having a certainty of presence or absence of a multiple iodine unstained zone has a multiple iodine unstained zone in the endoscopic image, while the multiple iodine unstained band is present.
  • the diagnostic imaging apparatus determines whether or not there is a multiple iodine unstained zone for each endoscopic image, and determines whether or not there is a multiple iodine unstained zone by majority voting of the endoscopic image for each case. Was done.
  • the endoscopist looks at the endoscopic images constituting the evaluation test data set and looks at the endoscopic images. A diagnosis was made as to whether or not multiple iodine-unstained bands were present in the endoscopic image.
  • 10 endoscopists with 8 to 17 years of experience as doctors of the Japan Gastroenterological Endoscopy Society and 3,500 to 18,000 endoscopy cases were selected.
  • the 10 selected endoscopists diagnosed whether or not there was a multiple iodine-unstained zone in each endoscopic image, and the multiple-iodine-stained zone was present in each case by a majority of the endoscopic images. A diagnosis was made as to whether or not to do so.
  • the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and correct diagnosis rate for the diagnostic ability of the diagnostic imaging device (or endoscopist) are calculated by the following formula (5). Calculated using (9).
  • Sensitivity (Number of cases in which it was possible to correctly diagnose the presence of multiple iodine-unstained zones in the esophagus) / (Total number of cases in which multiple iodine-unstained zones actually exist in the esophagus) ...
  • Positive predictive value (PPV) (Among the cases diagnosed as having multiple iodine-unstained zones in the esophagus, the number of cases in which multiple iodine-unstained zones actually exist in the esophagus) / (Multiple iodine-free zones in the esophagus) Number of cases diagnosed as having a dyed band) ⁇ ⁇ ⁇ (7)
  • Negative predictive value (NPV) (Number of cases diagnosed as having no multiple iodine stains in the esophagus, but actually no multiple iodine stains in the esophagus) / (Multiple iodine in the esophagus) Number of cases diagnosed as having no unstained zone) ⁇ ⁇ ⁇ (8)
  • Correct diagnosis rate (number of cases in which it was possible to correctly diagnose whether or not there are multiple iodine-unstained zones in the esophagus) / (number of all cases) ...
  • an experienced endoscopist will be useful in correctly diagnosing the presence of multiple iodine-unstained zones in all endoscopic images that make up the evaluation test data set.
  • Possible background The presence or absence of endoscopic findings in the esophageal mucosa was evaluated, and the presence or absence of multiple iodine-unstained zones in the esophagus was diagnosed by a majority of the endoscopic findings for each endoscopic image. Then, regarding whether or not the presence of multiple iodine-unstained zones in the esophagus can be correctly diagnosed (sensitivity), which is superior between the diagnostic imaging apparatus and the endoscopic findings was compared.
  • FIG. 14 is a diagram showing various endoscopic findings in the endoscopic image.
  • FIG. 14A shows an endoscopic image in which two or more glycogen acanthosis are confirmed in one visual field when the esophagus is irradiated with white light, that is, no endoscopic finding (a) is observed.
  • FIG. 14B shows an endoscopic image in which two or more glycogen acanthosis are confirmed in one visual field when the esophagus is irradiated with narrow band light, that is, no endoscopic finding (a) is observed.
  • FIG. 14C shows an endoscopic image in which keratosis is confirmed when the esophagus is irradiated with white light, that is, endoscopic findings (b) are observed.
  • FIG. 14D shows an endoscopic image in which keratosis is confirmed when the esophagus is irradiated with narrow band light, that is, endoscopic findings (b) are observed.
  • FIG. 14E shows an endoscopic image in which a rough esophageal mucosa is confirmed when the esophagus is irradiated with white light, that is, an endoscopic finding (c) is observed.
  • FIG. 14F shows an endoscopic image in which a rough esophageal mucosa is confirmed when the esophagus is irradiated with narrow band light, that is, an endoscopic finding (c) is observed.
  • FIG. 14G shows an endoscopic image in which vascular fluoroscopy is confirmed when the esophagus is irradiated with white light, that is, no endoscopic finding (d) is observed.
  • FIG. 14H shows an endoscopic image in which a reddish background mucosa is confirmed when the esophagus is irradiated with white light, that is, no endoscopic findings (e) are observed.
  • FIG. 14I shows an endoscopic image in which a brown background mucosa is confirmed when the esophagus is irradiated with narrow band light, that is, an endoscopic finding (f) is observed.
  • FIG. 15 is a diagram showing the sensitivity, specificity, positive predictive value, negative predictive value, and correct diagnosis rate of the diagnostic imaging apparatus and the endoscopist.
  • a bilateral McNemar test was used to compare sensitivity, specificity, and accuracy between diagnostic imaging equipment and endoscopists.
  • the correct diagnosis rate for the presence or absence of multiple iodine-unstained zones was 76.4% for diagnostic imaging equipment and 63.9% for endoscopists.
  • the diagnostic imaging system was significantly more sensitive than 9 out of 10 endoscopists in correctly diagnosing the presence of multiple iodine-unstained zones in the esophagus.
  • FIG. 16 shows the evaluation results of the presence or absence of endoscopic findings on an endoscopic image having multiple iodine-unstained zones by an endoscopist, and an endoscope for an endoscopic image without multiple iodine-unstained zones. It is a figure which shows the evaluation result of the presence or absence of a finding. Pearson's comparison of the number of endoscopic findings evaluated as having findings for each endoscopic finding between an endoscopic image with multiple iodine-unstained zones and an endoscopic image without multiple iodine-unstained zones is described by Pearson. A chi-square test and Fisher's rigorous test were used.
  • endoscopic images in which multiple iodine-unstained zones are present in the esophagus show glycogen acanthosis (less than 2), keratosis, crude esophageal mucosa, loss of vascular see-through, and redness.
  • the number of endoscopic findings of the tone background mucosa and the brown background mucosa was evaluated to be significantly higher than that of endoscopic images in the absence of multiple iodine-unstained zones. That is, if it is evaluated as having endoscopic findings, it is considered that there is a high possibility that multiple iodine-unstained zones are present in the esophagus.
  • FIG. 17 shows the results of comparison between the diagnostic imaging apparatus and endoscopic findings regarding whether or not it is possible to correctly diagnose the presence of multiple iodine-unstained zones in the esophagus with reference to endoscopic images (sensitivity). It is a figure which shows. A two-sided McNemar test was used to compare the sensitivity between the diagnostic imaging system and each endoscopic finding.
  • the diagnostic imaging apparatus is more sensitive than the case where each endoscopic finding is evaluated as having a finding, and among the endoscopic findings, it is evaluated that there is a finding about "disappearance of vascular fluoroscopy". In the case, the sensitivity was the highest.
  • FIG. 18 shows squamous cell carcinoma of the esophagus and squamous cell carcinoma of the head and neck, which were detected as simultaneous / metachronous cancers in cases in which multiple iodine-unstained zones were diagnosed as present (not present) in the esophagus by an diagnostic imaging apparatus. It is a figure which shows the number of epithelial cancer. Pearson's chi-square test and Fisher's rigorous test were used to compare cases diagnosed with multiple iodine-unstained zones and cases diagnosed without multiple iodine-unstained zones.
  • squamous cell carcinoma of the esophagus As described above, for squamous cell carcinoma of the esophagus, squamous cell carcinoma of the esophagus, and squamous cell carcinoma of the head and neck, cases diagnosed as having multiple iodine-unstained zones in the esophagus are located in the esophagus.
  • the incidence of simultaneous / metachronous cancer was significantly higher than in cases diagnosed as having no multiple iodo-squamous zones. Therefore, the diagnostic imaging system not only determines the presence or absence of multiple iodine-unstained zones in the esophagus, but also the risk of developing esophageal squamous cell carcinoma and head and neck squamous cell carcinoma as simultaneous and metachronous cancers. I was able to separate it.
  • the diagnostic imaging apparatus uses the diagnostic ability of the endoscopic image possessed by the convolutional neural network, and in the endoscopic image of the esophagus that has not been iodine-stained, the esophageal squamous epithelial cancer and the head and neck.
  • the presence or absence of multiple iodine-unstained zones which is an index of high-risk cases of partial squamous epithelial cancer, could be diagnosed with higher sensitivity than experienced endoscopists.
  • the iodine staining is usually used only for cancer or lesions suspected of being cancer, and its usefulness is limited. ..
  • the risk of developing esophageal squamous epithelial cancer can be determined from endoscopic images taken without iodine staining in the initial endoscopy (EGD) of all subjects. Can be done.
  • the esophagus and pharynx should be carefully observed under narrow-band light irradiation, and the esophagus should be observed with iodine staining. Ideally, it is not practical to perform the iodine staining in all cases. Iodine staining is used for people with or suspected of having cancer to pick up the cancer without missing it and to diagnose the extent of the cancer. It is also possible to determine the risk of cancer based on the degree of multiple iodine-unstained zones.
  • the presence or absence of 6 endoscopic findings was evaluated in order to diagnose the presence or absence of multiple iodine-unstained zones from the endoscopic image of the esophagus that was not stained with iodine. All of these endoscopic findings are frequently confirmed in cases with multiple iodine-unstained zones.
  • the sensitivities of the two endoscopic findings "less than two glycogen acanthosis in one field of view” and "no vascular fluoroscopy is confirmed when the esophagus is irradiated with white light" are more sensitive than expected.
  • the presence or absence of multiple iodine-unstained zones can be diagnosed from endoscopic images of the esophagus, which is high in water and is not stained with iodine.
  • the endoscopist's sensitivity to correctly diagnosing the presence of multiple iodine-unstained zones was as low as 46.9% (see FIG. 15). It is presumed that the reason is that the above two endoscopic findings were not confirmed by many endoscopists. The other four endoscopic findings were all low in sensitivity.
  • the diagnostic imaging apparatus was more sensitive than each of the six endoscopic findings, and was more sensitive than the experienced endoscopist. In other words, it is suggested that the diagnostic imaging system is superior to the human endoscopist in diagnosing the presence or absence of multiple iodine-unstained zones by comprehensively judging these endoscopic findings. ing.
  • MDV multiple lesions of dilated blood vessels
  • the present inventor has a risk of developing multiple iodine-unstained zones from endoscopic images of the esophagus that are not stained with iodine, that is, squamous cell carcinoma of the esophagus and squamous cell carcinoma of the head and neck.
  • iodine that is, squamous cell carcinoma of the esophagus and squamous cell carcinoma of the head and neck.
  • the present invention is useful as a diagnostic imaging device, a diagnostic imaging method, a diagnostic imaging program, and a trained model capable of improving the diagnostic accuracy of esophageal cancer in esophageal endoscopy.

Landscapes

  • Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Surgery (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Biomedical Technology (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Pathology (AREA)
  • Physics & Mathematics (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Veterinary Medicine (AREA)
  • Optics & Photonics (AREA)
  • Biophysics (AREA)
  • Epidemiology (AREA)
  • Primary Health Care (AREA)
  • Databases & Information Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Quality & Reliability (AREA)
  • Theoretical Computer Science (AREA)
  • Gastroenterology & Hepatology (AREA)
  • Data Mining & Analysis (AREA)
  • Signal Processing (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Image Analysis (AREA)
  • Endoscopes (AREA)
  • Image Processing (AREA)
PCT/JP2021/015555 2020-04-27 2021-04-15 画像診断装置、画像診断方法、画像診断プログラムおよび学習済みモデル Ceased WO2021220822A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US17/997,028 US20230255467A1 (en) 2020-04-27 2021-04-15 Diagnostic imaging device, diagnostic imaging method, diagnostic imaging program, and learned model
CN202180030877.3A CN115460968A (zh) 2020-04-27 2021-04-15 图像诊断装置、图像诊断方法、图像诊断程序和学习完毕模型
JP2022517627A JP7550409B2 (ja) 2020-04-27 2021-04-15 画像診断装置、画像診断方法、および画像診断プログラム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020078601 2020-04-27
JP2020-078601 2020-04-27

Publications (1)

Publication Number Publication Date
WO2021220822A1 true WO2021220822A1 (ja) 2021-11-04

Family

ID=78332385

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/015555 Ceased WO2021220822A1 (ja) 2020-04-27 2021-04-15 画像診断装置、画像診断方法、画像診断プログラムおよび学習済みモデル

Country Status (4)

Country Link
US (1) US20230255467A1 (https=)
JP (1) JP7550409B2 (https=)
CN (1) CN115460968A (https=)
WO (1) WO2021220822A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025027815A1 (ja) * 2023-08-02 2025-02-06 オリンパスメディカルシステムズ株式会社 内視鏡診断支援方法、推論モデル、内視鏡画像処理装置、内視鏡画像処理システムおよび内視鏡画像処理プログラム
WO2025057404A1 (ja) * 2023-09-15 2025-03-20 オリンパスメディカルシステムズ株式会社 情報処理装置、内視鏡システム、及び処理方法

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020194568A1 (ja) * 2019-03-27 2020-10-01 Hoya株式会社 内視鏡用プロセッサ、情報処理装置、内視鏡システム、プログラム及び情報処理方法
JP7377769B2 (ja) * 2020-06-08 2023-11-10 Hoya株式会社 プログラム、情報処理方法及び情報処理装置
EP4024405A1 (en) * 2020-12-29 2022-07-06 Koninklijke Philips N.V. An apparatus and method for moderating a skin lesion detection model
US12062169B2 (en) * 2022-04-25 2024-08-13 Hong Kong Applied Science and Technology Research Institute Company Limited Multi-functional computer-aided gastroscopy system optimized with integrated AI solutions and method
US12551090B2 (en) 2022-07-28 2026-02-17 Nec Corporation Endoscopic examination support apparatus, endoscopic examination support method, and recording medium
CN118314202B (zh) * 2022-12-29 2026-01-09 中元汇吉生物技术股份有限公司 成像位置判定方法、装置、终端设备及存储介质
EP4497368A1 (en) * 2023-07-25 2025-01-29 Olympus Medical Systems Corporation Endoscopic imaging manipulation method and system
CN116703798B (zh) * 2023-08-08 2023-10-13 西南科技大学 基于自适应干扰抑制的食管多模态内镜图像增强融合方法
US12597506B2 (en) * 2023-12-05 2026-04-07 Nec Corporation Endoscopic examination support apparatus, endoscopic examination support method, and recording medium
CN117934519B (zh) * 2024-03-21 2024-06-07 安徽大学 一种非配对增强合成的食管肿瘤ct图像自适应分割方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017216878A1 (ja) * 2016-06-14 2017-12-21 オリンパス株式会社 内視鏡装置
WO2018216618A1 (ja) * 2017-05-25 2018-11-29 日本電気株式会社 情報処理装置、制御方法、及びプログラム
WO2018221033A1 (ja) * 2017-06-02 2018-12-06 富士フイルム株式会社 医療画像処理装置、内視鏡システム、診断支援装置、並びに医療業務支援装置
WO2018225448A1 (ja) * 2017-06-09 2018-12-13 智裕 多田 消化器官の内視鏡画像による疾患の診断支援方法、診断支援システム、診断支援プログラム及びこの診断支援プログラムを記憶したコンピュータ読み取り可能な記録媒体
WO2019088121A1 (ja) * 2017-10-30 2019-05-09 公益財団法人がん研究会 画像診断支援装置、資料収集方法、画像診断支援方法および画像診断支援プログラム

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10376181B2 (en) * 2015-02-17 2019-08-13 Endochoice, Inc. System for detecting the location of an endoscopic device during a medical procedure
CN206462964U (zh) * 2016-08-15 2017-09-05 承德市中心医院 一种内窥镜检查用自动碘染装置
US11175296B2 (en) * 2016-10-26 2021-11-16 Washington University Methods of diagnosing and treating cancer comprising ME1
WO2019145951A1 (en) * 2018-01-23 2019-08-01 Mobileodt Ltd. Automated monitoring of medical imaging procedures
WO2020256568A1 (en) * 2019-06-21 2020-12-24 Augere Medical As Method for real-time detection of objects, structures or patterns in a video, an associated system and an associated computer readable medium
CN110201190B (zh) * 2019-06-27 2021-12-14 施瑞华 一种用于食管癌早期诊断的染色剂及其制备方法
CN110495847B (zh) * 2019-08-23 2021-10-08 重庆天如生物科技有限公司 基于深度学习的消化道早癌辅助诊断系统和检查装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017216878A1 (ja) * 2016-06-14 2017-12-21 オリンパス株式会社 内視鏡装置
WO2018216618A1 (ja) * 2017-05-25 2018-11-29 日本電気株式会社 情報処理装置、制御方法、及びプログラム
WO2018221033A1 (ja) * 2017-06-02 2018-12-06 富士フイルム株式会社 医療画像処理装置、内視鏡システム、診断支援装置、並びに医療業務支援装置
WO2018225448A1 (ja) * 2017-06-09 2018-12-13 智裕 多田 消化器官の内視鏡画像による疾患の診断支援方法、診断支援システム、診断支援プログラム及びこの診断支援プログラムを記憶したコンピュータ読み取り可能な記録媒体
WO2019088121A1 (ja) * 2017-10-30 2019-05-09 公益財団法人がん研究会 画像診断支援装置、資料収集方法、画像診断支援方法および画像診断支援プログラム

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025027815A1 (ja) * 2023-08-02 2025-02-06 オリンパスメディカルシステムズ株式会社 内視鏡診断支援方法、推論モデル、内視鏡画像処理装置、内視鏡画像処理システムおよび内視鏡画像処理プログラム
WO2025057404A1 (ja) * 2023-09-15 2025-03-20 オリンパスメディカルシステムズ株式会社 情報処理装置、内視鏡システム、及び処理方法

Also Published As

Publication number Publication date
CN115460968A (zh) 2022-12-09
JP7550409B2 (ja) 2024-09-13
US20230255467A1 (en) 2023-08-17
JPWO2021220822A1 (https=) 2021-11-04

Similar Documents

Publication Publication Date Title
JP7550409B2 (ja) 画像診断装置、画像診断方法、および画像診断プログラム
JP7335552B2 (ja) 画像診断支援装置、学習済みモデル、画像診断支援装置の作動方法および画像診断支援プログラム
de Groof et al. Deep learning algorithm detection of Barrett’s neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video)
US20250143543A1 (en) Method for Real-Time Detection of Objects, Structures or Patterns in a Video, an Associated System and an Associated Computer Readable Medium
Ohmori et al. Endoscopic detection and differentiation of esophageal lesions using a deep neural network
Nakagawa et al. Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists
Pan et al. Bleeding detection in wireless capsule endoscopy based on probabilistic neural network
JP5242381B2 (ja) 医療用画像処理装置及び医療用画像処理方法
JP2021112593A (ja) 消化器官の内視鏡画像による疾患の診断支援方法、診断支援システム、診断支援プログラム及びこの診断支援プログラムを記憶したコンピュータ読み取り可能な記録媒体
US20230162356A1 (en) Diagnostic imaging device, diagnostic imaging method, diagnostic imaging program, and learned model
WO2021054477A2 (ja) 消化器官の内視鏡画像による疾患の診断支援方法、診断支援システム、診断支援プログラム及びこの診断支援プログラムを記憶したコンピュータ読み取り可能な記録媒体
CN115018767B (zh) 基于本征表示学习的跨模态内镜图像转换及病灶分割方法
KR102637484B1 (ko) 인공지능 기반의 내시경 진단 보조 시스템 및 이의 제어방법
Xu et al. Upper gastrointestinal anatomy detection with multi‐task convolutional neural networks
CN117854705A (zh) 上消化道多病变多任务智能诊断方法、装置、设备及介质
Babu et al. Gastrointestinal tract disease detection via deep learning based Duo-Feature Optimized Hexa-Classification model
JP2023079866A (ja) 超拡大内視鏡による胃癌の検査方法、診断支援方法、診断支援システム、診断支援プログラム、学習済みモデル及び画像診断支援装置
US12562279B2 (en) Portable edge AI-assisted diagnosis and quality control system for gastrointestinal endoscopy
Katayama et al. Development of Computer-Aided Diagnosis System Using Single FCN Capable for Indicating Detailed Inference Results in Colon NBI Endoscopy
WO2023212595A1 (en) System having a multi-ordered classification modules
Bhargavi et al. Exploring Filters in Color Segmentation for Detection of Duodenal Cysts in Wireless Capsule Endoscopy Images
Einlou et al. Style-enhanced CycleGAN for unpaired image-to-image translation: A new approach to medical image fusion refinement
JP2025067633A (ja) 画像処理装置、画像処理方法、及び電子内視鏡システム
Photiou et al. Classifying Pre-Malignant Colon Polyps Using Hybrid Deep Learning on Ex Vivo Optical Coherence Tomography Images
KR20250152725A (ko) 인공지능 기반 내시경 병변 통합 탐지 시스템

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21796898

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022517627

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21796898

Country of ref document: EP

Kind code of ref document: A1