WO2023281738A1 - 情報処理装置および情報処理方法 - Google Patents

情報処理装置および情報処理方法 Download PDF

Info

Publication number
WO2023281738A1
WO2023281738A1 PCT/JP2021/025970 JP2021025970W WO2023281738A1 WO 2023281738 A1 WO2023281738 A1 WO 2023281738A1 JP 2021025970 W JP2021025970 W JP 2021025970W WO 2023281738 A1 WO2023281738 A1 WO 2023281738A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
inference model
images
endoscope
timing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/025970
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
浩一 新谷
憲 谷
修 野中
学 市川
智子 後町
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Olympus Corp
Original Assignee
Olympus Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Olympus Corp filed Critical Olympus Corp
Priority to JP2023533019A priority Critical patent/JP7688704B2/ja
Priority to PCT/JP2021/025970 priority patent/WO2023281738A1/ja
Priority to CN202180098769.XA priority patent/CN117480572A/zh
Publication of WO2023281738A1 publication Critical patent/WO2023281738A1/ja
Priority to US18/537,951 priority patent/US12592062B2/en
Anticipated expiration legal-status Critical
Priority to JP2025086428A priority patent/JP2025119024A/ja
Priority to US19/557,581 priority patent/US20260196029A1/en
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06V10/7753—Incorporation of unlabelled data, e.g. multiple instance learning [MIL]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/776—Validation; Performance evaluation
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/10—Recognition assisted with metadata

Definitions

  • the present invention annotates an image acquired by an inspection device such as an endoscope, creates teacher data, and efficiently and effectively utilizes the acquired image data when generating an inference model using this teacher data.
  • the present invention relates to an information processing apparatus and an information processing method capable of
  • an inference model (AI model)
  • a signal analysis system and method have been proposed that first collect raw data, extract and interpret data features from this raw data for inference model generation (Patent Reference 1). That is, in this signal analysis system, the input is raw signal data, the origin of features is traced to the signal data, and mapped to domain/application knowledge. Features are then extracted by using deep learning networks, machine learning models are implemented for sensor data analysis, and causality analysis is performed for failure prediction.
  • Patent Document 1 In the signal analysis system described in Patent Document 1 mentioned above, the classification is continuously performed according to the instruction of the expert, and the data can be used effectively.
  • the sensors for generating raw data have different characteristics for each sensor. Although the difference in characteristics of each sensor, etc., must be considered, this point is not taken into consideration in Patent Document 1 at all. For this reason, in Patent Document 1, it is difficult to make annotations of the same quality for each model when the models are different.
  • the present invention has been made in view of such circumstances, and when data such as sensors with different characteristics are input, or when data that can be said to be an unknown situation is input, an inference model can be efficiently generated. It is an object of the present invention to provide an information processing apparatus and an information processing method capable of generating
  • an information processing apparatus provides the first endoscope for determining a specific image feature for images continuously obtained from a first endoscope. It is determined whether the group of images obtained in time series by the scope is the group of images obtained at the first timing or the group of images obtained at the second timing, and at the second timing.
  • an information processing device capable of cooperating with a learning unit for obtaining a first inference model for image feature determination of the first endoscopic image learned by using a result of annotating an obtained image group as teacher data, Using the image group obtained at the first timing when the first inference model was created, among the newly acquired image group from the first or second endoscope, a teacher data candidate and a classifying unit for classifying the image group.
  • An information processing apparatus is the information processing apparatus according to the first aspect, wherein an image selected from a group of images serving as a candidate for teaching data by the classifying unit is used as teacher data, and a specification similar to that of the first inference model is provided. However, it has a second inference model corresponding to the image acquired by the second endoscope.
  • An information processing apparatus uses an existing inference model with similar specifications using teacher data based on a group of images obtained in time series by the first endoscope Unnecessary images are those that were not annotated when the was created.
  • An information processing apparatus is the image group obtained at the first timing, which is acquired for creating the first inference model in the first aspect, and the image group obtained at the second timing. A group of obtained images is recorded in the recording unit.
  • An information processing apparatus is the information processing apparatus according to the first aspect, wherein the first inference model is improved for the image group according to the reliability of the result of inferring the same image with the first inference model. 3, further comprising a first requesting unit for requesting annotations for training data for generating the inference model.
  • An information processing apparatus according to a sixth aspect is the information processing apparatus according to the first aspect, wherein the group of images acquired from the second endoscope and classified is used as an inference model generation for the second endoscope. Classifying and annotating the classified image group to obtain an inference model having the same specifications as the first inference model but corresponding to the image acquired by the second endoscope. and a second requesting unit for requesting.
  • An information processing apparatus determines that the image is from the second endoscope according to the result of inputting the image to the first inference model.
  • An input determination unit is further provided.
  • An information processing apparatus is the information processing apparatus according to the first invention, further comprising an appending unit for appending metadata to each image forming the image group.
  • An information processing apparatus is the information processing apparatus according to the first aspect, wherein the classification unit uses the image group obtained at the first timing when the first inference model is created, and The features are determined, and the images acquired from the second endoscope different from the first endoscope are classified as annotation candidate images.
  • An information processing apparatus according to a tenth aspect is the information processing apparatus according to the first aspect, wherein the second timing is determined according to an input/output relationship for which an inference model is obtained.
  • An information processing apparatus provides existing teacher data obtained by the first endoscope to determine a specific image feature for an image obtained by the first endoscope.
  • an information processing apparatus having an inference engine that sets a first inference model for image feature determination of the first endoscopic image learned by using a result of annotating a group of endoscopic inspection images as teacher data, 1 image when the inference model was created, and was acquired from a second endoscope different from the first endoscope, including a group of images that were not used as the teacher data.
  • a classification unit for classifying inspection image groups is provided.
  • An information processing apparatus according to a twelfth aspect of the invention, in the eleventh aspect, has a second requesting unit that learns a result of annotating the inspection image group as teacher data and requests generation of a second inference model.
  • An information processing apparatus is a learning unit that creates the first inference model for the endoscope by learning, as teacher data, an image group included in an inspection video obtained from the endoscope.
  • a recording unit that records at least a part of the inspection moving image together with the result of classifying the group of images when the teacher data is selected from the image frames included in the inspection moving image according to the time information of the moving image frame; , a video acquisition unit that acquires a new inspection video during learning for creating a second inference model different from the first inference model; and a classification recorded in the recording unit from the new inspection video.
  • a selection unit that selects a new training data candidate according to the image group.
  • An information processing apparatus is the information processing apparatus according to the thirteenth invention, wherein the first inference model is an inference model for displaying an inference result for the image obtained by the endoscope.
  • An information processing apparatus is the information processing apparatus according to the thirteenth aspect, wherein at least part of the inspection moving image includes an annotation candidate image.
  • An information processing method provides images continuously obtained from the first endoscope in time series by the first endoscope in order to determine a specific image feature. It is determined whether the obtained image group is the image group obtained at the first timing or the image group obtained at the second timing, and the image group obtained at the second timing is determined.
  • the first inference model The newly acquired image group from the first or second endoscope is classified by using the image group obtained at the first timing when creating .
  • a learning method comprises a dividing step of dividing an inspection moving image obtained from a first endoscope according to timing, and a frame corresponding to the first timing among the divided timings.
  • an image classification step of removing from teaching data candidates and adopting frames corresponding to a second timing as teaching data candidates; a recording step of classifying and recording the inspection moving images for each division timing; using at least one to learn a first inference model for inferring affected part information included in the frames of the inspection moving image.
  • a learning method is the learning method according to the seventeenth aspect, further comprising an annotation step of using a result of annotating a frame corresponding to the second timing as the teacher data candidate.
  • a learning method is the learning method according to the seventeenth invention, when creating a second inference model different from the first inference model, acquiring a new inspection video; and a selection step of selecting a training data candidate for the second inference model by classifying the inspection moving image into a classification reflecting the image classification result.
  • An inference model creation method is obtained by, for endoscopic video information obtained during a specific examination, using frames corresponding to teacher images in the endoscopic video as annotation information.
  • the video information is used as training data, and the endoscopic video images of multiple inspection results are converted into training data in the same way. Create an inference model by training so that is the output.
  • an information processing apparatus capable of efficiently generating an inference model when inputting data from sensors having different characteristics, or when inputting data in an unknown situation, and An information processing method can be provided.
  • FIG. 1 is a block diagram showing the configuration of an information processing system and its peripheral systems according to an embodiment of the present invention
  • FIG. 1 is a block diagram showing the configuration of part of an information processing system and its peripheral systems according to an embodiment of the present invention
  • FIG. 4 is a diagram showing the relationship between image data and inference model generation in the information processing system and its peripheral systems according to an embodiment of the present invention
  • FIG. 4 is a diagram showing a flowchart showing the operation of creating a new inference model and an example of an input image in the information processing system and its peripheral systems according to an embodiment of the present invention
  • 4 is a flowchart showing a modification of the operation of creating a new inference model in the information processing system and its peripheral systems according to one embodiment of the present invention
  • 4 is a diagram showing a case where an existing inference model can correctly infer an input image and a case where the existing inference model cannot correctly output an input image in an information processing system and its peripheral systems according to an embodiment of the present invention; be.
  • 4 is a flow chart showing the operation of the endoscope 1 in the information processing system and its peripheral systems according to one embodiment of the present invention.
  • 4 is a flow chart showing the operation of the information processing system in the information processing system and its peripheral systems according to one embodiment of the present invention.
  • 4 is a flow chart showing the operation of the endoscope 2 in the information processing system and its peripheral systems according to one embodiment of the present invention.
  • FIG. 4 is a flow chart showing the operation of determining whether the information processing system and its peripheral systems according to one embodiment of the present invention are an improved current model, a new model, or other.
  • 4 is a flow chart showing the operation of the learning device in the information processing system and its peripheral systems according to one embodiment of the present invention.
  • FIG. 4 is a diagram illustrating generation of an inference model for inferring an image to be annotated and inference using the generated inference model in an information processing system and its peripheral systems according to an embodiment of the present invention;
  • the information processing device and information processing method according to the present application efficiently collect new training data when creating a new or improved inference model. For this reason, when creating an inference model that has already been proven, the characteristics of the past training data used in machine learning are used. The past training data used to create proven inference models were often created with a great deal of effort. This is because it has already required a great deal of labor to select an image containing an object to be detected from among a myriad of images.
  • frames for teacher data are used. Since it is desirable to narrow down the number of image frames that are too large for annotation, the annotation image frames are selected prior to annotation, following the creation history of the model inference model.
  • annotation candidates among the newly obtained sequential images for a new (or improved) inference model A step of forming an image is provided (see, for example, S5 and S7 in FIG. 3A(a), S103 and S107 in FIG. 3B, and S29 and S39 in FIG. 5B). If the inference model has a proven specification, it will also have a proven track record for training data. Therefore, if you create something with similar specifications, you can imitate and select image frames as training data candidates to reduce unnecessary work.
  • Inference models are used in a variety of situations. For example, in the case of an endoscope, an inference model that performs navigation display, etc. in the process of guiding to access somewhere, and a lesion, etc., are not overlooked. Therefore, an inference model for calling attention to the doctor, an inference model for guiding how to observe when observing what seems to be a lesion, and an inference model for distinguishing what it is and various specifications. In the following examples, inference models for discrimination and observation will be mainly explained, but it goes without saying that the inference models for other specifications can also be applied. In any case, based on a video of a series of examinations and treatments, the necessary frames are selected and turned into teaching data.
  • the number of frames will be a huge 36,000 frames.
  • the access images from the body cavity to the affected area inside the gastrointestinal tract remain unused, and these images require a huge number of frames. number.
  • the image of the diseased part is searched for from the remaining images from which such portions have been removed.
  • the above is an example of creating an inference model that checks the affected area, distinguishes what kind of lesion it is, and detects the extent of the spread of the affected area.
  • an inference model that correctly guides the endoscope operation
  • an inference model for oversight prevention for finding an easily overlooked affected area.
  • images of scenes that should guide the operation of the endoscope, images that are easy to miss, and the like are training data.
  • target images which are candidates for training data corresponding to the important scene, from among tens of thousands of frames, and ask experts to annotate them. It is not efficient to do so.
  • An inference model can be created efficiently by referring to the method of image selection used when creating an inference model of specifications. Images obtained by using an endoscope are collected through a similar process for each examination. is not given to all the frames of the endoscopic image, if there is information on the results selected in the process of creating a large number of training data, this information can be diverted and what kind of image can be used as training data.
  • an inference model that makes inferences using the image that has become training data and the entire endoscopic image.
  • annotation information is added to the frame corresponding to the teacher image in the endoscopic video information obtained during a specific examination.
  • Video information is used as training data.
  • annotations are similarly added to endoscopic moving images (which may include inserted or removed images) of a plurality of inspection results to create teacher data.
  • an inference model can be created by training such that the input is an endoscopic video including insertion or removal images and the output is a teacher data frame.
  • the training data image to be acquired may be omitted in the following two situations. That is, in the following two situations, it is assumed that training data candidates may be overlooked.
  • the assumed endoscope one that has already created an inference model
  • endoscopic images other than normal patterns, such as those accompanying such patterns there is a possibility that they may not be correctly inferred as teacher data because the obtained images and the manner of observation and operation are different.
  • teacher data cannot be inferred correctly for an endoscope device that differs from an assumed endoscope device in operation method, image quality, and the like.
  • This system is originally a system for creating and improving an inference model for an inspection device such as a first endoscope (for example, an existing endoscope). It also manages teaching data when creating inference models for inspection equipment. When improving the inference model, creating an inference model with other specifications, or clarifying the history of the inference model, the training data and the endoscopic images that are the basis of the training data are used. This is because management is preferable. However, it is a system that considers not only the improvement of the existing model but also the creation of an inference model for an inspection device such as a second endoscope that is different from the inspection device such as the first endoscope. be.
  • the inference model for the inspection device such as the first endoscope created here is based on whether or not there is a lesion such as cancer in the image obtained from the inspection device such as the first endoscope. (See S25 in FIG. 5B, for example).
  • the teacher data and the endoscopic images that are the basis of the teacher data are managed.
  • the images obtained during the endoscopic examination are recorded in a format such as a moving image or, as appropriate, in a format such as a still image taken by the doctor during the inspection. Since the images are obtained in the process of inserting an endoscope into a body cavity, performing an examination, and removing the endoscope from the body cavity, a moving image consisting of a sequence of image frames lasting several minutes or more, or an accompanying still image, It is documented for each patient and case.
  • This movie contains a group of images in chronological order at the time of insertion, inspection, and withdrawal. There is an image group that seems to be That is, it explains that the image groups of image sequences along various time axes are divided into a plurality of image groups.
  • the plurality of image groups may be most simply referred to as first and second image groups.
  • the group of images obtained in chronological order by the first endoscope is regarded as the group of images obtained at the first timing (unnecessary images during access).
  • a group of images obtained at the second timing may be used.
  • the image groups at the time of insertion and at the time of withdrawal may be used as the third and fourth image groups.
  • the group of inserted images and the group of extracted images may be considered to be screened and included in the first timing.
  • the image at the second timing includes the affected area image.
  • the group of images at the first timing may be set as the group of images at the second timing. may be reclassified into the image group of The first timing and the second timing do not divide the time-series image group into two at a specific point in time, but the first timing here, the second timing here, and the first timing again. , etc., in the time series and time axis, these are appropriately divided into multiple timings.
  • the information system also cooperates with a learning unit (for example, the learning unit 18 in FIG. 1B) that has a track record of creating inference models that have already been created.
  • This inference model is a first inference model for image feature determination of the first endoscopic image. Learning is performed using the results of annotation as training data.
  • this information processing system Since this information processing system has a track record of creating an inference model for the first endoscope, it will take a long time to create an inference model for other types of endoscopes. be. Therefore, by using a group of images suitable for creating the first inference model as necessary, the inference model can be used for a completely unknown second endoscope that is different from the first endoscope.
  • the inference model may have a function of collecting and managing training data, and a classification unit that classifies images so that training data can be efficiently created from the group of images acquired from the second endoscope. (for example, the classification unit 15).
  • images can be sorted from a second endoscope that is different from the first endoscope. Therefore, even if the second endoscope is an unknown endoscope different from the first endoscope, the information processing system generates an inference model for the second endoscope. It is possible to efficiently collect images suitable for That is, efficiency is improved by applying teacher data, technology, logic, know-how, etc., when an inference model was created for the first endoscope. In other words, when creating the training data for the first endoscope, we extracted the training frames from the large amount of video frames collected for each examination, and created an inference model using the training data. There is a track record.
  • an inference model for the second endoscope is created in the same way as when the inference model was created with the first endoscope.
  • Frames that can be used as teacher data can be selected from the obtained moving images for creation (learning the teacher data).
  • the input information for learning is a group of endoscopic image frames including inserted or removed images, and the output information is frames adopted as teaching data, and learning is performed to create an inference model. Once the inference model is created, the group of inspection images from the second endoscope is input to the inference model, and as a result of inference, training data candidates are output.
  • FIG. 1A and 1B are block diagrams showing configurations of an information processing system and its peripheral systems according to an embodiment of the present invention.
  • This system has an information processing system 10 , an image system 20 and a recording section 30 .
  • FIG. 1A shows the overall system
  • FIG. 1B shows the internal configurations of the control unit 11 and the recording unit 30 in the information processing system 10.
  • the imaging system 20 is included in an inspection apparatus such as an endoscope, and acquires image data such as an endoscopic image.
  • the information processing system 10 acquires the image data acquired by the image system 20 , and the acquired image data is recorded in the recording unit 30 .
  • the acquired image data is annotated to create training data, and the training data is used to generate an inference model.
  • the information processing system 10 is provided in one or more servers or the like.
  • a recording unit 30 is also provided in the server or the like.
  • the information processing system 10 and the recording unit 30 may be provided in the same server, or may be distributed.
  • the learning unit 18 is provided inside the information processing system 10 in this embodiment, it may be provided outside the information processing system 10 .
  • the image system 20, the information processing system 10, and the recording unit 30 can be connected via a communication network such as the Internet or an intranet, and can perform data communication.
  • the image system 20 is provided in an inspection device such as an endoscope, as described above.
  • This imaging system 20 has a control section 21 , a treatment section 22 , an image acquisition section 23 and a display section 24 .
  • the image system 20 has a communication section (having a communication circuit) in addition to the above-described sections, and can perform transmission/reception communication with the information processing system 10 and the like through the communication section.
  • the control unit 21 controls the image system 20 as a whole.
  • the control unit 21 is composed of one or a plurality of processors having a processing device such as a CPU and a memory storing a program, and can control each unit in the image system 20 by executing the program.
  • the treatment section 22 is inserted into the inside of the body, etc., and performs treatment for observing the inside.
  • Industrial endoscopes and medical endoscopes are basically imaging devices that are inserted into something to observe the inside. There are devices and mechanisms.
  • the treatment section 22 is equipped with a dedicated tube, etc., for adjusting the position and orientation of the imaging section provided at the distal end of the endoscope, and requiring functions such as water injection and suction as necessary. , which satisfies these functional requirements. It also drives treatment tools (biopsy forceps, snares, high-frequency knives, injection needles, etc.) for biopsy and treatment. Functions such as drug administration are also included here.
  • the treatment section 22 has an illumination section and the like, and the illumination section irradiates the inside, and also switches the light source characteristics (wavelength, etc.) to obtain a reflection image of the surface of the target object and the target at a deeper position. is available. Moreover, even when it has a function that enables staining and fluorescence observation, the treatment section 22 may have a corresponding function.
  • the image acquisition unit 23 includes a photographing lens, an imaging device, an imaging control circuit, an image processing circuit, etc., and acquires image data. During examination, the tip of the endoscope is inserted into the body, and the image acquisition unit 23 can acquire image data of the inside of the body, for example, the esophagus, stomach, and intestines.
  • the image acquisition unit 23 may have functions for zooming, close-up photography, and 3D information acquisition.
  • the image acquisition unit 23 may have a plurality of imaging units, and as described above, in cooperation with switching of irradiation light, etc., images with different characteristics are acquired even in the same part, and various images are obtained. Analysis may be possible.
  • the image acquisition unit 23 can continuously acquire images when the distal end of the endoscope is inserted into the body. Endoscopy, which observes the inside of the human body along the gastrointestinal tract, in particular, is inserted through a body cavity such as the oral cavity, nasal cavity, or anus. Various parts are inspected, and the endoscope is taken out from the place where it was last inserted, and the inspection ends. Therefore, in endoscopy, very similar images are obtained for each target examination. It can be said that driving along a course while checking the imaging result is a technique similar to driving a car, but the points described above are significantly different. Therefore, the image acquisition unit 23 can continuously acquire images from the time when the distal end of the endoscope is inserted into the body to the end of the examination until the endoscope is pulled out from the same place. .
  • input images P1, P2, and P3 are data of continuously acquired images (expressed as moving images, and each frame constituting them may be used).
  • the input image SP is image data of a so-called still image, which is one frame in a moving image, or is intentionally acquired for recording by a doctor or a medical worker.
  • a still image which is one frame in a moving image, or is intentionally acquired for recording by a doctor or a medical worker.
  • videos are used for inference, and each frame of the video is input to the inference model, so there is no need to dare to say that it is a still image, but it is easy to understand.
  • the category information Pc (a kind of annotation described above) is, for example, information about the device that acquired the image, such as the endoscope manufacturer and model name.
  • Each of the images P1 to P3 may also be provided with category information, or a plurality of image groups may be filed and the category information may be associated with this image file.
  • the image quality differs depending on the sensor, image processing circuit, etc. in the image acquisition unit 23, but if this category information Pc is added to the image data, it is easy to know what kind of image quality characteristics it has.
  • a blank space under the category information Pc is an area for recording other annotations as metadata.
  • the display unit 24 has a display and the like, and can display the endoscopic image acquired by the image acquisition unit 23 .
  • the display content 25 inferred by the inference engine 19 in the information processing system 10, that is, the inference result 25a and the reliability 25b are also displayed.
  • the inference result 25a the endoscopic image acquired by the image acquisition unit 23 is displayed so that the position of the lesion such as cancer can be seen.
  • the information processing system 10 has an inference engine 19 having a control unit 11 and an existing inference model 19a.
  • the control unit 11 controls the entire information processing system 10 .
  • the control unit 11 is composed of one or more processors having a processing device such as a CPU and a memory storing a program, etc., and can control each unit in the information processing system 10 by executing the program.
  • the control unit 11 includes an input determination unit 12, an image similarity determination unit 13, a metadata addition unit 14, a classification unit 15, a first request unit 16, a second request unit 17, and a learning unit 18. , and a recording control unit 18A.
  • the input determination unit 12 inputs image data acquired by the image acquisition unit 23 .
  • image data such as input images P1 to P3 and SP, category information, and the like are input through this communication network.
  • the input determination unit 12 determines category information, metadata, and the like associated with the input image data. Further, it may be determined whether or not the image is from the second endoscope based on the inference result when the image acquired by the image acquisition unit 23 is input to the existing inference model 19a (for example, FIG. 5B, S23). Since the existing inference model 19a is generated based on the image from the first endoscope, the reliability of the inference result is low for the image from the second endoscope. .
  • the input determination unit 12 functions as an input determination unit that determines whether the image is from the second endoscope according to the result when the image is input to the first inference model (for example, see S23 in FIG. 5B). ).
  • the input determination unit 12 functions as a video acquisition unit that acquires a new inspection video when the learning unit learns to create a second inference model that is different from the first inference model (for example, S21).
  • the input determination unit 12 determines the model of the imaging system 20, the image quality (optical specification), the illumination condition of the image, the angle of view of the image, the image In consideration of various information such as the shape of the input image, the characteristics of the input image are determined. Further, the characteristics of the input image may be determined based on the shape of the treatment instrument appearing together in the image, or, in the case of a moving image, on the state of operation. This makes it possible to determine what kind of device an image is from, for what purpose the same device is being used, and to determine the skill of operating doctors and medical staff. It is a thing.
  • the image similarity determination unit 13 determines whether the image input by the input determination unit 12 is recorded in the first timing image portion 31c and the second timing image portion 31d in the inspection image group 31 at the time of existing teacher data creation in the recording unit 30. It is determined whether the image is similar to any of the existing images.
  • the existing inference model 19a is an inference model for discrimination, it is an image acquired at the first timing, that is, an image acquired when a doctor or the like accesses a lesion such as a tumor.
  • the first timing image recorded in the first timing image section 31c is an image that is generally not used as teaching data for generating an existing inference model (first inference model) for the first endoscope. is.
  • the second timing image recorded in the second timing image section 31d is an image acquired at the second timing, that is, at the timing when a doctor or the like searches for and closely examines a lesion such as a tumor, or is adopted as the lesion. It is an image.
  • an image similar to the image at the time of insertion passes through the same site at the time of removal, so it is possible to obtain an image as if going back in time.
  • the insertion image is the first timing image
  • the removal image can be classified as the same first timing image, which is different from the second timing image.
  • the examination moving image is evidence recording examination procedures that change from moment to moment, and can be easily arranged as a series of still image frames classified along the time axis.
  • the first timing is when a doctor or the like operates the endoscope or the like and is in the process of moving it to a target site in the body.
  • the timing corresponds to the timing at which the doctor or the like accesses the affected area.
  • image Ps in FIG. 3B The image that can be acquired at this first timing is not intended for observation, and should only be able to access the affected area efficiently, and the image quality may be such that it can be seen that the tip is being sent into the body without damaging the inside.
  • the image acquisition cannot keep up with the operation of inserting the endoscope, and the image is blurred and blurred, resulting in low image quality. Therefore, the image quality is not suitable for creating an inference model for observation and discrimination, and is not recorded as an unnecessary image (see S3 in FIG. 3A(a), for example).
  • the endoscope or the like is operated by a doctor or the like in order to observe or discriminate an object, and the tip of the endoscope is moved slowly while it is near the target site in the body. , bending and scrutinizing (see image Po in FIG. 3B).
  • the doctor or the like moves the tip of the endoscope slowly, bends it, changes the image quality, enlarges it, and adjusts the affected area, while also performing controls such as removing dirt by pouring water and sucking. , changing the angle of confirmation, special observation, staining and fluorescence observation.
  • the image acquired at the second timing is in the vicinity of the target site, continues to capture the same site in the image, and often has high image quality with less blurring and focus.
  • the image acquired at this timing can be determined at least as an image acquired at a timing different from the first timing.
  • the images acquired at the first timing and the images acquired at the second timing are divided into groups according to the timing of image capturing, and recorded in the recording unit 30 as the first image or the second image (FIG. 3A).
  • first image is the first timing image recorded in the first timing image portion 31c
  • second image is a simplified notation of the second timing image recorded in the second timing image portion 31d.
  • the image similarity determination section 13 determines whether the input image is similar to the first timing image recorded in the first timing image section 31c or the second timing image recorded in the second timing image section 31d. Similarity to the image is determined based on the degree of similarity of the image.
  • the first timing image portion 31c and the second timing image portion 31d record which data is or is not teacher data, and whether or not it is annotation data. Therefore, if you use the images recorded here, it will also be evidence of what kind of data was adopted as training data and learned. It is possible to classify by referring to it when it arrives.
  • a first timing image and a second timing image are recorded in the first timing image section 31c and the second timing image section 31d when creating an inference model for an existing endoscope.
  • Information processing system 10 may then input images from various endoscopes (see, eg, input images PI1-PI3 in FIG. 2). Each time these images are input, the image similarity determination unit 13 records which data was or was not teacher data and whether or not it was annotation data when creating an inference model with similar specifications. , the first timing image, and the second timing image. This determination result is sent to the classification section 15 .
  • the metadata adding unit 14 adds metadata to image data.
  • category information 31a and specification information 31b are associated as metadata with the image data (first timing image and second timing image) of the inspection image group 31 at the time of existing teacher data creation.
  • the category information 31a is, for example, endoscope manufacturer information, and the sensor, optical characteristics, image processing circuit characteristics, etc. of the endoscope used as the image system 20 .
  • images are collected for each endoscope manufacturer, model, etc., training data is created, and an inference model is created using this teaching data. This is because even if an inference model is suitable for a certain model, the reliability of the inference may be low for another model. Therefore, the category information 31a is associated with the image.
  • the spec information 31b is information about specs of the inference model when the first inference model (existing inference model) is created using the image file 32A.
  • the spec information includes various information about the inference model, such as what kind of image was used when creating the inference model, and the structure of the inference model.
  • the metadata adding unit 14 functions as an appending unit that appends metadata to each image forming the image group.
  • the image file recorded in the recording unit 30 is also assumed to be a group of images obtained in time series (for example, a moving image obtained in a specific endoscopy) that is the source of the training data.
  • Information such as what kind of device the image was obtained from, what the image was obtained from, and the date and time when the image was captured can be organized by metadata as the category information 31a.
  • Metadata and a recording unit 30 indicating what kind of inference model was created from this image.
  • the specification information 31b of the inference model 19a is shown in FIGS. 1A and 1B, a plurality of inference models may be generated from the same image. Also, it may be possible to record which part (frame) of the image group has become the teacher data.
  • the information of the first timing image and the second timing image is simplified as being able to be recorded in the first timing image portion 31c and the second timing image portion 31d. In addition to this, it may be divided according to whether it is a training data candidate or not.
  • the image frame may be identified by the elapsed time or the number of frames, or may be managed by adding a flag or the like to the image frame.
  • teaching data may be classified and recorded. This may be the second timing image, or may be further classified as the third timing image.
  • the first timing and the second timing are used to show that the characteristics of the frames that form a group of images obtained in time series, such as a moving image, can be divided into at least two, that is, divided into a plurality of recordings. bottom.
  • annotated images are not necessarily adopted as training data, images that have undergone the labor-intensive work of annotation have a certain level of importance. It can be used and referred to when creating a model. In FIG. 1B, it seems that only one piece of inference model information is included in the image file.
  • the two-timing image portion 31d is arranged and recorded separately.
  • the category information Pc may be used as the category information 31a.
  • the input determination unit 12 determines the model of the imaging system 20, the image quality (optical specification), the illumination condition of the image, the angle of view of the image, the shape of the image, and so on.
  • the category information 31a may be given in consideration of various information such as. Further, the category information 31a may be determined and assigned based on the shape of the treatment instrument appearing together in the image, or, in the case of a moving image, on the state of operation.
  • spec information 31b is added only to the image file 32A of the inspection image group 31 at the time of existing training data creation. You may make it record the specification information of an inference model.
  • the classification unit 15 classifies the input image into the first timing image or the second timing image based on the determination result of the image similarity determination unit 13 . Depending on the result of this classification, the input image is attached with category information 32a indicating that it is the first timing image or the second timing image.
  • the first timing image may be used in generating a second inference model for unknown endoscopes (endoscopes of unknown identity) that are not well known in the market. .
  • the classification unit 15 utilizes the image group obtained at the first timing when the first inference model is created, and classifies the newly acquired image group from the first or second endoscope into It functions as a classification unit (processor) that classifies a group of images serving as training data candidates (see, for example, S35 and S37 in FIG. 5B).
  • the classification unit uses training data based on a group of images obtained in chronological order by the first endoscope to create an existing inference model with similar specifications, eliminating the need for items that were not subject to annotation. image (see, for example, image Plo in FIG. 2).
  • the classification unit 15 also utilizes an image group that was not used as teacher data when the first inference model was created, to obtain a second endoscope different from the first endoscope. It functions as a classification unit that classifies the group of inspection images obtained from the mirror (see, for example, S35 and S37 in FIG. 5B).
  • the classification unit 15 uses the features of the image group obtained at the first timing when the first inference model is created to classify the first endoscope (existing endoscope, manufactured by a specific manufacturer). A group of images acquired from a second endoscope different from the first endoscope) is classified. That is, the classification unit 15 uses the image group obtained at the first timing when the first inference model is created, and classifies the features of this image group (for example, the first timing image group and the second timing image group). The images obtained from the second endoscope, which is different from the first endoscope, are divided temporally and compared with the same specifications, image quality, and other performance. , finds the difference in features between the first image and the second image, and classifies annotation candidate images from all the image frames of the inspection image group at the time of creating the existing teacher data.
  • the classification unit 15 utilizes images that were used when the first inference model was created, including a group of images that were not used as teacher data, determines their characteristics, and identifies the first endoscope. It functions as a classification unit that classifies images acquired from different second endoscopes as annotation candidate images (see, for example, S35 and S37 in FIG. 5B). It should be noted that the second timing is determined according to the input/output relationship (specification of the inference model) required for the inference model.
  • the recording control unit 18A stores an image determined by the input determination unit 12, similarity determination by the image similarity determination unit 13, metadata addition by the metadata addition unit 14, and classified by the classification unit 15, to the recording unit 30.
  • control recording to In recording control to the recording unit 30, the group of images input when creating the existing teacher data is recorded in the group of inspection images 31 when creating the existing teacher data. Record in group 32.
  • the number of existing inference models using the existing training data is not limited to one, and there may be a plurality of existing inference models. Therefore, a plurality of image files 31A are included in the inspection image group 31 at the time of creating the existing training data according to the respective existing inference models. Recorded. In FIG. 1A, it appears that there is only one image file 31A, but many image files are recorded.
  • the first requesting unit 16 creates an inference model using the second timing image group acquired at the second timing and recorded in the second timing image portion 31d of the inspection image group 31 at the time of existing teacher data creation.
  • the learning unit 18 is requested to do so.
  • This inference model is used as the existing inference model 19a.
  • the first requesting unit 16 inputs an endoscopic image using a first endoscope (an existing endoscope, an endoscope manufactured by a specific manufacturer).
  • the learning unit 18 is requested to generate an improved inference model of the existing inference model 19a using this image (for example, FIG. 3A (a ), S27 to S33 in FIG. 5B, etc.).
  • the first requesting unit 16 provides training data for generating a third inference model obtained by improving the first inference model for the image group according to the reliability of the result of inferring the same image with the first inference model. It functions as a first requesting unit that requests annotation (see FIG. 4(a), S7 in FIG. 3A(a), S107 and S109 in FIG. 3B, and S27 to S33 in FIG. 5B).
  • the first requesting unit 16 for requesting the creation of an existing inference model does not necessarily have to be provided within this system. However, this first requesting unit has a track record of creating an existing inference model by selecting training data, and has some kind of cooperation with the information processing system 10. Since it is recorded as the image group 31, it is shown in the figure to clearly show that the first requesting unit has controlled what kind of image was used as teacher data at that time.
  • the second requesting unit 17 instructs the learning unit 18 to create an inference model using an image suitable for generating the second endoscope inference model from among the images 32b recorded in the inspection image group 32.
  • endoscopes with a first endoscope an existing endoscope, an endoscope manufactured by a specific manufacturer
  • a second endoscope are included in the inspection image group 32.
  • the image data acquired when performing a mirror examination is recorded.
  • the image data acquired by the second endoscope is annotated and used as teacher data.
  • the inference model generated by the request by the second requesting unit 17 is an inference model used when a doctor or the like performs an examination using the second endoscope. may be set in the inference engine 19 in the image system 20 as an inference model of
  • the second requesting unit 17 classifies the group of images acquired from the second endoscope and classified as those for generating an inference model for the second endoscope. It functions as a second requesting unit that requests annotations in order to obtain an inference model that has the same specifications as the first inference model but corresponds to the image acquired by the second endoscope (for example, FIG. 5B (see S43 of ).
  • the first requesting unit 16 and/or the second requesting unit 17 functions as a selection unit that selects new training data candidates from the new inspection video according to the classified image group recorded in the recording unit (for example, FIG. 7 (see S93, etc.).
  • the second requesting unit 17 also functions as a second requesting unit that learns the result of annotating the inspection image group as teacher data and requests generation of the second inference model (for example, S27 to S33 in FIG. 5B). reference).
  • the learning unit 18 receives requests from the first request unit 16 and the second request unit 17 and generates an inference model.
  • the learning unit 18 has an inference engine, inputs teacher data, and sets the weighting of intermediate layer neurons so that the inference result is the same as the annotated one.
  • Deep learning is a multilayer structure of the process of "machine learning” using neural networks.
  • a typical example is a "forward propagation neural network” that sends information from front to back and makes decisions.
  • the simplest forward propagation neural network consists of an input layer consisting of N1 neurons, an intermediate layer consisting of N2 neurons given by parameters, and N3 neurons corresponding to the number of classes to be discriminated. It suffices if there are three output layers composed of neurons.
  • the neurons of the input layer and the intermediate layer, and the neurons of the intermediate layer and the output layer are respectively connected by a connection weight, and the intermediate layer and the output layer are added with a bias value, thereby easily forming a logic gate.
  • the neural network may have three layers for simple discrimination, but by increasing the number of intermediate layers, it is possible to learn how to combine multiple feature values in the process of machine learning. In recent years, 9-layer to 152-layer systems have become practical in terms of time required for learning, judgment accuracy, and energy consumption. Also, a process called “convolution” that compresses the feature amount of an image may be performed, and a “convolution neural network” that operates with minimal processing and is strong in pattern recognition may be used. In addition, a “recurrent neural network” (fully-connected recurrent neural network), which can handle more complicated information and can handle information analysis whose meaning changes depending on the order and sequence, can be used.
  • NPU neural network processing unit
  • machine learning such as support vector machines and support vector regression.
  • the learning involves calculation of classifier weights, filter coefficients, and offsets, and there is also a method using logistic regression processing. If you want a machine to judge something, you have to teach the machine how to judge.
  • a method of deriving image determination by machine learning is used, but a rule-based method that applies rules acquired by humans through empirical rules and heuristics may also be used.
  • the learning unit 18 can also be used when generating an existing inference model.
  • the learning unit time-series data obtained by the first endoscope to determine a specific image feature for images continuously obtained from the first endoscope.
  • the group of images obtained at the first timing is classified here as images during access, and the group of images obtained at the second timing is classified as images at the time of confirmation of a tumor or the like. do.
  • Annotations may be made for images in which a tumor or the like has been confirmed, such as where it is located within the image frame and the result of distinguishing what kind of tumor it is.
  • an existing inference model can be generated.
  • the specification of this existing inference model can be written as tumor discrimination. Again, it may be learned for creating an inference model with other specifications, but at that time, the segmentation of the image group and the contents of the annotation will be different.
  • the learning unit that generates the existing inference model in the present embodiment uses the first endoscope to determine specific image features for the images continuously obtained from the first endoscope.
  • the group of images obtained in chronological order by the mirror is either the group of images obtained at the first timing (images during access) or the group of images obtained at the second timing (obtained at the timing of searching for tumors, etc.) image), and the result of annotating the image group obtained at the second timing is learned as teacher data.
  • the learning unit 18 is provided in the information processing system 10, but the existing learning unit for inference model generation does not have to be provided inside the information processing system 10. All that is required is that the learning unit and the information processing system 10 can cooperate with each other.
  • the learning unit 18 uses the images selected from the group of images as teaching data candidates by the classification unit as training data, and has the same specifications as the first inference model, but corresponds to the images acquired by the second endoscope.
  • a second inference model is generated (see S89 in FIG. 7, for example).
  • the second inference model generated here is assumed to have specifications similar to those of the existing inference model, and the specifications are set in the inference engine 19 .
  • the existing inference model infers a lesion such as cancer
  • the second inference model has similar specifications for inferring a lesion such as cancer with respect to the image acquired from the second endoscope.
  • the classification of the images already used for learning the first inference model can be used as a reference.
  • the learning unit 18 functions as a learning unit that creates a first inference model for endoscopes by learning, as teacher data, an image group included in an inspection video obtained from an endoscope.
  • the learning unit 18 annotated the first image group obtained by the first endoscope in order to determine a specific image feature for the images obtained by the first endoscope.
  • It functions as a learning unit that obtains a first inference model for image feature determination of the first endoscopic image learned by using the result as teacher data.
  • the first inference model is an inference model for displaying an inference result for an image obtained by an endoscope.
  • An existing inference model 19a is set in the inference engine 19, but in this example, it will be explained assuming that it is used for discriminating what the lesion is.
  • the image system 20 transmits the images (P1 to P3, SP, etc.) acquired by the image acquisition unit 23 to the information processing system 10 (see S13 in FIG. 5A, for example).
  • the inference engine 19 infers whether or not there is a lesion such as cancer and its position, and transmits the inference result to the image system 20 (see S25 in FIG. 5B, for example).
  • the image system 20 outputs an inference result 25a and a reliability 25b to the image system 20 when the inference result is input.
  • the imaging system 20 outputs the received inference result 25a and reliability 25b to the display unit 24 .
  • the inference engine 19 compares the image obtained from the first endoscope with the existing teacher data inspection image group obtained by the first endoscope in order to determine a specific image feature. It functions as an inference engine that sets a first inference model for image feature determination of the first endoscopic image learned using the annotated result as teacher data.
  • the recording unit 30 may be provided inside a server or the like in which the information processing system 10 is provided, or may be provided inside a server or the like outside the information processing system 10 .
  • the recording unit 30 is an electrically rewritable nonvolatile memory capable of recording a large amount of image data.
  • the recording unit 30 stores a group of inspection images 31 used when creating the existing teaching data, inspection images 32 input from various endoscopes after creating the existing teaching data, and specification information for creating a new inference model. 33 can be recorded.
  • the inspection image group 31 at the time of existing teacher data creation records image data when images were collected to generate an inference model suitable for existing endoscopes.
  • a doctor or the like performs an examination using an existing endoscope (first endoscope). It is recorded as an inspection image group when creating training data.
  • the second timing image acquired at the second timing that is, at the timing when the doctor or the like finds the affected area, is used for creating this existing teacher data.
  • the time of examination as described above, after the endoscope is inserted into the body and the affected area is found, until the endoscope is removed from the body, that is, the first timing is different from the second timing.
  • the first timing image is referred to and arranged as the first timing image.
  • the information processing system 10 When creating an inference model for existing endoscopes, this inference model is not limited to a single one, and multiple inference models may be created according to various uses and models. Therefore, the information processing system 10 generates the image file 32A each time the image acquisition unit 23 acquires image data (including an image data group). A plurality of image files 31A are recorded in the inspection image group 31 at the time of existing training data creation. Each image file 31A has category information 31a, spec information 31b, first timing image portion 31c, and second timing image portion 31d. The first timing image recorded in the first timing image portion 31c corresponds to image data such as the images lo and Pna acquired by the imaging system 20 and the second timing images Pad1 to Pad3 (see FIG. 2).
  • the inspection image group 32 is obtained after the images have been collected for creating the existing teaching data, using the first endoscope (existing endoscope, endoscope made by a specific manufacturer) or the second endoscope. Record the image data acquired during the endoscopy.
  • the information processing system 10 may receive images from an unknown endoscope in addition to existing endoscopes (for example, see image PI3 in FIG. 2). An image may be input by a mirror.
  • the inspection image group 32 corresponds to such images.
  • a new inference model is generated using this inspection image group 32 .
  • the inspection image group 32 in this case is used to improve the existing inference model. Therefore, the image data acquired by the information processing system 10 is associated with various information and recorded in the recording unit 30 as the inspection image group 32 .
  • the new inference model creation specification information 33 is information on specifications for creating an inference model when creating an inference model using the image data of the inspection image group 32, as will be described later.
  • specifications regarding the population of teacher data and specifications such as the definition of the configuration of the inference model are required. Also, when improving an existing inference model, the specification information 33a of additional teacher data is recorded.
  • the recording unit 30 records a group of images obtained at the first timing and a group of images obtained at the second timing for creating the first inference model. (For example, see images Plo, Pad1 to Pad3, Pna, etc. in FIG. 2). That is, not only the group of images obtained at the second timing for creating the first inference model (existing inference model) but also the images obtained at the first timing are recorded.
  • the recording unit 30 also functions as a recording unit that records at least a part of the inspection moving image together with the result of classifying the image group when the teacher data is selected from the image frames included in the inspection moving image based on the time information of the moving image frame. (For example, S35, S37, etc. in FIG. 5B).
  • the first timing and the second timing are determined when acquiring the inspection image for teacher data, and the image data is recorded based on the determination result.
  • the recording unit 30 shown in FIG. 1B when recording the inspection image group 32, it may be determined whether it is the first timing or the second timing, and this timing information may be recorded together. Note that at least part of the inspection moving image described above includes annotation candidate images.
  • the information processing system 10 can be connected to various endoscope apparatuses (image processing systems 20), and therefore various image data are transmitted.
  • the information processing system 10 can use this variety of image data to generate improved inference models and new inference models in addition to existing inference models.
  • the improved inference model is an improvement over an existing inference model.
  • the new inference model is a completely new type, and is an inference model that enables accurate inference even when image data with image quality different from that of the previous one is input. This new inference model is generated using the inspection image group 32 recorded in the recording unit 30 .
  • images Plo, Pad11 to Pad3, and Pna are images to be examined or images to be examined when the information processing system 10 creates the existing inference model 19a. These images have been used when an existing inference model (a specific inference model) was created, and the history is organized. When an inference model is well-established, it contains very important information and is organized.
  • the information processing system 10 includes the existing inference model 19a shown in FIG. 1A, the input determination unit 12, the image similarity determination unit 13, the metadata addition unit 14, and the classification unit shown in FIG. 1B. 15, a first requesting section 16, a second requesting section 17, a learning section 18, and a recording control section 18A.
  • the information processing system 10 first generates an existing inference model 19a based on the input image. That is, when the input images Plo, PI1 to PI3, and Pna are input, the input determination unit 12 determines that the images are acquired by an endoscope manufactured by a specific manufacturer based on information such as category 1 of the images. do. Then, the information processing system 10 excludes the image Plo of low importance among the input images, rejects the image Pna for some reason, and adds annotations indicating the position of cancer or the like to the images of the input images PI1 to PI3. is requested, and teacher data is created.
  • the information processing system 10 uses teacher data based on a group of images obtained in time series by the first endoscope to create an existing inference model with similar specifications, which was not subject to annotation. is an unnecessary image.
  • the learning unit 18 uses this teacher data to generate the existing inference model 19a.
  • the organized and used image group (images Plo, PI1 to PI3, Pna, etc.) can be annotated to become teacher data, and any image to be annotated can be used as teacher data. It is valuable know-how information including the results obtained through trial and error, such as whether it was used as a In other words, it is possible to determine whether or not new training data candidates can be used by using images that have not been used so far. Even so, this system makes it possible to make a decision. Other than the ones that should not be used, however, there is a possibility that new information is included. Better. In the present embodiment, this concept is used to collect images efficiently.
  • the generation of the existing inference model 19a will be specifically described below.
  • the image Plo is an image of low importance
  • the images Pad1 to Pad3 are images of existing teacher data that has been adopted
  • the image Pna is an image that was created as existing teacher data but has not been adopted.
  • These images Plo, Pad1 to Pad3, and Pna are all images acquired by the endoscope of the above-mentioned specific manufacturer.
  • the reason for using a specific manufacturer here is that, in general, different manufacturers of endoscopes have different sensors, optical systems, image processing, and the like. Decrease the reliability of the model. Even if the manufacturer is different, if the sensor, optical system, image processing, etc.
  • the information processing system 10 carefully judges and uses the image data of the endoscope of a specific manufacturer to generate the inference model 19a that takes into consideration the characteristics of the model. ing.
  • the unused image Plo is an image of low importance as described above, and has image data 51a and category information 51b. Since the image Plo is an image acquired by the aforementioned specific manufacturer, it is classified as category 1 in the category information 61b.
  • This image Plo may be an image obtained when a doctor or the like searches for a lesion using an endoscope, for example, in the case of creating an inference model for lesion discrimination (for example, FIG. 3 See image Ps in (b)). Since the imaging unit is moving, the image being searched is of low quality because the image is blurred, and the possibility that the lesion is clearly captured is low.
  • an image is determined to be a search image based on the characteristics of the image, the characteristics of the captured image, the characteristics of the image change, etc., it is classified into the image group of the first timing that is unnecessary when creating the inference model for discrimination. may In addition, it can be said that inference is made as to what kind of frame in the moving image has become the training data, and that the result does not match. As described above, the concept of the present embodiment is to use the circumstances in which such an image was rejected when generating the inference model when generating the subsequent inference model.
  • Images Pad1 to Pad3 are images acquired when a doctor or the like uses an endoscope to perform a detailed examination assuming that there may be a lesion.
  • an inference model for discrimination there is a possibility that teacher data will be created from the image at the time of scrutiny, so it is better not to classify this as the image of the first timing from the beginning.
  • These images are clear high-quality images with the imaging unit still or with little movement, and are likely to show lesions, because the same site is observed continuously.
  • the image of the second timing is selected as the image of the second timing.
  • washing water may be sprayed on the surface, dirt on the surface may be aspirated, staining may be performed, and the wavelength of light and image representation may be changed by a light source or image processing so that the surface can be seen clearly.
  • the second timing image is often selected according to an inference model creation manual or the like.
  • the image Pna is an image that has not been adopted as existing teacher data.
  • the category information 52e of this image Pna belongs to category 1 because it is an image acquired by an endoscope manufactured by a specific manufacturer, and the image quality is at a usable level. annotation is given. However, for some reason, it is not adopted when generating the existing inference model 19a. The reason for this is, for example, that if an inference model is generated using this image Pna, the reliability is lowered, so it is excluded. Such images are unlikely to be treated with high priority even if similar images are obtained in the future. If necessary, such images may be classified as first timing images.
  • the images Plo, Pad1 to Pad3, and Pna are images acquired using an endoscope manufactured by a specific manufacturer. performs deep learning to generate an existing inference model 19a.
  • FIG. 2 only one image Plo with a low degree of importance is shown, and only three images Pad1 to Pad3 of the existing training data that have been adopted are shown. Only one image Pna of existing teacher data that does not exist is described. However, it goes without saying that a larger number of images are actually used for these images.
  • the images collected and arranged for the proven inference model can be fully utilized when improving this inference model or creating a new inference model for a different model.
  • the images collected and arranged for the proven inference model can be fully utilized when improving this inference model or creating a new inference model for a different model.
  • the imaging system 20 uses this existing inference model 19a to input the endoscopic images P1 to P3 and SP, and It is possible to determine the presence or absence of etc.
  • Such an inference model may be re-learned if it does not perform as originally expected.
  • the information processing system 10 After the images Pad1 to Pad3 are selected through painstaking trial and error and the existing inference model is generated, the information processing system 10 then receives image data 51b, 51c, and 51d from various endoscope devices.
  • Category 1 is associated with each image data as category information 52b, 52c, 52d, and annotation information 53b, 53c, 53d is added to each image data. Positions of lesions such as cancer are given as annotation information 53b, 53c, and 53d.
  • the input image PI1 in FIG. 2 has the category information 62a of category 1 and is an existing endoscope, it is possible to infer whether or not there is a lesion such as cancer using the existing inference model 19a. can.
  • each image frame is input to the inference model in real time from the beginning to the end of an endoscopy, and when a frame that has been identified as having an affected part is detected, a lesion frame is displayed, a warning is issued, etc., and correct discrimination is performed. becomes possible.
  • the inference result IO1 and the reliability R1 at the time of inference are output.
  • the same effect may be displayed for a diagnosis requested by a doctor by inputting a specific still image.
  • the input image PI2 is an endoscope different from an existing endoscope manufactured by a manufacturer different from the existing endoscope manufacturer, or an endoscope manufactured by an existing endoscope manufacturer but with a different model.
  • 1 is an image acquired by an endoscope with a
  • This input image PI2 may be inferred using the existing inference model 19a for the input image PI1. Therefore, at this time, for example, each image frame is input to the inference model in real time from the beginning to the end of the endoscopy. Correct discrimination becomes possible. In addition, the same effect may be displayed for a diagnosis requested by a doctor by inputting a specific still image.
  • the inference as described above is performed on the input image, and the inference result IO2 and the reliability R2 at this time are output. If the reliability is low, the current inference model may be a lesion that cannot be judged with confidence. Request generation (see, for example, S83 and S87 in FIG. 6).
  • the images at this time are recorded as the inspection image group
  • the input image PI3 is an image belonging to category 3, is an image acquired by a completely unknown endoscope that is not known in the market, and has characteristics different from the image data acquired so far. For this reason, for example, for an input image PI3 in which each image frame is input to the inference model in real time from the beginning to the end of an endoscopy, or for which a doctor inputs a specific still image and requests identification, Existing inference models may not be able to output highly reliable inferences. Therefore, the above-described second requesting unit 17 requests generation of a new inference model. For this request, the images recorded in the inspection image group 32 are used to create the new inference model creation specification information IF, and the new inference model creation specification information IF includes the additional training data specification TD.
  • Efficient training data conversion is possible by reflecting effective knowledge at the time of creation. That is, by acquiring a new inspection video to be inferred by the second inference model, and classifying this new inspection image with a classification that reflects the above-mentioned image classification result, the second inference model A selection step is provided to select training data candidates for. This enables rapid inference model development (learning).
  • a step of learning a first inference model for inferring affected area information contained in the frame of the inspection video using at least one of the training data candidates may be provided in which the result of annotating the frame corresponding to is used as a training data candidate.
  • the learning unit that created the first inference model for the endoscope by learning the group of images included in the inspection video obtained from the endoscope as teacher data further learns and improves the model. It can also be explained as a device or method that enables continuous learning. That is, a recording unit is provided in which (at least a part of) an inspection movie is recorded along with the result of classifying a group of images when teacher data is selected from image frames included in the inspection movie in advance by time information of the movie frames.
  • a video acquisition unit that acquires a new inspection video when learning a second inference model that has similar specifications but has a different version and assumed device from the first inference model, such as additional learning by the learning unit.
  • a selection unit that efficiently selects new training data candidates from the new inspection moving image according to the classified image group recorded in the recording unit.
  • the images recorded as inspection images 32 in 30 are used as effectively as possible. That is, inspection images acquired by a doctor or the like during inspection are recorded, and these images are used (for example, see S28, S29, S37, etc. in FIG. 5B).
  • the new inference model creation specification information 33 and additional training data specification 33a are specification information for generating a new inference model for inferring images belonging to category 3.
  • the second requesting unit 17 requests the learning unit 18 to generate an inference model together with these pieces of information. What characteristics of the inference model for endoscopy are necessary and what kind of inference is required for the affected area image are reflected in the image data, changes in the image data, and captured images. It can be determined based on the image features of the generated object, or the inference result and reliability of an existing inference model.
  • FIG. 3A shows an operation in which an inference model with specific specifications exists as the existing inference model 19a in FIG. 1A, and a new inference model following the specifications of this existing inference model is generated.
  • the explanation will be given under the condition that the inference model of the existing specification is inferred with specific expectations, the explanation will be continued assuming that the specification selection has already been made.
  • the flow of FIG. 3A is realized by the control unit 11 in the information processing system 10 controlling each unit in the information processing system 10 according to the program stored in the memory. This flow can efficiently acquire training data for new inference model generation from a large number of images.
  • step S1 in FIG. 3A branches to Yes.
  • This is an image from a known device, but if it is an image that has never been obtained before, it is given priority as a training data candidate because it may have some material for improvement.
  • the countermeasure when step S1 in FIG. 3A is branched to No Since this is not a known endoscope, all images can be "unseen” so to speak. However, even with such images, if the endoscope is used for examination of a specific body part, the process of insertion through the body cavity, access, confirmation of the affected area, and removal should be captured as images.
  • the input determination unit 12 inputs image data and the like from the image acquisition unit 23 of the image system 20, and determines whether or not it is possible to make an inference using the existing inference model 19a. This determination is made based on the category information associated with the image data. Among the input images in FIG. 2, some images belonging to categories 2 and 3 may not be associated with category information. In this case, determination may be made using other information such as other attached data, information on image quality, information on screen shape, and the like. Alternatively, the determination may be made based on the reliability of the inference, instead of based on the category or the like.
  • step S7 If the result of determination in step S1 is that the image can be handled by the existing inference model, only unseen images are collected as teacher data (S7).
  • the determination result in step S1 since it is possible to correspond to the existing inference model, it is usually an image that has been seen (in other words, it is similar to the image acquired so far, or the characteristics of the image are the same as before). image). If the original endoscopic images (moving images) used when creating an inference model that has already been proven are recorded (recorded as the inspection image group 31 when creating the existing teacher data), this determination is similar.
  • the degree determination unit 13 can perform the determination by comparing with those characteristics. A similarity determination is performed, and images exceeding a specific threshold value may be determined as similar images (images that have been seen before).
  • an existing inference model when generating an existing inference model, if you record which part of the image has been used as training data, it will be possible to determine whether it is rare as training data. An image that is determined to be dissimilar by these determinations can be determined as an "unseen image". For example, for endoscopic video information obtained during a specific examination, the video information obtained by using the frame corresponding to the teacher image in this endoscopic video as annotation information is used as teacher data, and furthermore, a plurality of examinations are performed. The resulting endoscopic video images are converted into training data in the same way, and an existing inference model is generated. An endoscopic video including inserted or removed images is input to this existing inference model, and inference is performed in which the frames used as teacher data are output.
  • the collected images are recorded as an inspection image group 32 in the recording unit 30 . Metadata indicating that the image is a rare and never-seen image is attached at the time of recording. A specific example of a method for determining whether or not an image is an unseen rare image will be described later with reference to FIG.
  • step S1 For example, for endoscopic video information obtained during a specific examination using a specific model of endoscope, a video obtained by annotating the frame corresponding to the teacher image in this endoscopic video The information is used as training data, and the same training data is generated for the endoscopic video images of a plurality of inspection results to generate an existing inference model.
  • An endoscopic moving image including an insertion or removal image may be input to this existing inference model, and inference may be performed in which frames used as teacher data are output. Inferences in this step are made for images from endoscopes with different specifications and characteristics.
  • FIG. 3A is a simpler example, in which it is determined whether or not the image is of low importance (S3), and the other images are used as training data candidates.
  • Images with a low degree of importance are determined by reflecting the result of selection according to the timing from a sequence of images such as videos when creating an inference model with specifications as a model. For example, if an inference model for observation and discrimination is used as a model specification, an access image is necessary regardless of the endoscope model, so it is acquired (at least converted to electronic data) and used. and separate it from other parts (parts other than access). Comparing images from completely different models may reduce the reliability of the comparison because the objects to be compared are too different. It is easy to spot the difference with the confirmation image. Then, this difference may apply the logic when selecting a specific teacher data candidate portion (frame) from the known endoscopic images with proven results, or may apply inference.
  • the images obtained sequentially from the first endoscope when the known inference model was created were obtained in time series to determine a particular image feature, such as lesion detection.
  • the image group it is already known which frame was adopted as training data. In other words, it can be determined whether the image group is obtained at the first timing or the image group obtained at the second timing.
  • the teacher data was obtained from the second timing.
  • a second endoscope different from the first endoscope by using the feature of the image group obtained at the first timing when the first inference model is created for the learning unit A classifying unit for classifying the image group acquired from .
  • the above-described classification unit uses the image group obtained at the first timing when the first inference model is created, and classifies its features (for example, the characteristics of the first and second image groups). difference, etc.), divide the image acquired from the second endoscope different from the first endoscope in terms of time, and compare with the same specifications, image quality, etc., to determine the first It is possible to find the difference in features between the image and the second image and classify annotation candidate images from all image frames of the specific inspection image.
  • candidate frames for annotation are automatically selected, and the actual annotation is completed as training data by a specialist such as a doctor.
  • An inference model may do. If a process is provided in which the doctor confirms the results, high-speed training data creation and inference model creation will be possible while ensuring quality.
  • An image (see, for example, the image Plo in FIG. 2 and the image Ps in FIG. 3B) when a doctor or the like is screening for an endoscopy (see, for example, the image Ps in FIG. , is an image when the imaging unit is moved to search for lesions, etc., and the image is flowing and the image quality is low, so it is not suitable for teaching data used when improving the existing inference model.
  • this step S3 it is determined whether or not the image is suitable for such teacher data. Note that this determination may be made not only for one image but also for each continuous image. In other words, even if the image quality is low with only one image, continuous images may be used as teaching data in the case of, for example, an inference model for guide display during access.
  • images that are unlikely to be candidates for training data that is, images with low importance
  • images other than these images can be collected as training data candidates (S5).
  • the images determined to be of low importance in step S3 are excluded, and the image data of not low importance are collected as teacher data.
  • This image corresponds to the image Po under observation in FIG. 3B, and should include an image obtained by carefully observing (scrutinizing) the lesion area by a doctor or the like.
  • Images that cannot be dealt with by existing inference models and that make it difficult for conventional inference models to make inferences are valuable images for creating new inference models. Therefore, the images acquired here are recorded as an inspection image group 32 (see FIG. 1B). At the time of this recording, metadata indicating that the image is an unknown image and an image for the new inference model is added.
  • annotation 1 or annotation 2 is requested (S9).
  • the first requesting unit 16 uses the images collected in step S7 to request annotation 1 for generating an improved inference model.
  • This annotation 1 is mainly based on existing teacher data, and is an annotation for creating teacher data for learning in addition to it, so existing annotation tools and the like can be used as they are.
  • the second requesting unit 17 uses the images collected in step S5 to request annotation 2 for generating a new inference model.
  • This annotation 2 does not know whether the existing training data can be used (depending on the performance of the inference model), and it is necessary to obtain the output for this new unknown endoscope device, and it is not necessarily the same annotation tool as the conventional specification I don't know if it's okay, so I dare to separate it from Annotation 1 and call it Annotation 2.
  • the learning unit 18 is requested to generate a current improved inference model or a new inference model. When the annotation is requested, this flow ends.
  • the new inference model creation flow when inferring lesions such as cancer from the input image, it is determined whether inference is possible with the existing inference model.
  • rare images are collected as training data candidates (see S1 Yes ⁇ S7).
  • the collected training data candidates are used as data for improving existing inference models (see, for example, S81, S83, and S87 in FIG. 6).
  • this image is used as a teacher data candidate except for images with low importance.
  • the training data candidates collected here are used as data for generating a new inference model (see, for example, S81 No ⁇ S85 Yes ⁇ S91 in FIG. 6).
  • the information processing system 10 receives various images from various image processing systems 20 (including endoscopes), some of which cannot be inferred by existing inference models.
  • image processing systems 20 including endoscopes
  • images that could not be inferred are recorded as the inspection image group 32 (see S5) so that these images can be used efficiently when generating a new inference model.
  • the existing inference model can be inferred, if it is different from the image acquired so far (if it is not similar), this image is recorded as the inspection image group 32. It is prepared so that it can be used when improving an existing inference model.
  • step S1 it is determined whether or not the existing inference model can be handled. When possible, rare images that have never been seen were collected for improving the inference model (see S7).
  • images for creating a new inference model are collected regardless of whether the existing inference model can be used (see S101 and S103). After this collection, only when the existing inference model can handle it (S105 Yes), rare images that have never been seen are collected for improvement of the inference model (see S109).
  • steps S3, S5, S1, S7, and S9 of FIG. 3A perform the same processes as S101, S103, S105, S107, and S109 of FIG. 3B, respectively. , the only difference is the order of processing, so a detailed description will be omitted and a brief description will be given.
  • the classification unit 15 determines whether or not the image is of low importance based on the determination result of the similar image determination unit 13 performed on the input image. Images with a low degree of importance are determined by reflecting the result of selection according to the timing from a sequence of images such as videos when creating an inference model with specifications to be a model, as described above. For example, if the image is similar to an image that was not used when creating the inference model, the importance is determined to be low.
  • step S103 images other than those determined to have low importance are collected as teacher data (candidates) (S103).
  • the images other than the images with low importance are recorded in the recording unit 30 as teacher data (candidates). That is, the images selected from the group of images serving as training data candidates are recorded in the recording unit.
  • step S106 When images other than images with low importance are collected as training data (candidates), it is determined whether or not the existing inference model can handle the input image (S106), as in step S1. If it is not possible, only unseen images are collected as teacher data (candidates), as in step S7. Once the processing in steps S105 and S107 has been performed, next, similarly to step S9, annotation 1 or annotation 2 is requested (S109), and when the request ends, this flow ends.
  • image collection for creating a new inference model can be performed regardless of whether the existing inference model can be used. Therefore, a wide variety of images can be collected.
  • FIG. 4A shows whether or not the image is an unseen image in step S7 (S107 in FIG. 3) when it is determined in step S1 (S105 in FIG. 3B) that the existing inference model can handle the image. It is a figure explaining the method of judging whether.
  • an inference engine with two kinds of inference models a normal detection AI (inference model) 19a and a lesion detection AI (inference model) 19b, is prepared, and the same endoscope For image PI4, inferences are made using these AIs (inference models).
  • the normality detection AI 19A determines whether the endoscopic image PI4 is normal, in other words, whether the endoscopic image PI4 is abnormal.
  • the lesion detection AI 19B determines whether or not there is a lesion in the endoscopic image PI4.
  • An inference model for inferring whether or not there is a lesion such as cancer generally corresponds to this lesion detection AI. If this lesion detection AI is the first inference model, the normal detection AI corresponds to the third inference model.
  • the inference model set in the normal detection AI 19b is generated by learning using images without lesions as training data. This inference model can infer whether the input image is normal or not, as described above. Images for generating this inference model may be, for example, images determined to be other than the above in step S27 of FIG. 5B, which will be described later.
  • the normal detection AI19A determines that it is normal, and the lesion detection AI19B determines that there is no lesion.
  • the example shown in FIG. 4A is a case where the existing inference model is not suitable for inferring the endoscopic image PI4, and the inference result by the normal detection AI19A is "abnormal", while the lesion detection AI19B The inference result is "no lesion”. That is, the determination results by the two AIs are inconsistent.
  • the endoscopic image PI4 can be inferred using the existing inference model, if the determination results by the two AIs are inconsistent, the endoscopic image PI4 will not be seen. It can be said that it is a rare image that does not exist. In this case, an image that has never been seen is recorded as an inspection image 32 in the recording unit 30, annotated to create training data, and using this training data, an existing inference model is improved. should be used on occasion.
  • the first request unit 16 requests the image to be annotated to create teacher data.
  • Request generation of an inference model That is, the first requesting unit 16 improves the first inference model for the image group according to the reliability of the result of inferring the same image (for example, image PI4) with the first inference model (for example, normal detection AI 19a). It functions as a first requesting unit that requests annotations for teacher data for generating a third inference model (for example, see FIG. 4A and S27 to S33 in FIG. 5B).
  • the result inferred by the first inference model may be inferred to be the image acquired by the first endoscope (such as an endoscope manufactured by a specific manufacturer), or for other lesions If it is inferred that the inference model makes a correct decision, it may be determined to be reliable.
  • the first endoscope such as an endoscope manufactured by a specific manufacturer
  • FIG. 4(b) shows the case where it is determined in step S1 (S105 in FIG. 3B) that the existing inference model cannot cope.
  • the image change speed of the endoscopic image PI5 is detected, and it is determined whether or not screening (access) is in progress, ie, a doctor or the like is searching for a lesion and moving the imaging section.
  • the flowchart shown in FIG. 5A explains the main operation of the endoscope 1.
  • the information processing system 10 is connected to various endoscopes and acquires various information from the image acquisition section.
  • the endoscope 1 is an endoscope manufactured by a specific manufacturer, and the information processing system 10 has generated the existing inference model 19a based on the image from the endoscope 1 .
  • the main operation of the endoscope 1 shown in FIG. 5A is realized by the controller 21 in the imaging system 20 corresponding to the endoscope 1 controlling each part in the imaging system 20 .
  • an image is acquired and displayed (S11).
  • the image acquisition unit 23 acquires an endoscopic image and displays this endoscopic image on the display within the display unit 24 .
  • a doctor or the like operates the endoscope 1 while viewing the endoscopic image, moves the imaging unit of the endoscope 1 to a target site, and observes the target site.
  • the image is then transmitted to the information processing system 10 and the inference result is acquired (S13).
  • the control unit 21 transmits the image acquired by the image acquisition unit 23 to the information processing system 10 through the communication unit.
  • the model information (information indicating the endoscope 1) is transmitted to the information processing system 10 together with the image information (see S23 in FIG. 5B).
  • the information processing system 10 that has received the image uses the existing inference model 19a of the inference engine 19 to infer whether or not there is a lesion such as a cancerous area, and returns this inference result to the endoscope 1 (Fig. 5B, S25).
  • the imaging system 20 receives this inference result, it also receives the reliability of the inference result.
  • step S15 it is determined whether or not a highly reliable determination result has been obtained (S15).
  • the control unit 21 determines whether or not the reliability is higher than a predetermined value. As a result of this determination, if the reliability is lower than the predetermined value, the process returns to step S11.
  • step S15 determines whether the result of determination in step S15 is that the reliability is high.
  • step S13 the inference engine 19 infers the presence or absence of a lesion such as cancer and, if present, the position thereof, and the endoscope 1 receives this inference result.
  • the display unit 24 displays the position of the lesion such as cancer inferred by the inference engine 19 with a frame. It is of course possible to use a display method other than the frame display method.
  • an operation guide may be displayed for the operator of the endoscope. For example, a message such as "possibility of bleeding" may be displayed.
  • an inference model for guide may be prepared and the guide may be output by this inference model.
  • the information processing system 10 is connected to various endoscopes as described above, and acquires various information from the image acquiring section.
  • This information processing system determines whether the endoscopic images sent from various endoscopes are images that can be used for the current improved inference model, images that can be used to create a new inference model, or other images. judge. Then, according to the result of this determination, a request is made to add an annotation to create training data, and a request is made to generate an inference model.
  • the main operation of the information processing system shown in FIG. 5B is realized by the control unit 11 of the information processing system 10 controlling each unit in the information processing system 10 .
  • the information processing system 10 When the flow of the information processing system in FIG. 5B starts, it first enters an image acquisition standby state (S21). As described above, the information processing system 10 can be connected to various endoscopes, and endoscopic images are transmitted from each endoscope. In this step S21, the control unit 11 is placed in a standby state so that images from various endoscopes can be received.
  • the control unit 11 determines whether or not the image received by the information processing system 10 is an image to be determined as to the presence or absence of a lesion such as cancer.
  • the control unit 11 determines whether or not the image is the target image based on the scene of the image (for example, is it an image of the inside of the body and is it an image during close examination by a doctor or the like?). For example, when a doctor or the like is examining a target site (affected area, site where a lesion such as a tumor is likely to exist, etc.), the tip may stop at the same position or search around it. do.
  • Whether or not a detailed examination is being performed may be determined by analyzing an endoscopic image, or may be determined based on the operating state of a doctor or the like. As a result of this determination, if the image is not the target image, the process returns to step S21.
  • the inference engine 19 uses the existing inference model 19a to infer whether or not the target image has a lesion such as cancer, and if so, the position of the lesion. We also calculate the reliability of the inference theory at the time of inference. When the inference by the inference engine 19 is finished, the inference result and its reliability are sent to the image acquisition source (see S13 in FIG. 5A).
  • the control unit 11 determines whether or not the image has not been collected so far, and sets the image that has not been collected as teacher data.
  • the control unit 11 determines whether the received target image is suitable for generating the current improved inference model, suitable for generating a new inference model, or other.
  • the information processing system 10 can be connected to various inspection devices such as endoscopes through the Internet or the like, and these endoscopes or the like are manufactured by a specific manufacturer used when generating an existing inference model.
  • step S23 the endoscopic image received in step S23 is judged in order to generate an inference model for inferring a lesion such as cancer, even if the endoscopic image is not made by a specific manufacturer. Details of the determination in step S27 will be described later with reference to FIG.
  • step S28 If the result of determination in step S27 is that it is neither for current improvement nor for new inference model, other processing is performed (S28).
  • an image determined to be neither an improvement to the current state nor a new inference model is recorded.
  • S81 and S89 in FIG. 6 since it is an existing model and the reliability is other than 40 to 60%, it is an image that clearly shows whether it is a lesion such as cancer. be. Therefore, by recording this image and using it as training data, it is possible to create an AI for inferring that there is no abnormality.
  • the process After executing the process in step S28, the process returns to step S21.
  • step S27 determines whether the image is suitable for improving the current situation. If the result of determination in step S27 is that the image is suitable for improving the current situation, then the image is recorded as a teacher data candidate (S29).
  • the control unit 11 records the inspection image group 32 in the recording unit 30 as images suitable for generating an inference model for existing improvement. For example, rare images that physicians and others have never seen before can help improve existing inference models.
  • category information is preferable to record category information as metadata indicating that the image is an image for existing improvement.
  • an annotation is requested (S31).
  • the control unit 11 requests addition of annotations in order to generate an inference model for improving the current state.
  • a specialist such as a doctor associates data indicating the position of a lesion such as cancer with image data.
  • the annotation may be automatically added using AI or the like, or an annotation may be added by an expert with reference to AI advice.
  • the new training data are selected and learned to create an improved inference model (S33). Since new teacher data can be created when annotations are added, the control unit 11 appropriately selects new teacher data. That is, from the new teacher data, the teacher data suitable for generating the improved inference model are selected. After selecting the new teacher data, the second requesting unit 17 requests the learning unit to generate an improved inference model. The learning is requested to the learning section 18 in the information processing system 10, but if the information processing system 10 does not have a learning section, an external learning section is requested. After requesting generation of the improved inference model, the process returns to step S21.
  • step S35 determination of the inspection image for existing teacher data is performed (S35).
  • the existing teacher data inspection image group 31 is obtained at the first timing and the second timing as described above, and the image data is recorded when generating the inference model suitable for the existing endoscope.
  • the second timing is the timing when the doctor or the like is performing a detailed examination of the target site (see, for example, the image Po in FIG. Image quality.
  • the image similarity determination unit 13 determines whether or not the obtained image is similar to the existing teacher data inspection image group 31 .
  • the inspection image group other than the existing teacher data inspection image group is recorded as an inspection image group (S37).
  • the control unit 11 records only the images determined not to be similar to the existing teacher data inspection image group as the inspection image group in the recording unit 30 as the inspection image group 32. do. It is highly likely that this image was acquired by a completely unknown endoscope that is not known in the market. By inferring using this image, an inference model suitable for this completely unknown endoscope is created. be able to.
  • an annotation is requested, and images that have not been annotated are added to the inspection image group (S39).
  • the images determined to belong to the inspection image group in step S37 are images acquired by a completely unknown endoscope or the like, and thus the images are also images of an unknown type. Therefore, in order to generate a new inference model, we request annotations.
  • This annotation may be performed automatically using AI or the like, instead of being manually assigned by an expert such as a doctor.
  • the images that were not annotated will not be used when creating the new inference model this time, but since there is a possibility that they will be used when creating the inference model on another occasion, this image is used as the inspection image group 32 is added to the recording unit 30 as . At the time of this recording, a tag indicating that the image has no annotation is added.
  • the control unit 11 After annotating the inspection image group recorded in step S37, the control unit 11 records this image as new teacher data in the recording unit 30 as the inspection image group 32.
  • the new teacher data are selected, learned, and a new inference model created (S43).
  • the control unit 11 selects the new teacher data as appropriate. That is, from new teacher data, teacher data suitable for generating a new inference model are selected.
  • the second requesting unit 17 requests the learning unit to generate a new inference model. Learning is requested to the learning unit 18 in the information processing system 10 as in the case of generating the improved inference model, but if the information processing system 10 does not have a learning unit, an external learning unit request. After requesting generation of a new inference model, the process returns to step S21.
  • the information processing system 10 can be connected to various endoscopes, and acquires various information from the image acquiring section.
  • the endoscope 2 is an endoscope manufactured by a company other than a specific manufacturer, and is the endoscope that generated the image PI2 or the image PI3 in FIG.
  • the information processing system 10 has never generated an inference model based on this image from the endoscope 2 .
  • the main operation of the endoscope 2 shown in FIG. 5C is realized by the controller 21 in the imaging system 20 corresponding to the endoscope 2 controlling each part in the imaging system 20 .
  • the image acquisition unit 23 in the endoscope 2 acquires an endoscopic image, and displays this endoscopic image on the display in the display unit 24 .
  • a doctor or the like operates the endoscope 2 while viewing the endoscopic image, moves the imaging unit of the endoscope 2 to a target site, and observes the target site.
  • an inference support operation it is determined whether or not an inference support operation has been performed (S53). If the endoscope 2 is an unknown endoscope that is not known in the market, there is a possibility that an operator such as a doctor is unfamiliar with it. In such cases, the operator may request assistance by reasoning. In order to request an inference assistance, for example, an operation member for requesting an assistance operation to the endoscope 2 may be provided. In addition, even if an operator such as a doctor does not perform a requested operation, it can automatically determine whether or not an inference assist operation is being performed based on the analysis results of the endoscopic image and the operating state of the endoscope. may Of course, it may be determined by an inference model that infers whether or not the auxiliary operation is necessary. If the result of determination in this step is that the operator does not request an inference assist operation, the system enters a standby state.
  • step S53 If the result of determination in step S53 is that an inference assisting operation has been performed, then the image is transmitted to the information processing system 10 and an inference result is obtained (S55).
  • the control unit 21 of the endoscope 2 transmits the image acquired by the image acquisition unit 23 to the information processing system 10 through the communication unit.
  • the model information (information indicating the endoscope 2) may be transmitted to the information processing system 10 together with the image information.
  • the information processing system 10 that has received the image infers whether or not there is a lesion such as a cancerous part using the existing inference model 19a of the inference engine 19, and returns this inference result to the endoscope 2. (See S25 in FIG. 5B). When receiving this inference result, it also receives the reliability of the inference result.
  • step S57 it is determined whether or not a highly reliable determination result has been obtained (S57).
  • the control unit 21 determines whether or not the reliability is higher than a predetermined value. As a result of this determination, if the reliability is low, the process returns to step S53.
  • step S57 if the result of determination in step S57 is that the reliability is high, then the determined position is displayed in a frame (S59).
  • step S55 the inference engine 19 receives the inference result of the presence or absence of a lesion such as cancer, and if there is a lesion, the position of the lesion. In this step, the position of the lesion such as cancer inferred by the inference engine 19 is displayed in a frame. It is of course possible to use a display method other than the frame display method.
  • the process returns to step S51.
  • the inference model 19a performs inference, The results are returned to the imaging system (see S21 to S25). Therefore, even if the imaging system does not have an inference engine, it is possible to easily know the presence or absence of a lesion such as cancer and its position. In addition, if the acquired images are rare images that have never been seen even if they were acquired with an existing endoscope, these images are collected and the current improved inference model is used. Create (see S29 to S33).
  • step S27 the details of the operation of determining whether the current model is improved, new model, or other in step S27 (see FIG. 5B) will be described.
  • the model information is information such as the manufacturer name of the endoscope 1 in FIG. 5A, the endoscope 2 in FIG. 5C, and the model name of the manufacturer. Based on the model information and the like, it is determined whether or not the images received in steps S21 and S23 are images acquired by the endoscope manufactured by the existing manufacturer used when generating the existing inference model 19a. Since the model information may be included in the category information or the like associated with the image, the control unit 11 determines the model information based on the category information.
  • this step it may be determined whether or not the current model is related to an existing model based on information such as modes, accessories, and techniques, in addition to the model information. If tag data or the like indicating model information or the like is not attached to the image, determination may be made based on the characteristics of the image, the shape of the image, or the like.
  • step S61 determines whether or not the reliability of cancer determination is within a specific range (S63).
  • step S25 determines whether or not the reliability of cancer determination is within a specific range (S63).
  • Inference reliability is not limited to 0% and 100%, but can also take intermediate values. When the reliability is 40% to 60%, it cannot be said that the inference result is correct or incorrect when the existing inference model is used to estimate a lesion such as cancer. If the confidence is 0%, then the existing reasoning model does not fit at all, while if the confidence is 100%, then the existing reasoning model fits perfectly. If the reliability is within a specific range, it indicates that the endoscopic image that is the target of inference does not exactly fit the existing inference model, and that the existing inference model has room for improvement.
  • step S83 determines whether the reliability is not within the specific range (40 to 60%), it is determined otherwise (S73), and the process proceeds to step S28 in FIG. 5B.
  • the result of determination in step S83 is that the reliability is within the specified range, the current state is to be improved (S65), and the process proceeds to S29 in FIG. 5B.
  • step S67 it is determined whether or not the model is completely unknown in consideration of differences in image quality characteristics and surgical procedures.
  • the input image is an image acquired by a completely unknown endoscope model, is not significantly different from an existing endoscope, or is acquired by an endoscope model known in the market. It is determined whether the image is Since the determination result in step S61 is No, it can be said that the input image is an image acquired by an endoscope other than the existing model (the model used to generate the existing inference model). Therefore, the control unit 11 determines whether or not the model is completely unknown in consideration of the difference in image quality characteristics and the technique.
  • the model When considering the difference in image quality characteristics and surgical procedure, the model, image quality (optical characteristics), illumination, angle of view, operation state, treatment tool, etc., or reliability are taken into account.
  • the inference model differs depending on the surgical procedure. Therefore, the new inference model or the current improved model may be switched depending on the surgical procedure.
  • step S67 if model information is associated with the endoscope image received by the information processing system 10, based on this information, an endoscope model that is not significantly different from the model that generated the existing inference model is used. It is determined whether the model is present or completely unknown.
  • the image quality (optical characteristics) of the image, the illumination of the image, the angle of view, and the like show the characteristics of the manufacturer, so the model can be determined based on this information as well.
  • the model can be determined based on the obtained images (especially moving images). Furthermore, the model can be determined by the reliability of the inference.
  • step S69 As a result of the determination in step S67, if it is determined that the endoscopic image was not acquired by a completely unknown model, it is set to new model 1 (S69). Although the acquired endoscopic image is not that of an existing model, it is not significantly different from that of an existing model. Therefore, it can be said that a new inference model that improves the existing inference model should be generated. Therefore, in step S69, generation of the new model 1 is designated to improve the current state (S75), and the process proceeds to step S29 in FIG. 5B.
  • step S67 if it is determined that the endoscopic image was acquired by a completely unknown model, the new model 2 is used (S69). Since the acquired endoscopic images are completely different from existing models, a new inference model is generated. Therefore, in step S71, generation of the new model 2 is designated, the new model is selected (S77), and the process proceeds to step S35 in FIG. 5B.
  • the endoscopic image is determined based on the image quality characteristics and surgical procedure, such as the model information, reliability range, image quality, illumination, angle of view, operation processing, and treatment tools. is determined, and based on these determination results, the inference model to be generated is separated. That is, it is possible to efficiently use the images collected by the information processing system 10 and acquired by various endoscopes according to the characteristics of these images.
  • the learning device information processing device
  • the learning section 18 may have a control processor such as a CPU, and the control section 11 may control each section in the information processing system 10 so as to realize this flow.
  • the learning unit 18 may of course be provided outside the information processing system 10 instead of being provided inside the information processing system 10 .
  • the target scene is a scene in which a target site such as a lesion such as cancer is being observed, and is a scene to be learned besides observation.
  • the information processing system 10 determines whether or not the image is the target image based on the scene of the image (for example, is it an image of the inside of the body and is it an image during close examination by a doctor or the like?). This determination may be made based on the model information associated with the image, as described above, or based on the image quality (optical characteristics) or other information (see S67 in FIG. 6).
  • step S81 If the result of determination in step S81 is that the scene is not the target scene, it is recorded as the first timing image group (S83).
  • the acquired endoscopic image is a non-target scene, for example, from when the endoscope is inserted to when the target site is searched for and then until the endoscope is removed, that is, the first It is an image in timing.
  • the information processing system 10 records the acquired image in the first timing image section 31c of the recording section 30 as a first timing image.
  • step S81 determines whether it is the target scene. If the result of determination in step S81 is that it is the target scene, it is recorded as a second timing image group (S85). In this case, since it is an image of the target scene, it is often an image suitable for generating an inference model. In the case of an image captured by a model different from the one used when creating an existing inference model, it becomes an image for creating a new inference model.
  • the information processing system 10 records it in the second timing image section 31d of the recording section 30 as a second timing image.
  • the annotation result is recorded (S87).
  • the learning unit 18 (control unit 11) requests that an annotation indicating the position of a lesion such as cancer be added to an image, and the image with this annotation, that is, the teacher data is imaged. Record in file 31A. Annotations may be added using AI or the like.
  • learning is performed (S89).
  • machine learning is performed so as to output the position of a lesion such as cancer when the annotated teacher data acquired in step S67 is applied to the neural network of the inference engine. Set weights.
  • step S91 it is next determined whether or not the learning result has ensured high reliability. This judgment is made by inputting test data into the inference model and comparing the extent to which the error falls within, or the extent to which test data falls within a specific error, for example, with a predetermined reference value. By doing so, it is possible to determine whether or not high reliability is ensured.
  • step S71 If the result of determination in step S71 is that high reliability cannot be ensured, images are sorted out (S93). Here, the images that have lowered the reliability are eliminated, and the images that can improve the reliability are added. Once the images have been sorted out, the process returns to step S81 and the above operations are repeated.
  • step S91 determines whether the result of determination in step S91 is that high reliability can be secured.
  • the inference model, specifications, version, etc. are recorded (S95). Since a highly reliable inference model has been created, this inference model, specifications for generating this inference model, version information, etc. are recorded in the recording unit 30 or the like. After recording these, the inference model is determined and the operation of the learning device is completed.
  • the created inference model can be used when inferring a lesion such as cancer when the information processing system 10 receives an image of the same type as the image recorded as the inspection image group.
  • the information processing system 10 when the information processing system 10 performs learning for generating an inference model, it is determined whether or not the input endoscopic image is the target scene ( S81), if it is a target scene, this image is collected as a second timing image (S85), teacher data is created using this image, and an inference model is created (S87-S89). Therefore, the images collected by the information processing system 10 can be used efficiently.
  • the first timing image is not directly used for learning (S89) for creating an inference model, but it contains information indicating what kind of image can be adopted for the inference model in comparison with the second timing image. I'm in. When creating a new inference model, this information can be used to select an image for creating a new inference model from among the inspection image group 32 accumulated so far.
  • the idea in this embodiment is that there is an inference model with a specific specification, and if the teacher data when learning and creating it and the history of the source video etc. can be unraveled, inference with the same specification
  • This concept can also be realized by an embodiment that infers the image frame of this teacher data candidate. Therefore, using FIG. 8, when there is a series of images (a moving image or a plurality of still images), an inference model is generated that selects one of the images as an image for creating teacher data. Inference using an inference model is explained.
  • an object to be detected is captured from a myriad of images. You have selected an image that you are using. That is, by annotating and learning images selected for creating teacher data from among a series of images, an inference model for extracting images to be annotation candidates can be generated.
  • FIG. 8 shows a learning method (see FIG. 8(a)) for creating an inference model that selects annotation candidate images using inspection videos (still image frames) obtained so far, and how this learning
  • the state of input/output in the generated inference model (FIG. 8(b)) is shown.
  • a series of inspection videos Pm1 to Pm3 are inspection images as teacher data.
  • Each of the series of inspection videos Pm1 to Pm3 is treated as one image, and if one of the images is designated as an image to be annotated, it is possible to detect where it is in the same way as face detection technology in photographic images. It is possible to infer
  • the image Pmi1 is a group of images at the time of insertion
  • the image Pma1 is a group of images to be annotated
  • the image Pmr1 is a group of images at the time of removal.
  • an annotation candidate image is selected for that image. Annotate for creating an inference model.
  • FIG. 8A shows only three inspection videos Pm1 to Pm3, it indicates that there are many inspection videos.
  • endoscopic video information inspection image Pm1 as teacher data in the drawing
  • frames frames (frames) corresponding to the teacher image in the endoscopic video
  • moving image information obtained by using an inspection image Pma1) to be annotated as annotation information is used as training data
  • similar training data is created for endoscopic moving images of a plurality of inspection results ( Inspection images Pma2 and Pma3) as teacher data in FIG. 8(a).
  • an endoscopic moving image including inserted or removed images is input to the input layer In of the neural network NNW, and learning is performed so that the output layer Out becomes teacher data frames (Pma1 to Pma3). Go and create an inference model.
  • the generated inference model is set to InEn in the inference engine shown in FIG. 8(b).
  • the examination video may include treatment time information Tix1, Tix2, and the like.
  • each image included in the video is arranged in chronological order, and judged as a huge image data that can be represented as if it were a panoramic composite image. good. Also, if treatment information can be used effectively, it can also be used at the time of inference to infer annotation candidates.
  • the inference model determines the image group at the first timing and the image group at the second timing, and determines the image group at the second timing or among them. is determined by inferring the training data candidate frames.
  • the examination moving image contains similar image information, so it is possible to make an accurate inference.
  • a teacher image of this endoscopic video is provided for endoscopic video information (see, for example, inspection images Pm1 to Pm3, etc.) obtained during a specific examination.
  • the video information obtained by using the frame to be displayed as annotation information is used as teacher data, and the endoscopic video images of multiple inspection results are similarly converted into teacher data (for example, refer to annotation images Pma1 to Pma3).
  • input the endoscopic video including the inserted or removed image to the learning device (see input layer In of neural network NNW), so that frames of teacher data are output (see output layer Out of neural network) , is trained to create an inference model.
  • the information processing apparatus uses the first endoscope to determine a specific image feature for an image acquired from the first endoscope. It is determined whether the image group obtained in time series by is the image group obtained at the first timing or the image group obtained at the second timing, and the image group obtained at the second timing It is possible to cooperate with a learning unit that obtains a first inference model for image feature determination of a first endoscopic image that has been learned using a result of annotating a group of images as teacher data.
  • This information processing apparatus utilizes an image group obtained at a first timing when a first inference model (existing inference model) is created to perform the newly acquired first or second endoscopy. It has a classifier for classifying images from the mirror. Therefore, images for generating the second inference model different from the first inference model can be efficiently collected.
  • the group of images obtained in chronological order by the first endoscope may be a group of images obtained at the first timing (images during access) or a group of images obtained at the second timing (cancer, etc.).
  • the image group for determining which one is the image when searching for and scrutinizing the lesion area) and generating the second inference model is not limited to the second timing, but at the first timing Select an image from the obtained images. For this reason, the image at the time of access can also be used, and the image for the second inference model can be efficiently collected.
  • a dividing step of dividing the inspection video obtained from the first endoscope according to timing eg, first timing or second timing
  • An image classification step of removing frames corresponding to the first timing from the teaching data candidates and adopting frames corresponding to the second timing as teaching data candidates see, for example, S35 and S37 in FIG. 5B.
  • a recording step (see, for example, S37 in FIG. 5B) of classifying and recording the inspection moving image for each division timing, and at least one of the teaching data candidates are used to record the affected area information included in the frames of the inspection moving image.
  • It has a step of learning a first inference model for inference (for example, see S89 in FIG. 7). Therefore, images for generating the second inference model different from the first inference model can be efficiently collected and the inference model can be generated.
  • an annotation step in which the result of annotating the frame corresponding to the second timing is used as a teacher data candidate (see, for example, S85 and S87 in FIG. 7).
  • a step of acquiring a new inspection video see, for example, S81 of FIG. 7
  • It has a selection step for example, see S85, S87, etc. in FIG. 7 of selecting training data candidates for the second inference model by classifying the new inspection moving image into a classification that reflects the image classification result.
  • the technique of selecting teacher image candidates from time-series image frames is expected to be utilized in various fields.
  • an example of insertion and withdrawal from a body cavity, which is unique to an endoscope, has been described, and moving images obtained in the process of examination are classified and analyzed in chronological order for each timing.
  • a so-called normalized procedure is not limited to endoscopic imaging, as it can be in any field.
  • the medical field it can be applied to other image diagnostic equipment such as ultrasound and radiation, and it can also be applied to emergency judgment in the operating room.
  • the process of finding specific data from a large amount of data obtained over time is laborious.
  • the present embodiment can be applied to any technology that requires teacher data for creating an inference model and selects candidate information to be turned into teacher data from a large amount of information.
  • the video frame part sandwiched in a specific process can be used to create structures such as roads, bridges, and tunnels.
  • various situation determination inference models are required for relationships with other vehicles, signs, etc., but the concept of this embodiment can be applied.
  • the information processing system 10, the image system 20, and the recording unit 30 are described as separate units, but these two may be integrated, or three may be integrated. may be configured.
  • the controllers 11 and 21 have been described as devices configured from CPUs, memories, and the like. However, in addition to being configured as software by a CPU and a program, part or all of each part may be configured as a hardware circuit, and is described in Verilog, VHDL (Verilog Hardware Description Language), etc. A hardware configuration such as a gate circuit generated based on a program language may be used, or a hardware configuration using software such as a DSP (Digital Signal Processor) may be used. Of course, these may be combined as appropriate.
  • DSP Digital Signal Processor
  • control units 11 and 21 are not limited to CPUs, and may be elements that function as controllers, and the processing of each unit described above may be performed by one or more processors configured as hardware.
  • each unit may be a processor configured as an electronic circuit, or may be each circuit unit in a processor configured with an integrated circuit such as an FPGA (Field Programmable Gate Array).
  • FPGA Field Programmable Gate Array
  • a processor composed of one or more CPUs may read and execute a computer program recorded on a recording medium to execute the function of each unit.
  • the information processing system 10 includes a control unit 11, an input determination unit 12, an image similarity determination unit 13, a metadata addition unit 14, a classification unit 15, a first request unit 16, a second
  • a control unit 11 an input determination unit 12, an image similarity determination unit 13, a metadata addition unit 14, a classification unit 15, a first request unit 16, a second
  • the requesting unit 17 and the learning unit 18 are provided. However, they do not need to be provided in an integrated device, and the above-described units may be distributed as long as they are connected by a communication network such as the Internet.
  • control described mainly in the flowcharts can often be set by a program, and may be stored in a recording medium or recording unit.
  • the method of recording in the recording medium and the recording unit may be recorded at the time of product shipment, using a distributed recording medium, or downloading via the Internet.
  • the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the spirit of the present invention at the implementation stage. Also, various inventions can be formed by appropriate combinations of the plurality of constituent elements disclosed in the above embodiments. For example, some components of all components shown in the embodiments may be omitted. Furthermore, components across different embodiments may be combined as appropriate.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Public Health (AREA)
  • Biomedical Technology (AREA)
  • Pathology (AREA)
  • Epidemiology (AREA)
  • Primary Health Care (AREA)
  • Endoscopes (AREA)
PCT/JP2021/025970 2021-07-09 2021-07-09 情報処理装置および情報処理方法 Ceased WO2023281738A1 (ja)

Priority Applications (6)

Application Number Priority Date Filing Date Title
JP2023533019A JP7688704B2 (ja) 2021-07-09 2021-07-09 情報処理装置および情報処理方法
PCT/JP2021/025970 WO2023281738A1 (ja) 2021-07-09 2021-07-09 情報処理装置および情報処理方法
CN202180098769.XA CN117480572A (zh) 2021-07-09 2021-07-09 信息处理装置以及信息处理方法
US18/537,951 US12592062B2 (en) 2021-07-09 2023-12-13 Information processing device and information processing method
JP2025086428A JP2025119024A (ja) 2021-07-09 2025-05-23 情報処理方法および情報処理装置
US19/557,581 US20260196029A1 (en) 2021-07-09 2026-03-05 Information processing device and information processing method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/025970 WO2023281738A1 (ja) 2021-07-09 2021-07-09 情報処理装置および情報処理方法

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/537,951 Continuation US12592062B2 (en) 2021-07-09 2023-12-13 Information processing device and information processing method

Publications (1)

Publication Number Publication Date
WO2023281738A1 true WO2023281738A1 (ja) 2023-01-12

Family

ID=84800617

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/025970 Ceased WO2023281738A1 (ja) 2021-07-09 2021-07-09 情報処理装置および情報処理方法

Country Status (4)

Country Link
US (2) US12592062B2 (cg-RX-API-DMAC7.html)
JP (2) JP7688704B2 (cg-RX-API-DMAC7.html)
CN (1) CN117480572A (cg-RX-API-DMAC7.html)
WO (1) WO2023281738A1 (cg-RX-API-DMAC7.html)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024181506A1 (ja) * 2023-02-28 2024-09-06 京セラ株式会社 情報処理システム、情報処理システムの制御方法、制御プログラム、記録媒体

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12475989B2 (en) * 2020-12-16 2025-11-18 Koninklijke Philips N.V. Image selection for presentation
US12525337B2 (en) * 2022-02-09 2026-01-13 Lunit Inc. Method and apparatus for selecting medical data for annotation

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020110278A1 (ja) * 2018-11-30 2020-06-04 オリンパス株式会社 情報処理システム、内視鏡システム、学習済みモデル、情報記憶媒体及び情報処理方法
WO2020194472A1 (ja) * 2019-03-25 2020-10-01 オリンパス株式会社 移動支援システム、移動支援方法、および移動支援プログラム
JP2021048928A (ja) * 2019-09-20 2021-04-01 株式会社Micotoテクノロジー 内視鏡画像処理システム
JP2021100555A (ja) * 2019-12-24 2021-07-08 富士フイルム株式会社 医療画像処理装置、内視鏡システム、診断支援方法及びプログラム

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3480714A1 (en) 2017-11-03 2019-05-08 Tata Consultancy Services Limited Signal analysis systems and methods for features extraction and interpretation thereof
US11308614B2 (en) * 2018-03-20 2022-04-19 EndoVigilant Inc. Deep learning for real-time colon polyp detection
JP2020067592A (ja) * 2018-10-25 2020-04-30 オリンパス株式会社 移動量判定装置、移動量判定方法、及び移動量判定プログラム
EP3936027A4 (en) * 2019-03-08 2022-04-27 FUJIFILM Corporation DEVICE AND METHOD FOR PROCESSING MEDICAL IMAGES
JP7200030B2 (ja) * 2019-03-29 2023-01-06 オリンパス株式会社 学習装置、撮像装置、学習システム、ai情報提供装置、学習方法及び学習プログラム
WO2020256568A1 (en) * 2019-06-21 2020-12-24 Augere Medical As Method for real-time detection of objects, structures or patterns in a video, an associated system and an associated computer readable medium
WO2021085258A1 (ja) * 2019-10-31 2021-05-06 キヤノン株式会社 画像処理装置、画像処理装置の制御方法、識別器の生成方法、識別方法、識別装置、識別器の生成装置、及び識別器
CN111126574B (zh) * 2019-12-30 2023-07-28 腾讯科技(深圳)有限公司 基于内镜图像对机器学习模型进行训练的方法、装置和存储介质
CN112016451B (zh) * 2020-08-27 2024-09-17 贵州师范大学 一种用于迁移学习的训练样本标注成本削减方法
CN112466466B (zh) * 2021-01-27 2021-05-18 萱闱(北京)生物科技有限公司 基于深度学习的消化道辅助检测方法、装置和计算设备
JPWO2024181506A1 (cg-RX-API-DMAC7.html) * 2023-02-28 2024-09-06

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020110278A1 (ja) * 2018-11-30 2020-06-04 オリンパス株式会社 情報処理システム、内視鏡システム、学習済みモデル、情報記憶媒体及び情報処理方法
WO2020194472A1 (ja) * 2019-03-25 2020-10-01 オリンパス株式会社 移動支援システム、移動支援方法、および移動支援プログラム
JP2021048928A (ja) * 2019-09-20 2021-04-01 株式会社Micotoテクノロジー 内視鏡画像処理システム
JP2021100555A (ja) * 2019-12-24 2021-07-08 富士フイルム株式会社 医療画像処理装置、内視鏡システム、診断支援方法及びプログラム

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024181506A1 (ja) * 2023-02-28 2024-09-06 京セラ株式会社 情報処理システム、情報処理システムの制御方法、制御プログラム、記録媒体

Also Published As

Publication number Publication date
JP7688704B2 (ja) 2025-06-04
CN117480572A (zh) 2024-01-30
JPWO2023281738A1 (cg-RX-API-DMAC7.html) 2023-01-12
JP2025119024A (ja) 2025-08-13
US20260196029A1 (en) 2026-07-09
US12592062B2 (en) 2026-03-31
US20240112450A1 (en) 2024-04-04

Similar Documents

Publication Publication Date Title
Nakawala et al. “Deep-Onto” network for surgical workflow and context recognition
US12592062B2 (en) Information processing device and information processing method
US12451239B2 (en) Systems and methods for assessing surgical ability
JP6927211B2 (ja) 画像診断学習装置、画像診断装置、方法およびプログラム
US12137879B2 (en) Image processing apparatus, image processing method, and endoscope apparatus
KR102427171B1 (ko) 영상 내의 객체 레이블링 제공 방법 및 이를 위한 장치
KR20220130855A (ko) 인공 지능 기반 대장 내시경 영상 진단 보조 시스템 및 방법
JP7705391B2 (ja) 機械学習モデルをトレーニングまたは用いるためのシステム、顕微鏡システム、方法およびコンピュータプログラム
Gong et al. Using deep learning to identify the recurrent laryngeal nerve during thyroidectomy
Jaafari et al. Towards more efficient CNN-based surgical tools classification using transfer learning
KR20210054140A (ko) 내시경 영상에 대한 복수의 의료 영상 판독 알고리듬들을 이용하는 의료 영상 판독 지원 장치 및 방법
CN118119329A (zh) 内窥镜插入引导装置、内窥镜插入引导方法、内窥镜信息取得方法、引导服务器装置及图像推导模型学习方法
Binol et al. OtoXNet—automated identification of eardrum diseases from otoscope videos: a deep learning study for video-representing images
Konduri et al. Full resolution convolutional neural network based organ and surgical instrument classification on laparoscopic image data
WO2024180385A1 (en) Method for diagnosing pancreatic lesions using ultrasound images
CN120783933A (zh) 一种内窥镜报告生成方法、装置、设备及可读存储介质
CN117678028A (zh) 使用对象检测与活动估计的医学视频注释
Taguelmimt et al. Uncertainty-aware deep learning classification for MRI-based prostate cancer detection
CN113570592A (zh) 肠胃病检测和模型训练方法、装置、设备及介质
KR20240159306A (ko) 딥러닝 모델을 활용한 내시경 영상 분석 서비스 제공 방법 및 장치
WO2022190170A1 (ja) 推論用学習装置および推論用学習方法
JP7245906B2 (ja) 画像記録装置、情報処理装置、情報処理方法および情報処理プログラム
JP2019013461A (ja) プローブ型共焦点レーザー顕微内視鏡画像診断支援装置
CN115428011A (zh) 估计用学习装置和估计用学习方法
Dhyani et al. Survey on Prediction of Cardiomegaly Disease Using Convolutional Neural Networks (CNN)

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21949370

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023533019

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 202180098769.X

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21949370

Country of ref document: EP

Kind code of ref document: A1