JP2021048928A - Endoscopic image processing system - Google Patents

Abstract

To provide a technique capable of assisting an endoscopic operation using an endoscopic image.SOLUTION: An endoscopic image processing system comprises output processing means which, on the basis of area designation data acquired by giving an acquired endoscopic image to a learned model, outputs a display obtained by adding guide information of an endoscope relating to an image area designated by the area designation data within the endoscopic image to the endoscopic image. The learned model machine-learned by using a plurality of pieces of teacher data obtained by respectively associating a correct answer of the area designation data with each endoscopic image for teachers.SELECTED DRAWING: Figure 3

Description

The present invention relates to a technique for processing an image captured by an endoscope (referred to as an endoscopic image).

Endoscopy is widely used in the medical field, and there are many types of endoscopes such as upper gastrointestinal endoscope, colonoscope, bronchoscope, thoracoscope, vascular endoscope, capsule endoscope, etc., depending on the organ and purpose of use. A type of endoscope is provided. In addition, there are endoscopes that not only allow viewing of images inside organs, but also enable tissue collection, polyp resection, and the like.

In endoscopy, the insertion part of the endoscope must be inserted into the organ without damaging the inside of the organ, and the lesion must be detected without missing it. You will need it.
Therefore, in Patent Document 1 below, a learning image group classified into each pathological type is stored, and the feature amount matching between the image of the identification target region of the obtained endoscopic image and the learning image group is performed. An endoscopic diagnostic imaging support system that identifies each pathological type has been proposed.

Japanese Unexamined Patent Publication No. 2015-146970

According to the above-mentioned system, the accuracy of finding lesions in endoscopic images can be improved.
However, in order to capture an endoscopic image in which a lesion can be detected, it is necessary to operate the endoscope properly.
On the other hand, when a trainee tries to learn a technique for operating an endoscope properly, he / she must receive training under the guidance doctor, which causes problems such as an increase in the burden on the guidance doctor. In addition, there are problems that the operation of the endoscope operated by the doctor by himself / herself is good or bad and that the proficiency level of his / her own endoscopic technique cannot be judged by himself / herself.

The present invention has been made in view of such circumstances, and provides a technique capable of supporting endoscopic operation using an endoscopic image.

The endoscopic image processing system according to one aspect of the present invention adopts the following configuration. That is, the endoscopic image processing system obtains an image acquisition means for acquiring an endoscopic image captured by an endoscope in a lumen organ, and the acquired endoscopic image for a trained model. By giving, a model processing means for acquiring the area designation data corresponding to the endoscope image, and an image designated by the area designation data in the endoscope image based on the acquired area designation data. It is equipped with an output processing means that outputs a display in which the guide information of the endoscope regarding the region is added to the endoscope image, and the trained model is the region designation data for each teacher's endoscope image. Machine learning is performed using multiple teacher data in which the correct answers are associated with each other.
The endoscopic image processing system may be one device or a plurality of devices.
Further, the trained model may be provided in the endoscopic image processing system or may be provided outside.
In addition, the endoscopes that capture the endoscopic images to be processed by the endoscopic image processing system are upper gastrointestinal endoscopy, colonoscope, bronchoscope, thoracic endoscope, vascular endoscope, and capsule. It is an endoscope, etc., and is not limited.
The luminal organ shown in the endoscopic image may be an organ model that imitates an organ in a human body model, or may be an actual luminal organ of a living body.

According to the present invention, it is possible to provide a technique capable of supporting the operation of an endoscope using an endoscopic image.

It is a figure which shows the appearance of a part of the endoscopic technique trainer system which concerns on this embodiment. It is a figure which conceptually shows the control structure of the endoscopic technique trainer system which concerns on this embodiment. It is a block diagram which conceptually shows the software structure realized by a control part. It is a schematic diagram of the esophagus, stomach and duodenum for explaining the region position data used in the positioning AI model. FIG. 5 is a schematic cross-sectional view of a slice of the stomach for explaining the region orientation data used in the positioning AI model. It is a figure for demonstrating the 1st region designation data used in the 1st guide AI model. It is a figure for demonstrating the 2nd region designation data inferred by the 2nd guide AI model. It is a figure which shows the display example of the guide information of an endoscope. It is a figure which shows the example of the guide display of an observation point. It is a flowchart which shows the operation example of a control part. It is a figure which conceptually shows the control structure of the endoscope system which concerns on other embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments listed below are examples, and the present invention is not limited to the configurations of the following embodiments.

Hereinafter, an endoscopic procedure trainer system as an embodiment of the present invention will be described.
FIG. 1 is a diagram showing the appearance of a part of the endoscopic procedure trainer system according to the present embodiment.
FIG. 2 is a diagram conceptually showing a control configuration of the endoscopic procedure trainer system according to the present embodiment.
The endoscopic procedure trainer system (hereinafter referred to as this system) 1 according to the present embodiment is mainly composed of a human body model 3, an input / output panel 5, a control unit 10, and the like. Enables individual learning and individual training. Specifically, in this system 1, a person to be trained (hereinafter referred to as a trainee) is output to an input / output panel 5 or the like while actually training an endoscopic technique using a human body model 3. By referring to the guide information, the endoscopic technique can be self-taught.
In this way, the system 1 can instruct trainees such as residents in endoscopic procedures, so that the burden on the instructor can be reduced.
In the following, we will mainly explain the configuration of the upper gastrointestinal endoscopy procedure. However, the medical procedures that enable self-learning and self-training by this system 1 include colonoscopy, enteroscopy, biliary / pancreatic endoscopy, and these endoscopic treatment techniques. , Other intubation procedures may also be included.

[Human model]
The human body model 3 is a humanoid model operated by a trained person (hereinafter referred to as a trainee), and has a shape imitating the outer shape and internal organs of the human body. In the present embodiment, as shown in FIG. 1, the human body model 3 simulates the outer shape of the entire human body and, as internal organs, the oral cavity, nasal cavity, pharynx, larynx, trachea, esophagus, bronchus, stomach, and duodenum. The shape of the nasal organ is simulated internally.
The human body model 3 is placed on the pedestal portion in a posture corresponding to the procedure for training purposes. For example, during training for tracheal intubation, the mannequin 3 is placed on the pedestal in a lying position, and during training for endoscopic procedures, the mannequin 3 is placed on the pedestal in a sideways position, as shown in FIG. It will be placed.

The outer surface of the human body model 3 is covered with a skin sheet that imitates the outer shape of the human body, and a wig is attached to the head.
The skin sheet is made of a flexible material such as silicone rubber. The term "flexibility" as used herein means a property that is unlikely to break or be damaged even when bent, and may include one or both of stretchability and elasticity.

The internal structure inside the skin sheet in the human model 3 is composed of an internal organ structure, a skeleton base, and the like. The skeleton base portion is a group of components forming a framework that forms the basis of the shape of the human body model 3, and is formed of a material having strength and hardness that can withstand operations by a trainee, such as metal and synthetic resin. The skeletal base includes skeletal members corresponding to the skull, cervical spine, and the like.
The internal organ structure is a group of components having a shape imitating a luminal organ, and is connected and fixed to a skeletal base portion at an arbitrary position and method. The internal organ structure includes an internal modeling part (not shown).

In the present embodiment, the in-vivo model includes an oral cavity, a nasal cavity, an oral cavity, a pharynx, a larynx, a trachea, an esophagus, a stomach, and a duodenum. , And the duodenal model.
Such an in-vivo shaped portion is formed of a material having flexibility close to that of a living body's luminal organ, such as silicone rubber. Further, in order to realistically reproduce the luminal organ, the internal molding portion is preferably integrally molded using a flexible material so as to eliminate joints as much as possible.

As described above, in the human body model 3, it is preferable that the skeleton base portion and the internal organ structure simulate the living body with high accuracy, but in the present embodiment, the specific shape, material, manufacturing method, etc. of the human body model 3 and the like. Does not limit. In this embodiment, various known human body models 3 can be used.
For example, depending on the medical procedure to be trained, the mannequin 3 may simulate the outer shape of the human body only in the upper body, or the gastrointestinal tract such as the large intestine, small intestine, gallbladder, and bladder, and urethra as the internal modeling part. Other luminal organs, such as the ureteral system such as the canal, bladder, and urethra, may be simulated.

In the present embodiment, object detection sensors for detecting the presence of an endoscope inserted into the internal modeling portion are provided at a plurality of predetermined portions of the internal modeling portion. Each object detection sensor is located at a position where it does not come into contact with the inserted endoscope (for example, outside the inner wall surface defining the lumen of the tract organ model) so as not to give a sense of discomfort to the endoscope operator. Provided. In the present embodiment, a photoelectric sensor is used as the object detection sensor, and is provided at the esophageal entrance, the gastroesophageal junction, and the duodenal descending leg.
However, the object detection sensor may not be provided, may be provided at a portion different from the present embodiment, and the number thereof is not limited. Further, the object detection principle by the object detection sensor is not limited.

[Control configuration]
The control unit 10 has a configuration that controls the control of the system 1, and may be a so-called computer such as a PC (Personal Computer), an embedded system, or a control board.
The control unit 10 is housed in a device mounting table having a pedestal on which the human body model 3 is placed, together with the input / output panel 5, the speaker 6, and the like.
As shown in FIG. 2, the control unit 10 has a processor 11, a memory 12, an input / output interface (I / F) unit 13, and the like as a hardware configuration.
The processor 11 may be one or more general CPUs or MPUs (Micro Processing Units), and may be replaced with or together with an integrated circuit (ASIC) for a specific application, a DSP (Digital Signal Processor), and a GPU (Graphics). It may be a Processing Unit), an FPGA (Field Programmable Gate Array), or the like.

The memory 12 is a RAM (Random Access Memory) and a ROM (Read Only Memory), and may include an auxiliary storage device (hard disk or the like). A control program that realizes various functions of the system 1 is stored in the memory 12. The control program also includes an AI (Artificial Intelligence) model.
Here, the AI model can be described as a machine learning (ML) model or a trained model, and can also be described as a combination of a computer program and parameters, or a combination of a plurality of functions and parameters. ..
This embodiment does not limit the structure and learning algorithm of the AI model as long as it is an AI model of supervised machine learning. For example, the AI model has a structure in which a plurality of nodes are connected by edges in each layer of an input layer, an intermediate layer (hidden layer), and an output layer, and the value of each node is connected to that node of another node. It may be realized by a model called a neural network (NN) or a deep neural network (DNN) calculated by an activation function from values and edge weights (nodes). Further, in the present embodiment, since the AI model uses an endoscopic image, it may be realized by a convolutional neural network (CNN).
The "AI model" in the present specification may refer to one neural network when it is constructed by a neural network and when the input layer, the intermediate layer and the output layer are regarded as one neural network unit. Alternatively, it may refer to a combination of a plurality of neural networks.

The input / output I / F unit 13 is a device that controls the input or output of a signal to be processed or processed by the processor 11, and is a user interface device such as an input / output panel 5, a speaker 6, a sensor group 7, and an internal view. It is connected to a mirror 8 or the like. The connection form between the endoscope 8 and the sensor group 7 and the input / output I / F unit 13 may be not only wired but also wirelessly connected so as to be communicable. Further, the input / output I / F unit 13 may include a communication unit that communicates with another computer or device, and may be connected to a portable recording medium or the like.

The sensor group 7 is a plurality of various sensors provided inside or outside the human body model 3, and includes the above-mentioned object detection sensor. The sensor group 7 may include a sensor other than the object detection sensor, such as a pressure sensor provided at a predetermined portion of the internal modeling portion and an atmospheric pressure sensor for detecting the atmospheric pressure in the lumen of the internal modeling portion.
The input / output panel 5 is installed above the equipment mounting base, and operates a training menu, a display device that displays the operation mode of the system 1, implementation details, evaluation results, etc., and a screen displayed on the display device. Includes an input device for. In the example of FIG. 1, the input / output panel 5 is realized as a touch panel in which a display device and an input device are integrated. The display contents of the input / output panel 5 will be described later.

In the present embodiment, the endoscope 8 connected to the input / output I / F unit 13 is an upper gastrointestinal endoscope. The upper gastrointestinal endoscope is composed of an insertion portion including a tip portion and a curved portion, an operation unit for performing various operations on the tip portion and the curved portion, an image processing device, and the like. The operation unit is provided with an angle knob (left / right angle and up / down angle), a suction button, an air supply / water supply button, a forceps insertion port, and the like. However, the endoscope 8 connected to the input / output I / F unit 13 may be a colonoscope, a bronchoscope, a thoracoscope, a blood vessel endoscope, or the like, other than the upper gastrointestinal endoscope.
Further, the control unit 10 may include hardware elements (not shown in FIG. 2), and the hardware configuration of the control unit 10 is not limited.

When the control program stored in the memory 12 is executed by the processor 11, the control unit 10 receives the input signals from the sensor group 7 and the endoscope 8 and displays, outputs, and outputs to the input / output panel 5. The input information is acquired from the panel 5, the audio output is controlled from the speaker 6, and the like.
The control program may be stored in advance at the time of shipment, or may be stored in advance from a portable recording medium such as a CD (Compact Disc) or a memory card or another computer on the network via the input / output I / F unit 13. It may be installed and stored in the memory 12.

[Software configuration]
FIG. 3 is a block diagram conceptually showing a software configuration realized by the control unit 10.
By executing the control program stored in the memory 12 by the processor 11, the control unit 10 realizes the software configuration as shown in FIG. Specifically, the control unit 10 has an image processing module 21, an AI processing module 22, a storage processing module 23, an output processing module 24, and the like as software configurations. The image processing module 21 can be described as an image acquisition means, the AI processing module 22 can be described as a first, second or third model processing means, and the storage processing module 23 can be described as a storage means. The output processing module 24 can be described as an output processing means.
However, since each software component shown in FIG. 3 is conceptually shown separately for convenience of explanation, the software configuration realized by the control unit 10 is as shown in FIG. It does not have to be clearly separated into each component.

This system 1 estimates the position and orientation of the endoscope 8 and estimates the orientation of the endoscope 8 by the operation of the control unit 10 based on the software configuration as described above in order to enable individual learning and individual training of the endoscope procedure by the trainee. Guide the endoscope 8.
Here, the "position and orientation of the endoscope" specifically means the endoscope 8 inserted in the luminal organ model of the human body model 3 (here, mainly the esophageal model, the stomach model and the duodenum model). It means the position and the imaging direction of the imaging element provided at the tip of the image.
The guide of the endoscope 8 outputs information that guides or guides what the tip of the endoscope 8 should do at any given time. By operating the endoscope 8 while referring to the guide information of the endoscope 8 output by the system 1, the trainee can independently learn and train the endoscope technique without an instructor. ..
Hereinafter, the process related to "estimation of the position and orientation of the endoscope" and the process related to "guide of the endoscope" executed by the control unit 10 will be described in detail.

When the control unit 10 executes the process, the positioning AI model 31, the first guide AI model 32, and the second guide AI model 33 are stored in the memory 12. The positioning AI model 31, the first guide AI model 32, and the second guide AI model 33 can be described as the first, second, or third trained model.
The positioning AI model 31, the first guide AI model 32, and the second guide AI model 33 are AI models that have been trained by a supervised machine learning algorithm. Hereinafter, the positioning AI model 31 is referred to as the P-AI model 31, the first guide AI model 32 is referred to as the G1-AI model 32, and the second guide AI model 33 is referred to as the G2-AI model 33. To.
Before explaining the above-mentioned processing of the control unit 10, first, each of these AI models will be described.

The P-AI model 31 is an image classification type AI model in which an endoscope image, which is an image captured by an endoscope, is input and region position data and region direction data corresponding to the endoscope image are inferred. Is. The P-AI model 31 corresponds to the first trained model.
The "region position data" includes one or more organ regions specified based on the inference result of the P-AI model 31 from a plurality of organ regions in which the luminal organs are virtually divided in the long axis direction. It is data that can be identified as the position information of the endoscope 8, and is data for specifying the position of the tip portion of the endoscope 8 in the luminal organ.
The "region direction data" is a plurality of directions (six directions in the present embodiment) indicated by three-dimensional orthogonal axes virtually set for each organ region in which the luminal organ is virtually divided in the long axis direction. It is data that can identify one or more directions specified based on the inference result of the P-AI model 31 as the direction information of the endoscope 8, and is the tip of the endoscope 8 in the luminal organ. It is data for specifying the direction of.

FIG. 4 is a schematic view of the esophagus, stomach and duodenum for explaining the region position data used in the positioning AI model 31, and FIG. 5 is a stomach for explaining the region orientation data used in the positioning AI model 31. It is a cross-sectional schematic diagram of the round slice of.
In the present embodiment, the luminal organs consisting of the esophagus, stomach and duodenum are arranged in the longitudinal direction in the oral esophagus part E0, the gastric side esophagus part E1, the corpus E2, the gastric corpus E3, the angular incisure E4, and the antrum. It is virtually divided into each organ region of E5, duodenal bulb E6, duodenal descending leg E7, and lower duodenal angle E8. Note that FIG. 4 does not show the oral esophageal portion E0 and the lower duodenal angle E8.
The region position data is data that can identify each one or more organ regions specified based on the inference result of the P-AI model 31 from these plurality of organ regions as the position information of the endoscope 8. For example, it is indicated by a numerical value or a character string from E0 to E8.

FIG. 5 shows six directions indicated by three-dimensional orthogonal axes virtually set in the body of stomach E3. Specifically, in the circular section of the gastric body E3, the direction from the posterior wall to the anterior wall (hereinafter referred to as the anterior wall direction) D1 and the direction from the anterior wall to the posterior wall (hereinafter referred to as the posterior wall direction). D2, the direction from the lesser curvature to the lesser curvature (hereinafter referred to as the lesser curvature direction) D3, the direction from the lesser curvature to the lesser curvature (hereinafter referred to as the greater curvature direction) D4, from the esophageal side to the duodenum side The direction (hereinafter referred to as the insertion direction) D5 and the direction from the duodenum side to the esophagus side (hereinafter referred to as the pull-out direction) D6 are shown.
The region direction data is data that can identify one or more directions specified based on the inference result of the P-AI model 31 from the six directions indicated by the three-dimensional orthogonal axes for each organ region, for example. , D1 to D6, or a character string. The three-dimensional orthogonal axis is virtually set not only for the body of the stomach E3 but also for each other organ region, and since the luminal organ does not extend linearly, it is set for each organ region. Each method may be different.
Further, the area direction data may include data indicating that no direction is specified.

The P-AI model 31 is machine-learned based on a plurality of teacher data in which the correct answer of the region position data and the correct answer of the region direction data are associated with the teacher's endoscopic image. Specifically, a plurality of teacher's endoscope images are prepared, and the position and direction of the endoscope that captured each teacher's endoscope image is the above-mentioned region position data and region direction data (for example, by a person). A plurality of teacher data are generated by tagging each teacher's endoscopic image with the specified area position data and area direction data (correct answer data). The P-AI model 31 is trained by a predetermined machine learning algorithm using the plurality of teacher data generated in this way. For example, the P-AI model 31 is trained by CNN's known learning algorithm suitable for image classification.

Here, the endoscopic image for the teacher can be collected by actually performing the endoscopic procedure using the human body model 3 by a skilled doctor who is good at the endoscopic procedure. Multiple patterns of endoscopic images for teachers may be collected by multiple endoscopic procedures by one skilled doctor, or multiple patterns of endoscopic images may be collected by multiple endoscopic procedures by multiple skilled doctors. A group of endoscopic images may be collected.
In the present embodiment, by using the luminal organ model of the human body model 3, the endoscopic image for the teacher can be easily collected, and the correct answer data of the region position data and the region direction data can be easily collected. Can be identified.

The P-AI model 31 learned in this way is obtained for each organ region of the luminal organ when an endoscopic image, which is an image captured by the endoscope 8 inserted in the human body model 3, is input. The probability value of and the probability value of each of the six directions indicated by the three-dimensional orthogonal axes are output. Specifically, the input endoscopic image shows each region (mouth side esophagus part E0, gastric side esophagus part E1, gastroscopy part E2, gastric corpus E3, angular incisure E4, antrum E5, duodenal bulb Probability values classified into E6, descending duodenum E7, and lower duodenal angle E8), and the endoscopic image is in the six directions (anterior wall direction D1, posterior wall direction D2, small curvature direction D3, greater curvature direction D4). , The insertion direction D5, and the pull-out direction D6) are calculated. In other words, the probability value of each organ region indicates the probability that the endoscope 8 exists in the organ region (position), and the probability value in each direction is the direction in which the endoscope 8 is facing. It shows the probability.

As a result, the region position data can identify one or more organ regions having a probability value equal to or higher than a predetermined threshold based on the probability value for each organ region, which is the inference result of the P-AI model 31, or the maximum. It is acquired as data that can identify the organ region having a probability value. The area direction data is data that can identify one or more directions having a probability value equal to or higher than a predetermined threshold based on the probability value for each direction that is the inference result of the P-AI model 31, or a direction having the maximum probability value. Is acquired as data that can identify.

The G1-AI model 32 is an image classification type AI model that inputs an endoscopic image and infers first region designation data corresponding to the endoscopic image. The G1-AI model 32 corresponds to the second trained model.
The "first region designation data" is data that designates (points to) a certain image region in the endoscopic image input to the G1-AI model 32. In the present embodiment, the image area designated by the first area designation data is an image area corresponding to the route to be taken by the endoscope 8 that is capturing the endoscope image. However, the image area designated by the first area designation data is not limited to such an example, and may be an image area corresponding to a place (point) where something should be done with the endoscope 8. For example, even if the image area corresponding to the point to be observed (the part to be imaged and recorded) and the image area corresponding to the point (site) to be treated or tissue is collected are specified by the first area designation data. Good.
Hereinafter, a portion to be observed with the endoscope 8 or to be imaged and recorded may be referred to as an observation point.

FIG. 6 is a diagram for explaining the first region designation data used in the first guide AI model 32.
The endoscopic image input to the G1-AI model 32 is normalized to a predetermined size and shape, and as shown in FIG. 6, a predetermined grid line is used for a plurality of unit image regions. It is virtually divided. In the example of FIG. 6, it is divided into 25 unit image areas of 5 in the vertical direction and 5 in the horizontal direction.
In the first area designation data, any one unit image area or any two unit image areas vertically, horizontally, or diagonally adjacent to each other is designated as an image area. The first area designation data is indicated by, for example, "vertical, horizontal" coordinate values. For example, the first area designation data indicating the image area consisting of the third unit image area from the top in the vertical direction and the fourth unit image area from the left in the horizontal direction is indicated by "3, 4", and is the second from the top in the vertical direction and horizontal. The first area designation data indicating the image area consisting of the third and fourth unit image areas from the left is indicated by "2, 3-2, 4".
In the present embodiment, each unit image area has a quadrangular shape separated by grid lines, but the shape and size of the unit image area are not limited, and even if it is a circular shape, it is triangular. You may.

The G1-AI model 32 is machine-learned using a plurality of teacher data in which the correct answers of the first region designation data are associated with each teacher's endoscopic image. Specifically, a plurality of teacher's endoscope images are prepared, and for each teacher's endoscope image, an image area corresponding to a route to be taken by the endoscope that captured the image is specified (for example, by a person). Then, a plurality of teacher data are generated by tagging each teacher's endoscope image with the correct answer of the first area designation data that specifies the specified image area. The G1-AI model 32 is trained by a predetermined machine learning algorithm using the plurality of teacher data generated in this way. For example, the G1-AI model 32 is trained by a known learning algorithm of CNN suitable for image classification similar to the P-AI model 31.
The method of collecting endoscopic images for teachers may be the same as that of the P-AI model 31.

When an endoscope image, which is an image captured by the endoscope 8 inserted in the human body model 3, is input to the G1-AI model 32 learned in this way, each image in the endoscope image is input. Output the probability value for each area. For example, in the G1-AI model 32, in addition to each unit image area divided by the grid lines in FIG. 6, all combinations of two unit image areas adjacent to the left and right, and two unit image areas adjacent to the top and bottom are used. The probability value is output for each image area of all combinations of the above and all combinations of two diagonally adjacent unit image areas.
That is, the probability value of each image area indicates the certainty that each image area of the target endoscopic image is a route to be taken or a place to do something.
As a result, the region designation data is data that designates one or more image regions having a probability value equal to or higher than a predetermined threshold based on the probability value for each image region that is the inference result of the G1-AI model 32, or the maximum probability. It is acquired as data that specifies the image area that has a value.

The G2-AI model 33 is an image detection type AI model that inputs an endoscopic image and detects an image area tagged with predetermined job information in the endoscopic image. As a result, the AI processing module 22, which will be described later, gives an endoscopic image to the G2-AI model 33, thereby providing second area designation data for designating an image area detected in the endoscopic image. Job information corresponding to the second area designation data can be acquired. The G2-AI model 33 also corresponds to the second trained model.
The "second area designation data" is in agreement with the above-mentioned first area designation data in that it is data that specifies (points to) a certain image area in the endoscopic image. However, since the types of AI models are different and job information is associated with the second area designation data, here, in order to make the explanation easier to understand, the second area designation data is distinguished from the first area designation data. It shall be written as. The second area designation data is sometimes called a Bounding Box.
However, when the term "area-designated data" is used in the description below, it means either one or both of the first area-designated data and the second area-designated data.
Here, the "job information" of the second area designation data is tag information corresponding to the image area designated by the second area designation data, and indicates the route that the endoscope 8 should take, or the endoscope 8 This is information that indicates the place (point) where something should be done. For example, the job information may indicate an observation point, or may indicate a point (site) where some treatment should be performed or tissue collection should be performed.

The G2-AI model 33 is machine-learned using a plurality of teacher data in which the correct answers of the second region designation data tagged with job information are associated with each teacher's endoscope image. Specifically, a plurality of teacher endoscopic images are prepared, and necessary tags are prepared. The tag here is the above-mentioned job information, and a tag indicating a route to be followed by the endoscope 8, a tag indicating an observation point, a tag indicating a point at which tissue collection should be performed, and the like are prepared. Then, if there is an image area corresponding to the prepared tag (job information) for each teacher's endoscope image, the image area is specified for each tag (for example, by a person), so that a plurality of teacher data can be obtained. Will be generated. The G2-AI model 33 is trained by a predetermined machine learning algorithm using the plurality of teacher data generated in this way. For example, the G2-AI model 33 is trained by CNN's known learning algorithm suitable for image detection.
The method of collecting endoscopic images for teachers may be the same as that of the P-AI model 31 and the G1-AI model 32.

FIG. 7 is a diagram for explaining the second region designation data inferred by the second guide AI model 33.
In the example of FIG. 7, five image regions B1, B2, B3, B4 and B5 are detected, and five second region designation data for designating each image region are represented. The image area B1 is tagged with job information indicating the observation point of the curvature fold, the image area B2 is tagged with the job information indicating the observation point of the angular incisure, and the image area B4 is the pylorus. It is tagged with job information that indicates the observation point of. The image area B5 is tagged with job information indicating the tissue collection of the ulcer, and the image area B3 is tagged with the job information indicating the route to be followed.

When an endoscope image, which is an image captured by the endoscope 8 inserted in the human body model 3, is input, the G2-AI model 33 learned in this way is included in the endoscope image. The detection result for each image area tagged in advance with the job information and the second area designation data for designating the detected image area are output respectively. The detection result for each image area may be the existence probability value for each image area, or may be the presence / absence of detection for each image area. In the former case, an image region having an existence probability value equal to or higher than a predetermined threshold value may be the detected image region.
According to the G2-AI model 33 trained in this way, an image region that is the same as or similar to any of a plurality of specific local images tagged in advance is detected from the input endoscopic image, and the detection is detected. The second area designation data for designating the image area and the corresponding specific local image tag (job information) are acquired.

[Estimation of endoscope position and orientation]
Using the trained P-AI model 31 as described above, the control unit 10 estimates the position and orientation of the endoscope 8. Hereinafter, the process related to the estimation of the position and orientation of the endoscope 8 executed by the control unit 10 will be described in detail.

The control unit 10 receives a video signal captured by an image sensor provided at the tip of the endoscope 8 from an image processing device of the endoscope 8 connected via an input / output I / F unit 13. I'm receiving.
The image processing module 21 acquires an image frame (endoscopic image) of an endoscopic image obtained from the image signal. The image processing module 21 can also sequentially acquire endoscopic images by thinning out the endoscopic images at predetermined intervals.
The image processing module 21 normalizes the acquired endoscopic image for input of the P-AI model 31. For example, the image processing module 21 can perform trimming and size adjustment on the acquired endoscopic image.

By inputting the endoscopic image acquired and normalized by the image processing module 21 into the P-AI model 31, the AI processing module 22 inputs the region position data and the region direction data corresponding to the endoscopic image. get. Specifically, the P-AI model 31 calculates the probability value for each organ region of the luminal organ and the probability value for each direction indicated by the three-dimensional orthogonal axis with respect to the input endoscopic image. To do. For example, the organs of the oral esophagus E0, the gastroesophageal E1, the lesser curvature E2, the body of the stomach E3, the angular incisure E4, the vestibule E5, the duodenal bulb E6, the descending duodenum E7, and the lower duodenal angle E8. The probability values for the region and the probability values for each of the front wall direction D1, the rear wall direction D2, the lesser curvature direction D3, the greater curvature direction D4, the insertion direction D5, and the pull-out direction D6 are calculated.

The AI processing module 22 acquires data capable of identifying the organ region having the maximum probability value as region position data based on the probability value for each organ region, which is the calculation result of the P-AI model 31, and further, P- Based on the probability value for each direction indicated by the three-dimensional orthogonal axis, which is the calculation result of the AI model 31, data capable of identifying the direction having the maximum probability value is acquired as region direction data. Here, the AI processing module 22 acquires data capable of identifying one or more organ regions having a probability value equal to or higher than a predetermined threshold value as region position data, and identifies one or more directions having a probability value equal to or higher than a predetermined threshold value. The data to be obtained can also be acquired as area direction data.
In the present embodiment, the AI processing module 22 also acquires the probability values calculated for the area position data and the area direction data.

The storage processing module 23 stores the area position data and the area direction data acquired by the AI processing module 22 in association with the endoscopic image corresponding to the data in the memory 12. The storage processing module 23 can store the area position data and the area direction data in association with the endoscopic image in the memory 12 so that the data can be reproduced later, or the area can be stored together with the display of the endoscopic image. In order to display the position information and the orientation information of the endoscope 8 indicated by the position data and the area direction data, both may be associated with each other and temporarily stored in the memory 12 and deleted immediately.

The output processing module 24 displays a certain image frame (endoscope image) while displaying the endoscope image on the display device of the input / output panel 5 based on the image signal received from the image processing device of the endoscope 8. ) Is displayed, the position information and orientation information of the endoscope 8 indicated by the area position data and the area direction data acquired for the endoscope image are displayed on the display device of the input / output panel 5.
The position information and orientation information of the endoscope 8 may be displayed in characters, or may be displayed by adding a display capable of grasping the position and orientation of the endoscope 8 to the schematic diagram of the luminal organ. May be good.

[Endoscope guide]
Using the trained G1-AI model 32 and G2-AI model 33 as described above, the control unit 10 guides the endoscope 8. Hereinafter, the processing related to the guide of the endoscope 8 executed by the control unit 10 will be described in detail.
The process related to the guide of the endoscope 8 is executed in parallel with the process related to the estimation of the position and orientation of the endoscope 8 described above. The method of acquiring the endoscopic image by the image processing module 21 is as described above.

In the present embodiment, the AI processing module 22 inputs the endoscopic image acquired and normalized by the image processing module 21 into both the G1-AI model 32 and the G2-AI model 33. As a result, the G1-AI model 32 and the second guide AI model 33 are executed in parallel.
The G1-AI model 32 calculates the probability value for each image area in the input endoscopic image. For example, in the G1-AI model 32, in addition to each unit image area divided by the grid lines in FIG. 6, all combinations of two unit image areas adjacent to the left and right, and two unit image areas adjacent to the top and bottom are used. The probability value is output for each image area of all combinations of the above and all combinations of two diagonally adjacent unit image areas.
The G2-AI model 33 sets the detection result for each image area tagged in advance with the job information in the input endoscopic image and the second area designation data for designating the detected image area, respectively. calculate. For example, the detection result for each image region indicates the presence / absence of detection of each image region based on whether or not it has an existence probability value equal to or higher than a predetermined threshold value.

Here, the inference accuracy of the G1-AI model 32 and the G2-AI model 33 may differ from each other depending on the input endoscopic image. For example, the G2-AI model 33 that detects a specific image region of an endoscopic image has high detection accuracy for an endoscopic image including the entire contour that is an image feature, but includes only a partial contour. Detection accuracy tends to deteriorate in endoscopic images. On the other hand, even in the G1-AI model 32 that classifies the input endoscopic image in the image area, the classification accuracy may deteriorate depending on the input endoscopic image.
Therefore, in the present embodiment, both the G1-AI model 32 and the G2-AI model 33 are executed in parallel, and one of the outputs is used to maintain high inference accuracy. For example, when the maximum value of the probability value for each image region calculated by the G1-AI model 32 is lower than the predetermined threshold value, the output of the G2-AI model 33 is used and the maximum value is equal to or higher than the predetermined threshold value. The output of the G1-AI model 32 is used for this.

In this way, the AI processing module 22 acquires the area designation data from either the G1-AI model 32 or the G2-AI model 33. Specifically, when the output of the G1-AI model 32 is used, the AI processing module 22 has an image region having a maximum probability value based on the probability value for each image region calculated by the G1-AI model 32. Acquires the first area specification data that specifies. Here, the AI processing module 22 can also acquire first region designation data that designates one or more image regions having a probability value equal to or higher than a predetermined threshold value.
In the present embodiment, the AI processing module 22 also acquires the probability value calculated for the first area designation data.
When using the output of the G2-AI model 33, the AI processing module 22 outputs each image region in the endoscopic image output by the G2-AI model 33, which is tagged with job information in advance. The detection result and the second area designation data for designating the detected image area are acquired respectively. Further, the AI processing module 22 also acquires the tag (job information) attached to the second area designation data that designates the detected image area.

By the way, when the first region designation data acquired from the G1-AI model 32 specifies an image region corresponding to the route to be followed by the endoscope 8, each image region is tagged with the G2-AI model 33. In addition to the route, the job information being performed may indicate information other than the route such as a place (point) to be something to be done by the endoscope 8. In this case, the output of either the G1-AI model 32 or the G2-AI model 33 is used for the route that the endoscope 8 should take, and the G2-AI for the other job information. The output of the model 33 may be used fixedly.

Further, in the present embodiment, as described above, both the G1-AI model 32 and the G2-AI model 33 are used, but the G1-AI model 32 depends on the position in the luminal organ to be captured by the endoscopic image. And the G2-AI model 33 may be switched and used. In this case, if the AI processing module 22 is used by switching between the G1-AI model 32 and the G2-AI model 33 according to the position of the endoscope 8 estimated using the inference result of the P-AI model 31. Good. For example, when the position of the endoscope 8 is the stomach, both the G1-AI model 32 and the second guide AI model 33 are used, and when the position of the endoscope 8 is the esophagus or the duodenum, the position is the esophagus or the duodenum. The G2-AI model 33 may be utilized.

The output processing module 24 is acquired by the AI processing module 22 at the timing when the target image frame (endoscopic image) is displayed while displaying the endoscopic image on the display device of the input / output panel 5 as described above. A display in which the guide information of the endoscope 8 is added to the endoscope image based on the area designation data is displayed on the display device.
The guide information displayed in the present embodiment includes information that guides the route or direction in which the endoscope 8 should go, a place where something should be done by the endoscope 8 such as observation, tissue collection, treatment, etc. ( There is information to guide the point).

Specifically, the output processing module 24 displays a direction display toward the image area indicated by the area designation data acquired by the AI processing module 22 in the endoscope image acquired from the image processing module 21. Overlay on the image.
This direction display may be displayed so as to clearly point to a specific image area, or may be a display pointing to a direction such as upward, downward, left, or right.

Further, when the output processing module 24 acquires the second area designation data from the G2-AI model 33 and further acquires the job information corresponding to the second area designation data, the display corresponding to the job information is obtained. In the form, a display indicating an image region designated by the second region designation data can be added to the endoscope image as guide information of the endoscope 8.
For example, when the job information indicates the route to be taken by the endoscope 8, a direction display (for example, an arrow display) toward the image area indicated by the second area designation data is added to the endoscope image. .. Further, when the job information indicates an observation point, a marker display may be added on the image area indicated by the second area designation data.

FIG. 8 is a diagram showing a display example of guide information of the endoscope.
In the example of FIG. 8, the arrow display G1 toward the image region corresponding to the route to be taken by the endoscope 8 is superimposed on the endoscope image. In addition, a marker G2 is attached to the image area corresponding to the observation point, and an annular broken line G3 is displayed around the marker G2 for easier understanding.
However, the display of the guide information in this embodiment is not limited to the example of FIG.

Here, in the endoscopy manual, many observation points are determined in order to prevent the lesion from being overlooked. The observation point requires not only observation from a slightly distant position but also close observation of details. Therefore, in the present embodiment, when an image area such as an observation point is designated by the first area designation data or the second area designation data, the image area is of a certain size in the endoscopic image. Guide it so that it is placed in the center position.
For example, the output processing module 24 notifies that the image area indicated by the acquired area designation data in the acquired endoscopic image has reached a predetermined position or a predetermined size in the endoscopic image. Output the notification display. This is a notification display when the area designation data for designating the image area corresponding to the observation point is acquired for the acquired endoscopic image, that is, when the observation point appears in the endoscopic image. Including output.
This notification display will be displayed if the viewer can grasp that the image area corresponding to the observation point has appeared in the endoscopic image or has reached a predetermined position or a predetermined size in the endoscopic image. Display contents and display forms may be used.

FIG. 9 is a diagram showing an example of guide display of observation points.
In the example of FIG. 9, when the image area B10 designated by the area designation data acquired by the AI processing module 22 corresponds to the observation point, the output processing module 24 performs the guide display as follows. That is, the output processing module 24 superimposes and displays the aiming displays F1, F2, and F3 on the endoscopic image, and the timing at which the image area B10 reaches a predetermined size within the frame of the aiming display F2 shown at a predetermined position. The endoscopic image is set as a still image G4, and the still image G4 is rotated and reduced to frame out.
The frame-out effect in this display can be regarded as corresponding to a notification display for notifying that the image area indicated by the area designation data has reached a predetermined position or a predetermined size in the endoscopic image, and can be regarded as an aim in the display. It can also be considered that the displays F1, F2 and F3 correspond to the notification display. In the latter case, the output processing module 24 may display the aiming points F1, F2 and F3 when the area designation data corresponding to the observation point is acquired by the AI processing module 22, or may be designated by the area designation data. Aims F1, F2 and F3 may be displayed when the image area B10 to be formed has a predetermined position or a predetermined size.
However, the content and display form of the notification display are not limited to such an example.

The guide information as described above may be always displayed when the area designation data is acquired by the image processing module 21. However, when the guide information is displayed frequently, if the trainee's skill level increases to some extent, the guide information may become an obstacle and cause discomfort to the trainee.
Therefore, the output processing module 24 takes the opportunity of detecting that the endoscope 8 is stagnant in the luminal organ for a predetermined time based on the endoscopic images sequentially acquired by the image processing module 21. A display to which the guide information is added may be output.
In this case, for example, by detecting that the contents of the endoscopic image are almost unchanged for a predetermined time, it is possible to detect that the endoscope 8 is stagnant in the luminal organ for a predetermined time. The fact that the endoscope 8 is stagnant for some time may mean that the trainee is confused by the endoscopic procedure.
Only when the trainee is confused by outputting a display with the guide information added when the endoscope 8 detects that it is stagnant in the luminal organ for a predetermined time as described above. Since the guide information can be displayed, it is possible to prevent the guide information from becoming an obstacle and causing discomfort to the trainee.

In the manual of endoscopic procedure, a large number of observation points are determined as described above, and it is recommended to capture and record each observation point to leave a still image.
Therefore, the control unit 10 (storage processing module 23) may hold the image pickup record information which is the history information of the organ part estimated to have been imaged and recorded by the endoscope 8. In this case, the output processing module 24 uses the information of the organ part group to be imaged and recorded, and based on the history information of the organ part indicated by the image recording information, the image recording is performed from the organ part group to be imaged and recorded. It is possible to identify the part of the organ that has not been made. Further, when the output processing module 24 identifies an organ part for which imaging recording has not been made, the output processing module 24 can also display information indicating that fact or the specified organ part on the display device of the input / output panel 5.
In this way, it is possible to point out the omission of imaging recording to the trainee.

The organ site presumed to have been imaged and recorded by the endoscope 8 can be specified as follows.
When the input / output I / F unit 13 can receive a signal indicating the operation of the switch instructing the recording of the still image provided in the operation unit of the endoscope 8, it corresponds to the timing at which the signal is received. The organ site can be specified by the area position data acquired for the endoscopic image, the first area designation data, the second area designation data, and the job information.
In addition, an image region tagged with job information indicating an observation point (organ site to be imaged and recorded) was detected by the output of the G2-AI model 33, and second region designation data for designating the image region was acquired. In the case, it is presumed that the organ part corresponding to the observation point was imaged and recorded by detecting that the image area indicated by the second area designation data has reached a predetermined position or a predetermined size in the endoscopic image. can do.
Furthermore, an endoscopic image in which a specific organ part corresponding to each observation point is captured at a predetermined position and a predetermined size is used as a teacher's endoscope image, and each specific organ part is compared with respect to the teacher's endoscope image. By using an AI model that is machine-learned with teacher data tagged with imaging record data indicating imaging records, it is also possible to identify an organ site that is presumed to have been imaged and recorded by the endoscope 8. In this case, when the imaging record data is acquired by giving the endoscope image to the AI model, the history information of the organ part estimated to have been captured and recorded by the endoscope 8 based on the imaging recording data. It is possible to retain the imaging record information.

[Operation example of control unit]
FIG. 10 is a flowchart showing an operation example of the control unit 10. Since the detailed operation contents of each process shown in FIG. 10 are as described above, the operation flow of the control unit 10 will be mainly described here.
The control unit 10 receives the image signal of the endoscope from the image processing device of the endoscope 8 connected via the input / output I / F unit 13, and the endoscope obtained from this image signal. The image is displayed on the display device of the input / output panel 5.
The control unit 10 thins out the image frames (endoscopic images) of the endoscopic image at a predetermined cycle and sequentially acquires them (S101), and each time the endoscopic image is acquired, the operation flow shown in FIG. 10 is performed. Run. In step (S101), the control unit 10 can also normalize the endoscopic image for input of various AI models.

The control unit 10 inputs the endoscopic images acquired in (S101) to the P-AI model 31, G1-AI model 32, and G2-AI model 33, respectively (S110), (S121), and (S122). Here, an example in which the same endoscopic image is input to the P-AI model 31, G1-AI model 32, and G2-AI model 33 is shown, but each AI model is input from the image frame of the endoscopic image. Different endoscopic images obtained by thinning out at different cycles may be input.
As a result, the P-AI model 31, the G1-AI model 32, and the G2-AI model 33 are executed in substantially parallel.

The control unit 10 acquires the area position data and the area direction data based on the calculation result of the P-AI model 31 (S111). At this time, if the maximum value of the probability value for each organ region or the maximum value of the probability value for each direction output by the P-AI model 31 is lower than the predetermined threshold value, the control unit 10 acquires the data in (S101). With respect to the endoscopic image obtained, the region position data or the region direction data may not be acquired.
In addition to the endoscope image, the control unit 10 displays the position information and orientation information of the endoscope 8 indicated by the area position data and the area direction data acquired in (S111) on the display device of the input / output panel 5. (S112).

In parallel with the above-mentioned operation, the control unit 10 operates as follows with reference to the outputs of the G1-AI model 32 and the G2-AI model 33.
The control unit 10 determines the detection result for each image area tagged in advance with the job information based on the output of the G2-AI model 33 (S123), and further, based on the output of the G1-AI model 32. , (S124) and (S131) determine whether or not the first area designation data can be acquired. Whether or not the first region designation data can be acquired can be determined by whether or not the maximum value of the probability value for each image region is equal to or greater than a predetermined threshold value. If the maximum value is equal to or greater than a predetermined threshold value, it can be determined that the first area designated data can be acquired, and if not, it can be determined that the first area designated data cannot be acquired.

When the control unit 10 has the detected image area and the first area designation data can be acquired (S123; YES) (S124; YES), the control unit 10 of the G1-AI model 32 or the G2-AI model 33. It is determined whether or not the output needs to be selected (S125). For example, the job information (tag) corresponding to the image area detected by the G2-AI model 33 is the semantic information of the image area classified by the G1-AI model 32 (for example, the route or observation point where the endoscope 8 should be advanced). ), It may be determined that selection is necessary (S125; YES), and if it is different, it may be determined that selection is not necessary (S125; NO).
When it is determined that it is necessary to select (S125; YES), the control unit 10 selects either the output of the G1-AI model 32 or the output of the G2-AI model 33 (S126). As a result, either the first area designated data or the second area designated data is selected. For example, if the probability value of the image region specified by the first region designation data is equal to or greater than a predetermined threshold value, the first region designation data that is the output of the G1-AI model 32 is acquired, and if not, the G2-AI The second area designation data which is the output of the model 33 and the job information corresponding thereto may be acquired.

When it is determined that there is no need to select (S125; NO), the control unit 10 specifies the first area designation data which is the output of the G1-AI model 32 and the second area designation which is the output of the G2-AI model 33. Acquire the data and the corresponding job information (S128).
If there is a detected image area and the first area designation data cannot be acquired (S123; YES) (S124; NO), the control unit 10 is the output of the G2-AI model 33. (2) Acquire the area designation data and the job information corresponding thereto (S127).
Further, when there is no detected image area and one area designation data can be acquired (S123; NO) (S131; YES), the control unit 10 is the output of the G1-AI model 32. Acquire area designation data (S132). If there is no detected image area and the first area designation data cannot be acquired (S123; NO) (S124; NO), the control unit 10 ends the process without displaying the guide information. ..

When the control unit 10 acquires either one or both of the first area designation data and the second area designation data, the input / output panel 5 inputs guide information based on the acquired area designation data in addition to the endoscopic image. Is displayed on the display device of (S129). The display of the guide information is as described above.

However, the operation flow of the control unit 10 is not limited to the example shown in FIG. In the flowchart shown in FIG. 10, a plurality of processes (processes) are described in order, but the execution order of each process is not limited to the order of description. Further, the method of using the output results of the G1-AI model 32 and the G2-AI model 33 is not limited to the example shown in FIG. Further, any one or more of the P-AI model 31, the G1-AI model 32 and the G2-AI model 33 may not be executed in parallel with the other AI models, but may be executed before and after. The order of the illustrated steps can be changed within a range that does not hinder the contents.

[Modification example]
The contents of the system 1 described above are merely examples, and can be partially changed as appropriate.
For example, in the above-mentioned system 1, the endoscope 8 is connected to the input / output I / F unit 13, but the endoscope 8 may not be connected. For example, after the moving image data of the endoscope image captured by the endoscope 8 is stored in a portable storage medium or an external device (PC or the like), the control unit 10 of the control unit 10 via the portable recording medium or communication. The endoscopic image stored in the memory 12 and obtained from the moving image data may be processed as described above.

Further, in the above-mentioned system 1, the position information, the orientation information, and the guide information of the endoscope 8 are acquired by using only the endoscope image, but other information is further added in addition to the endoscope image. You may use it.
For example, the control unit 10 further uses the detection information of the object detection sensor provided at a predetermined portion (for example, the esophageal entrance portion, the gastroesophageal junction, and the duodenal descending leg portion) of the internal modeling portion of the human model 3. May be good. According to this, the fact that the tip portion of the endoscope 8 has passed through the predetermined portion can be grasped as accurate information. That is, the control unit 10 may include as a software element detection information acquisition means for acquiring the presence detection information of the endoscope from each sensor provided at a plurality of predetermined parts of the luminal organ model.
In this case, by using the detection information of the endoscope 8 by the object detection sensor of the human body model 3, it is possible to confirm whether or not the region position data obtained as the inference result of the P-AI model 31 is correct. Therefore, the AI processing module 22 can also determine and correct the correctness of the region position data obtained from the output of the P-AI model 31 based on the detection information. For example, the AI processing module 22 may acquire region position data that indicates a position that matches the detection information and has the maximum probability value.
Further, an AI model may be provided for each organ region between the sites where the object detection sensor is provided, and the control unit 10 can switch the AI model to be used by using the detection information of the object detection sensor. For example, an AI model from the esophageal entrance to the gastroesophageal junction and an AI model from the gastroesophageal junction to the duodenal descending leg are provided. This may be divided into AI models for each organ region not only for the P-AI model 31 but also for the G1-AI model 32 and the G2-AI model 33.
Further, the P-AI model 31 can be formed so as to input the endoscopic image and the detection information of the object detection sensor and output the probability value for each organ region and the probability value for each direction.
By further using the more accurate detection information of the object detection sensor in this way, the inference accuracy of various AI models can be improved.

Further, although not particularly mentioned in the above-described embodiment, the control unit 10 can automatically determine an error in the inference result of the P-AI model 31, the G1-AI model 32, and the G2-AI model 33.
For example, comparing each output of the AI model for three endoscopic images adjacent to the time series to be processed sequentially, the output for the intermediate endoscopic image is significantly different from the output for the endoscopic images before and after it. If they are different, the output for the intermediate endoscopic image can be determined to be erroneous. Since the time interval between adjacent endoscopic images in chronological order is less than 1 second, the output is that the AI model outputs with a certain degree of reliability for the intermediate endoscopic image. Is significantly different from the output related to the front and rear endoscopic images, there is a high possibility of erroneous judgment. If the endoscope 8 is moved at high speed, the endoscope image will not be a clear image due to blurring, and if such an endoscope image is input to the AI model, it should be impossible to infer. Is.

Therefore, specifically, the control unit 10 obtains the region position data or the region direction data obtained from the output of the P-AI model 31 for the intermediate endoscopic image with respect to the front and rear endoscopic images. If it is significantly different from the region position data or region direction data to be obtained, the output related to the intermediate endoscopic image is determined to be incorrect. Similarly, the control unit 10 specifies the area in which the area designation data obtained from the output of the G1-AI model 32 or the G2-AI model 33 for the intermediate endoscope image is obtained for the front and rear endoscope images. If an image area significantly different from the data is specified, the output related to the intermediate endoscopic image is determined to be incorrect.
In this way, even though each AI model outputs with a certain degree of reliability, if it is determined to be an error, the control unit 10 may not use the output.

Furthermore, the control unit 10 can automatically generate the output of the AI model determined to be an error from the output of the AI model related to the front and rear endoscopic images. For example, from the region position data or region direction data acquired based on the output of the P-AI model 31 for the front and rear endoscope images, it is assumed that the endoscope does not move at high speed, and an endoscope in between. The area position data and the area direction data related to the image can be predicted. The same applies to the area designation data acquired based on the output of the G1-AI model 32 or the G2-AI model 33. For example, the area designation data that specifies the image area at the intermediate position between the image areas specified by the area designation data acquired for the front and rear endoscopic images is generated as the data corresponding to the intermediate endoscopic image. May be good.

Further, the control unit 10 automatically generates the intermediate endoscope that is automatically generated from the output of the AI model regarding the intermediate endoscope image when it is determined to be an error and the front and rear endoscope images as described above. The output of the AI model related to the image is associated and retained. As a result, the endoscopic image held in this way and the output of the automatically generated AI model can be used as teacher data at the time of re-learning of the AI model.
Such re-learning of the AI model may be automatically executed at a time when the system 1 is not used by the trainee.

The endoscopic procedure trainer system (system 1) described above is an example of an embodiment of the present invention.
The present system 1 includes an organ model of the human body model 3, and enables self-learning and self-training of endoscopic techniques using the organ model. However, as an embodiment of the present invention, the endoscopic technique is performed. The endoscope 8 itself may be used, not limited to those for the purpose of learning or training.
FIG. 11 is a diagram conceptually showing a control configuration of the endoscope system 80 (endoscope 8) according to another embodiment.
The endoscope system 80 includes an insertion portion including a tip portion and a curved portion, an operation unit for performing various operations on the tip portion and the curved portion, an image processing device 82, a display device 83, and the like. An endoscope imaging unit 81 is provided at the tip portion, and the endoscope imaging unit 81 and the display device 83 are connected to the image processing device 82 by a cable or the like.
The image processing device 82 has a processor 85, a memory 86, an input / output interface (I / F) unit 87, and the like, and the control program and the AI model stored in the memory 86 are executed by the processor 85. , The same processing as that of the control unit 10 described above may be realized.
The software configuration of the image processing device 82 may be the same as the software configuration of the control unit 10 shown in FIG. The image processing device 82 may also be referred to as an endoscopic image processing device or an endoscopic image processing system.

In the endoscope system 80, the endoscope image acquired by the image processing device 82 may be an image obtained by capturing the lumen organ model of the human body model 3, or an image obtained by capturing the lumen organ of a living body. There may be, or both. However, when the endoscopic image obtained by the endoscopic imaging unit 81 of the luminal organ of the living body is to be processed, the P-AI model 31, G1-AI model 32 and G2-AI model 33 are living bodies. It is desirable that machine learning is performed using a teacher's endoscopic image that images the luminal organs of. The correct answer of the area position data and the area direction data corresponding to the teacher's endoscope image, the correct answer of the area designation data corresponding to the teacher's endoscope image, and the job information of the endoscope for it are the teachers. It can be generated by confirming the endoscopic image for use with a plurality of skilled doctors.
Further, the P-AI model 31, the G1-AI model 32, and the G2-AI model 33 use the endoscopic images for teachers that image the lumen organ model of the human body model 3 in the same manner as the above-mentioned system 1. After machine learning, further machine learning may be performed using an endoscopic image for a teacher that images a luminal organ of a living body. Furthermore, we prepared multiple types of tract organ models that imitated tract organs of various shapes, sizes, and modes, and captured endoscopic images for teachers while switching between these multiple types of tract organ models. It is also possible to use it for machine learning.
By doing so, it is possible to improve the estimation accuracy of various AI models such as the P-AI model 31.

In addition, a teacher's endoscope image is collected using an endoscope system that can acquire the position and orientation of the tip of the endoscope in cooperation with a magnetic sensor, and the endoscope acquired by the endoscope system is used. It is also possible to machine-learn the P-AI model 31 by collecting the position information and the orientation information of the tip of the mirror and using a plurality of teacher data associated with them. In this case, the P-AI model 31 can be trained with the position information and the orientation information that are more subdivided than the above-mentioned "region position data" and "region direction data", and the position information is subdivided as such. And direction information can be acquired.

Further, the endoscope system 80 may be a capsule endoscopy system. In this case, the endoscope imaging unit 81 may not be provided at the tip of the endoscope, but may be provided at the capsule endoscope itself and connected to the image processing device 82 by wireless communication.
Further, the endoscope system 80 does not need to include a display device 83. The area position data and the area direction data acquired from the P-AI model 31 may be associated with the target endoscopic image and stored in the memory 86. Similarly, the area designation data acquired from the G1-AI model 32 and the G2-AI model 33 may also be associated with the target endoscopic image and stored in the memory 86.
Further, the generated guide information may be used as information for automatically operating the insertion portion including the tip portion and the curved portion of the endoscope, or the capsule endoscope itself.

The contents of each of the above-described embodiments can also be specified as follows.
(Appendix 1) An image acquisition means for acquiring an endoscopic image taken by an endoscope in a luminal organ, and
By giving the acquired endoscope image to the first trained model, the first model that acquires the position information and the orientation information indicating the position and direction of the endoscope that captured the endoscope image. Processing means and
A storage means for associating and storing the acquired position information and orientation information with the acquired endoscopic image, and
With
The first trained model is machine-learned based on a plurality of teacher data in which the correct answer of the position and orientation of the endoscope that captured the teacher's endoscope image is associated with the teacher's endoscope image. Has been
Endoscopic image processing system.
(Appendix 2) The correct answer of the position and orientation of the endoscope in the plurality of teacher data is the correct answer of the area position data in which each of the plurality of regions in which the luminal organ is virtually divided in the long axis direction can be identified as position information. , And the correct answer of the area direction data that can identify each direction indicated by the three-dimensional orthogonal axis virtually set in each area.
The first model processing means acquires region position data and region direction data corresponding to the acquired endoscopic image as the position information and the orientation information.
The endoscopic image processing system according to (Appendix 1).
(Appendix 3) The endoscopic image acquired by the image acquisition means is an image taken by an endoscope in a tract organ of a living body or in a tract organ model imitating a tract organ of a living body.
The plurality of teacher data includes a plurality of teacher endoscopic images captured by an endoscope in the luminal organ model.
The endoscopic image processing system according to Appendix 1 or 2.
(Appendix 4) Detection information acquisition means for acquiring the presence detection information of an endoscope from each sensor provided at a plurality of predetermined parts of a tract organ model imitating a tract organ of a living body.
Further prepare
The endoscopic image acquired by the image acquisition means is an image captured by the endoscope in the luminal organ model.
The teacher endoscopic image of the plurality of teacher data is an image taken by the endoscope in the tract organ model.
The first model processing means further uses the acquired existence detection information to acquire the position information and the orientation information.
The endoscopic image processing system according to (Appendix 1) or (Appendix 2).
(Appendix 5) A second model processing means for acquiring region designation data corresponding to the endoscopic image by giving the acquired endoscopic image to the second trained model.
An output processing means for generating endoscope guide information regarding an image region designated by the region designation data in the acquired endoscope image based on the acquired region designation data.
Further prepare
The second trained model is machine-learned using a plurality of teacher data in which the correct answers of the region designation data are associated with each teacher's endoscopic image.
The endoscopic image processing system according to any one of (Appendix 1) to (Appendix 4).
(Appendix 6) The second trained model associates the correct answer of the second area designation data that specifies the specific image area tagged with the job information of the endoscope with each teacher's endoscope image. It is machine-learned using multiple teacher data.
The second model processing means further acquires the job information corresponding to the area designation data together with the area designation data, and further acquires the job information.
The output processing means further uses the job information, and in a display form corresponding to the job information, the display indicating the image area designated by the acquired area designation data is used as the guide information. Add to the endoscopic image,
The endoscopic image processing system according to (Appendix 5).
(Appendix 7) The output processing means superimposes and displays a direction display toward an image area indicated by the acquired area designation data in the acquired endoscopic image on the endoscopic image.
The endoscopic image processing system according to (Appendix 5) or (Appendix 6).
(Appendix 8) The output processing means indicates that the image area indicated by the acquired area designation data in the acquired endoscopic image has become a predetermined position or a predetermined size in the endoscopic image. Output the notification display to notify,
The endoscopic image processing system according to any one of (Appendix 5) to (Appendix 7).
(Appendix 9) The storage means holds image recording information which is historical information of an organ part estimated to have been imaged and recorded by an endoscope based on the acquired area designation data.
The output processing means identifies an organ part for which imaging recording has not been made from the organ part group to be imaged and recorded based on the history information of the organ part indicated by the imaging recording information.
The endoscopic image processing system according to any one of (Appendix 5) to (Appendix 8).
(Appendix 10) The output processing means triggers the detection that the endoscope is stagnant in the luminal organ for a predetermined time based on the endoscopic images sequentially acquired by the image acquisition means. , Output the display with the guide information added,
The endoscopic image processing system according to any one of (Appendix 5) to (Appendix 9).
(Appendix 11) A second model processing means for acquiring the first region designation data corresponding to the endoscopic image by giving the acquired endoscopic image to the second trained model.
By giving the acquired endoscopic image to the third trained model, the second region designation data corresponding to the endoscopic image and the job information corresponding to the second region designation data are acquired. Third model processing means and
Either the acquired first region designation data or the acquired second region designation data and job information based on the outputs of the second trained model and the third trained model. Is selected, and output processing means for generating the guide information of the endoscope regarding the selected first region designation data or the image region designated by the second region designation data in the acquired endoscope image. When,
Further prepare
The second trained model is machine-learned using a plurality of teacher data in which the correct answers of the first region designation data are associated with each teacher's endoscopic image.
In the third trained model, a plurality of teachers are associated with the correct answers of the second area designation data that specifies the specific image area tagged with the job information of the endoscope for each teacher's endoscope image. Machine learning using data,
The endoscopic image processing system according to any one of (Appendix 1) to (Appendix 4).

1 Endoscope procedure trainer system (this system), 3 human body model, 5 input / output panel, 6 speakers, 7 sensor group, 8 endoscope, 10 control unit, 11 processor, 12 memory, 13 input / output I / F Unit, 21 Image processing module, 22 AI processing module, 23 Storage processing module, 24 Output processing module, 31 Positioning AI model (P-AI model), 32 First guide AI model (G1-AI model), 33 Second guide AI model (G2-AI model), 80 endoscopic system, 81 endoscopic imaging unit, 82 image processing device, 83 display device, 85 processor, 86 memory, 87 input / output I / F unit

Claims (1)

An image acquisition means for acquiring an endoscopic image taken by an endoscope in a luminal organ,
By giving the acquired endoscopic image to the trained model, a model processing means for acquiring the area designation data corresponding to the endoscopic image, and
Based on the acquired region designation data, a display in which the guide information of the endoscope regarding the image region designated by the region designation data in the acquired endoscope image is added to the endoscope image is output. Output processing means to be
With
The trained model is machine-learned using a plurality of teacher data in which the correct answers of the region designation data are associated with each teacher's endoscopic image.
Endoscopic image processing system.

Images