JP7264407B2 - Colonoscopy observation support device for training, operation method, and program - Google Patents

Description

The present invention relates to a colonoscopy support device, a colonoscopy support method, and a program.

Conventionally, image recognition processing is performed on images of hollow organs such as the digestive tract taken with an endoscope, and learning functions such as artificial intelligence (AI) automatically detect tumors, polyps, and other lesions. An image processing apparatus has been proposed (see, for example, Patent Document 1).

In addition, in order to prevent doctors from overlooking lesions, when lesions are automatically detected based on captured images, real-time alerts to operators such as doctors (e.g., presentation of images of lesion positions, scattered There is also disclosed a technique for assisting pathological diagnosis by displaying probability, calling attention by voice, etc. (see, for example, Patent Document 2).

Advances in artificial intelligence (AI) have improved the accuracy and speed of automatic detection of lesions by computers, and have contributed to the observation and diagnosis of lesions by doctors. Non-Patent Literature 1 describes that analyzing a plurality of frames with AI enables more accurate image analysis. It is highly desired to develop a technology that supports physicians in observing and diagnosing lesions with higher accuracy while referring to the detection results of artificial intelligence and without being unnecessarily affected by the detection results. It is

JP 2006-153742 A JP 2011-104016 A

Tran, Du, et al. "Learning spatiotemporal features with 3d convolutional networks." Computer Vision (ICCV), 2015 IEEE International Conference on. IEEE, 2015.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and aims to improve the performance of endoscope observation support, and to enable doctors to observe lesions more easily and accurately. is.

In order to achieve the above object, a colonoscopy observation support device according to the present application is a colonoscopy observation support device that supports observation of a large intestine with a colonoscope, and includes: an image information acquisition unit that acquires a captured image of the large intestine and displays it on a display unit; a lesion information acquisition unit that detects a predetermined lesion based on the captured image and acquires lesion information about the lesion; and a notification unit that notifies that the lesion has been detected after a short time has passed since the lesion was detected.

ADVANTAGE OF THE INVENTION According to this invention, the performance of the observation support by an endoscope can be improved, and a doctor's observation of a lesion etc. can be performed more easily and accurately.

1 is a schematic diagram showing a hardware configuration of an endoscopic observation support device according to an embodiment of the present application; FIG. 2 is a functional block diagram of the endoscopic observation support device of FIG. 1; FIG. FIG. 2 is a flowchart for explaining an example of the operation of endoscope observation support processing executed by the endoscope observation support device of FIG. 1; FIG. FIG. 4A is a diagram showing an example of an endoscopic image displayed on the display unit and the state of the sound output unit and the vibrating unit at that time, in which (a) is the state when a lesion appears on the endoscopic image; , and (b) shows the state when the lesion disappears from the endoscopic image. FIG. 10 is a diagram showing a display example of the display unit when displaying an alarm, (a) showing a display example before detection of a lesion (no lesion), and (b) showing a lesion disappearing from the display unit after detection of a lesion. (c) shows a modified example of the display when the lesion disappears from the display section after detection of the lesion. (d) shows another display example before lesion detection (no lesion), (e) shows another display example when the lesion disappears from the display unit after lesion detection, and (f) shows the lesion. FIG. 10 shows a modified example of another display example when the lesion disappears from the display section after detection.

(First embodiment)
An endoscope observation support device according to a first embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic diagram showing the hardware configuration of an endoscope observation support device 100 according to this embodiment. As shown in FIG. 1, the endoscope observation support device 100 mainly includes a support device main body 10 and an endoscope device 40 .

As shown in FIG. 1, the support device main body 10 includes a CPU 11, a ROM 12, a RAM 13, an input I/F (interface) 14, an output I/F 15, an external device I/F 16, a storage unit 17, and an information processing unit 20. , an input unit 21, a display unit 22, a sound output unit 23, a vibration unit 24, and the like. The support device body 10 may also include a printer as an output destination for printing endoscopic images and support results, a communication device for communicating with the outside, and the like.

The information processing section 20 can be configured from, for example, a personal computer (PC), a microcomputer, or the like. A CPU 11 is a central processing unit and controls the operation of the endoscope observation support apparatus 100 as a whole. The ROM 12 is a read-only memory, and stores a control program executed by the CPU 11, an endoscope observation support program, and the like. A RAM 13 is a read/write memory as needed, and is used as a work area for the CPU 11 . That is, the CPU 11 uses the RAM 13 as a work area and executes the endoscope observation support program stored in the ROM 12 to operate the endoscope observation support apparatus 100 .

The storage unit 17 is composed of a recording medium such as a hard disk and flash memory, and stores various information necessary for endoscope observation support processing, calculation results, and the like. The input I/F 14 is an interface for connecting the input unit 21 and the CPU 11 . The output I/F 15 is an interface for connecting the display unit 22 and the CPU 11 . The external device I/F 16 is an interface for connecting the external device and the CPU 11. In this embodiment, the endoscope device 40, the sound output unit 23, and the vibration unit 24 are connected to the corresponding external device I/F 16. It is connected to the CPU 11 via F16.

The input unit 21 is a device for inputting characters, numbers, various instructions, etc., and is operated by a doctor or the like as an operator who operates the endoscope apparatus 40 . Examples of the input unit 21 include a keyboard, mouse, numeric keypad, touch panel, and the like. Further, the input unit 21 inputs an instruction for the doctor to take or record a still image of the lesion, or an instruction for confirming the automatically detected lesion.

The display unit 22 is a device that displays a menu screen, an operation screen, and endoscope images (captured images) such as moving images and still images captured by the endoscope device 40 . Examples of the display unit 22 include a liquid crystal display (monitor) and the like.

The sound output unit 23 and the vibrating unit 24 are used to notify the doctor (alarm notification). The sound output unit 23 is a device that outputs a notification sound (alarm sound), a notification message (alarm message), or the like for drawing the doctor's attention. . The vibrating section 24 is a vibrating body that generates vibration to be applied to the doctor's body, and includes, for example, a vibrating device that generates vibration using a motor or the like. The vibrating section 24 is desirably configured to be attached to the doctor's body so that it can directly apply vibration to the doctor's body. Alternatively, if there is no problem in operating the endoscope device 40 , the vibrating section 24 may be arranged in the endoscope device 40 and attached to the hand of the doctor holding the endoscope device 40 .

In this embodiment, both the sound output unit 23 and the vibration unit 24 are provided. For example, if the alarm is notified only by vibration, it is possible to prevent the patient from noticing that a lesion has been detected. In addition, the display unit 22 may display an alarm to attract the doctor's attention visually as well as sound and vibration.

The endoscope device 40 is a device used for observing and treating hollow organs, and is configured by an electronic scope, a fiber scope, or the like. An endoscope apparatus 40 of the present embodiment includes an imaging section 41, an insertion section 42, an operation section 43, and the like. The endoscope device 40 also includes components that can be included in known endoscope devices, such as means for supplying water and air, suction means, and treatment tools such as forceps for collecting tissue and performing other treatments. there is

The imaging unit 41 is a device for capturing endoscopic images such as moving images and still images of the living body, and includes a CCD camera, a condenser lens, an eyepiece lens, a light source, a light guiding optical system, an imaging device, and the like. be done. The insertion section 42 is composed of a bendable and flexible tubular member, and is provided with a CCD camera and a condenser lens of the imaging section 41 at its tip. A hole or the like is provided. The operation portion 43 includes various operation buttons and operation switches for bending the insertion portion 42, supplying air, supplying water, sucking, and the like.

The endoscope device 40 is electrically connected to the support device main body 10 by a connection cable or the like. An image signal of an endoscopic image captured by the imaging unit 41 is output to the support device main body 10 in real time.

The endoscope device 40 is not particularly limited, and known devices can be used. For example, a laryngoscope to observe the nasal cavity, pharynx, larynx, and esophagus; a bronchoscope to observe the trachea and bronchi; an upper gastrointestinal endoscope to observe the esophagus, stomach, and duodenum; a small intestine endoscope to observe the small intestine; colonoscopy to observe the bile duct, cholangioscopy to observe the bile duct, pancreatic duct endoscope to observe the pancreatic duct, thoracoscope to observe the inside of the thoracic cavity, laparoscope to observe the inside of the abdominal cavity, urethra, bladder to observe the inside of the bladder Examples include, but are not limited to, mirrors, arthroscopes for observing joints, and angioscopes for observing coronary arteries.

Also, the endoscope device 40 is preferably a high-definition endoscope, for example, but is not limited to this.

Next, the functions of the information processing section 20 that controls the operation of the support device main body 10 will be described using the functional block diagram of FIG. As shown in FIG. 2, the information processing unit 20 includes an image information acquisition unit 31, a lesion information acquisition unit 32, a confirmation information acquisition unit 33, a lesion disappearance determination unit 34, a notification unit 35, and a lesion information output unit. It functions as part 36 .

The image information acquisition unit 31 converts an image signal input from the imaging unit 41 of the endoscope device 40 into a digital signal, performs appropriate image processing, and displays the signal on the display unit 22 in real time (Fig. 4(a)). )reference). In this embodiment, a moving image or a still image captured by the imaging unit 41 is displayed on the display unit 22. However, the present invention is not limited to this, and a 3D image or the like generated based on the moving image may be displayed. good too. In addition, the acquired endoscopic image is passed to the lesion information acquiring section 32 .

Based on the endoscopic image acquired by the image information acquisition unit 31, the lesion information acquisition unit 32 executes image recognition processing such as feature amount extraction and pattern matching to automatically detect lesions and obtain various information related to lesions. Acquire information (lesion information).

More specifically, the lesion information acquiring unit 32 acquires, for example, laryngoscopic video images of the nasal cavity, pharynx, larynx, and esophagus, bronchoscopic video images of the trachea and bronchi, and video images of the esophagus, stomach, and duodenum. upper gastrointestinal endoscopy video image, small bowel endoscopy video image of the small intestine, colonoscopy video image of the large intestine, cholangioscopy video image of the bile duct, pancreatic duct endoscopy image of the pancreatic duct Video images, thoracoscopic video images of the inside of the small thoracic cavity, laparoscopic video images of the inside of the abdominal cavity, cystoscopic video images of the urethra and bladder, arthroscopic video images of the inside of joints, blood vessels such as coronary arteries Tumors, polyps, hemorrhages, diverticula, stools, angiodysplasia, peritoneal dissemination, pleural dissemination, and other lesions are automatically detected by image recognition processing based on angioscopy images taken from

Lesion detection using image recognition processing performed by the lesion information acquisition unit 32 may be performed using any known technique. For lesion detection, for example, a method of analyzing a plurality of frames by artificial intelligence as described in the paper “Learning Spatiotemporal Features with 3D Convolutional Networks” cited as Non-Patent Document 1 can be preferably used. By using this technique, lesion detection can be performed with higher accuracy. Here, the detection of a lesion performed by the lesion information acquisition unit 32 is the result of mechanically estimating a lesion by artificial intelligence (computer) using image recognition technology, learning function, or the like. With the development of artificial intelligence, it has become possible to automatically detect lesions with higher speed and accuracy, and to acquire more detailed information such as the types and conditions of lesions. The endoscopic observation support apparatus 100 of the present embodiment effectively utilizes the detection result by the artificial intelligence to provide support that can contribute to pathological diagnosis by a doctor.

The lesion information acquired by the lesion information acquisition unit 32 is output to the lesion disappearance determination unit 34 in association with the endoscopic image. Lesion information includes, for example, flag information indicating that a lesion has been detected, and the position, size, and state of the detected lesion on the endoscopic image. Furthermore, it is more desirable to include the classification results of lesions by image recognition, more specifically, the classification results of lesions such as tumor, polyp, hemorrhage, diverticulum, stool, angiodysplasia, peritoneal dissemination, and pleural dissemination. .

The confirmation information acquisition unit 33 acquires confirmation information input from the input unit 21 . More specifically, for example, a photographing button for still image photographing and a lesion confirmation button are displayed on the display screen of the display unit 22 . Alternatively, these buttons may be set to keyboard keys. Then, when the doctor observes the endoscopic image displayed on the display unit 22 or directly observes the inside of the body with a microscope to confirm the presence of a lesion, the input unit 21 Operate the shooting button and lesion confirmation button on the This operation signal is input to the confirmation information acquisition unit 33 as confirmation information. In this embodiment, the confirmation information acquisition unit 33 is provided in order to perform support more efficiently, but it is not necessarily required.

The lesion disappearance determination unit 34 determines whether or not the lesion has disappeared from the endoscopic image based on the endoscopic image acquired by the image information acquisition unit 31 and the lesion information acquired by the lesion information acquisition unit 32. . Furthermore, in this embodiment, the lesion disappearance determination unit 34 determines whether or not the lesion has disappeared from the endoscopic image based on the confirmation information acquired by the confirmation information acquisition unit 33, without the doctor confirming the lesion. determine whether This determination result is output to the notification unit 35 .

The notification unit 35 performs alarm notification based on the determination result of the lesion disappearance determination unit 34 . The notification unit 35 controls the sound output unit 23 and the vibration unit 24 when the lesion disappearance determination unit 34 determines that the lesion has disappeared from the endoscopic image without confirmation of the lesion by the doctor. Then, an alarm sound is generated from the sound output section 23 and the vibrating section 24 is vibrated. With this alarm sound and vibration, the doctor can be notified that the automatically detected lesion has disappeared from the endoscopic image without confirming the detection of the lesion, thereby attracting the doctor's attention.

In addition, depending on the lesion classification results, the tone color, message content, sound volume and pitch are changed to output an alarm sound, and the wavelength and intensity are changed to generate vibrations according to the lesion classification results. may Further, weighting may be performed according to the classification result of the lesion, the degree of the condition, and the like, and the alarm sound and vibration may be changed according to the weight. As a result, the doctor's attention can be drawn more strongly, and more careful observation can be promoted.

The lesion information output unit 36 displays the endoscopic image corresponding to the lesion detected by the lesion information acquisition unit 32 on the display unit 22 as an output destination and stores it in the storage unit 17 . As a result, doctors can check the endoscopic image and lesion information at any time after the examination is completed and use it as a reference for pathological diagnosis, improving the support performance of endoscopic observation and reducing the storage capacity.

Moreover, it is desirable to display the lesion information on the display unit 22 or store it in the storage unit 17 in association with the endoscopic image. For example, if the lesion site on the endoscopic image is surrounded by a line and displayed on the display unit 22, the doctor can easily confirm the position of the automatically detected lesion, and the doctor's attention can be attracted. can.

Also, an alarm may be displayed on the display image of the display unit 22 . In this case, it is preferable to avoid displaying the lesion information in the endoscopic image in order to suppress the influence on the visibility of the endoscopic image. For example, the color of the area other than the endoscopic image is changed to attract the doctor's attention. Specifically, as shown in FIG. 5A and the like, on the display screen of the display unit 22, areas (four corners) outside the endoscope image display area 22a are used as an alarm display area 22b. In FIG. 5A, before a lesion such as a polyp is detected, the alarm display area 22b is displayed in a normal color (for example, black) without changing the color as shown in FIG. 5A. It should also remain black immediately after a lesion is detected.

After detection of the lesion, when the lesion disappears from the endoscopic image, the alarm display area 22b is displayed in a conspicuous color (for example, red) as shown in FIG. 5(b). Alternatively, it may be flashed. FIG. 5(c) is a modified example of the display example when the lesion disappears. The peripheral edge of the display screen is the alarm display area 22b, and when the lesion disappears, it is displayed in red. In this way, by changing the color and the like of the alarm display area 22b provided outside the display area 22a for the endoscopic image, the doctor or the like can clearly view the endoscopic image while only hearing sound and vibration. Disappearance of the lesion can be recognized visually as well.

5(d), (e), and (f) are other display examples of the display unit 22 when alarm notification is given by color change. 5A, 5B, and 5C, except that the endoscopic image display area 22a is substantially elliptical and the inner peripheral edge of the alarm display area 22b is also arcuate. displayed, and the same effect can be obtained.

Further, by storing the endoscopic image and lesion information in the storage unit 17, more detailed information on the lesion can be provided when the doctor makes a pathological diagnosis after the examination.

An example of endoscope observation support processing executed by the endoscope observation support apparatus 100 configured as described above will be described with reference to the flowchart of FIG. First, the doctor inserts the insertion section 42 into the body of the patient to be observed while operating the operation section 43 of the endoscope apparatus 40 . Following the insertion of the insertion portion 42, the imaging portion 41 captures an endoscopic image of the inside of the body in real time.

In step S<b>1 , the image information acquisition unit 31 acquires an endoscopic image (frame) captured by the imaging unit 41 , processes the image, and displays it on the display unit 22 . Then, based on the acquired endoscopic image, the processes after step S2 are performed, but these processes may be performed for each frame or for each several frames.

In step S<b>2 , based on the endoscopic image acquired by the image information acquisition section 31 , the lesion information acquisition section 32 automatically detects lesions by the above-described image recognition processing and learning function. FIG. 4(a) shows an example of an endoscopic image in which a lesion appears. At this time, even if the lesion is automatically detected, no notification is given from the sound output unit 23 or the vibration unit 24 . When a lesion is detected, the lesion information acquisition unit 32 acquires the position, size, state, and classification of the lesion on the endoscopic image, and generates lesion information. When a plurality of lesions are detected in one endoscopic image, each lesion information is generated.

In the next step S3, it is determined whether or not a lesion has been detected by the lesion information acquisition unit 32. If no lesion is detected (NO), the process skips steps S4 to S9 and proceeds to step S10.

On the other hand, if a lesion is detected in step S3 (YES), the process proceeds to step S4, and the confirmation information acquisition unit 33 acquires confirmation information in order to detect whether or not the doctor has performed a lesion confirmation operation. Next, in order to track the lesion on the confirmation information endoscopic image, the process proceeds to step S5, in which the image information acquisition unit 31 acquires the next frame of the endoscopic image, displays it on the display unit 22, and acquires the lesion information. A lesion is detected by the unit 32 . Next, in step S6, it is determined whether or not the lesion being tracked has disappeared from the endoscopic image.

If it is determined in step S6 that the lesion has not disappeared (NO), the process returns to step S4 to acquire confirmation information, and proceeds to step S5 to continue tracking the lesion. In this way, while the lesion appears on the endoscopic image, there is no notification that the lesion has been automatically detected. A pathological diagnosis or the like can be performed while observing the endoscopic image displayed on 22 .

On the other hand, if it is determined in step S6 that the lesion has disappeared (YES), the process advances to step S7 to determine whether confirmation information has been acquired for the endoscopic image before the lesion disappeared. If the confirmation information has been acquired (YES), the process proceeds to step S9. In step S<b>9 , the lesion information output unit 36 displays the endoscopic image before the lesion disappears and the lesion information on the display unit 22 and stores the endoscopic image and the lesion information in the storage unit 17 .

In this way, when the confirmation information is acquired, it can be determined that the lesion automatically detected by the lesion information acquisition unit 32 has also been confirmed by the doctor. That is, when a doctor who observes an endoscopic image or a microscope image on the display unit 22 detects a lesion, the input unit 21 issues an instruction to capture a still image including the lesion, or issues a confirmation instruction. A signal of confirmation information is input to the support device main body 10 . If a still image is instructed to be photographed, the still image is displayed and stored, and if only a confirmation instruction is given, the endoscopic image (frame) before the disappearance of the lesion is displayed and stored. .

On the other hand, in step S7, even though the lesion has disappeared, if it is determined that confirmation information from the doctor has not been obtained until just before the disappearance (NO), the process proceeds to step S8, and an alarm is issued. give notice. This is a state in which the result of automatic detection of a lesion by the lesion information acquiring unit 32 and the result of the lesion judgment by the doctor do not match, or in a state where the judgment agrees and the lesion is confirmed but the confirmation operation is not performed. It means that you have proceeded to shoot in the area of .

In the alarm notification process of step S8, the notification unit 35 controls the sound output unit 23 to generate an alarm sound, and controls the vibration unit 24 to vibrate. FIG. 4B shows an example of an endoscopic image in which the lesion has disappeared, and also schematically shows a state in which the sound output unit 23 outputs an alarm sound and the vibrating unit 24 vibrates. In addition, as shown in FIGS. 5(b), (c), (e), and (f), in the alarm display area 22b on the display screen of the display unit 22, a conspicuous color such as red is displayed or blinked. to notify the alarm.

This alarm sound and vibration, as well as the alarm display on the display unit 22, allow the doctor to notice that the lesion automatically detected by the computer has not yet been confirmed. In other words, there is a possibility that the doctor did not judge it as a lesion or overlooked the lesion. In addition, it is desirable to perform the alarm notification from disappearance of the lesion within 1 second. As a result, it is possible to prevent the endoscopic examination from being continued without confirmation of the lesion by the doctor. In addition, it is possible to return the imaging area of the imaging unit 41 to the area before the disappearance of the lesion by operating the operation unit 43 in the reverse direction. can increase

After that, by proceeding to step S9, the lesion information output unit 36 displays the automatically detected endoscopic image before disappearance of the lesion and the lesion information on the display unit 22 (or another display unit). The endoscopic image and lesion information are stored in the storage unit 17 . The doctor can also observe the suspected lesion area from the endoscopic image displayed on the display unit 22 or the like. Also, the endoscopic image of the lesion and the lesion information stored in the storage unit 17 can be reconfirmed after the examination. After that, the process proceeds to step S10.

In step S10, it is determined whether or not the imaging of the endoscopic image by the endoscope apparatus 40 has been completed. If it is determined that the imaging has been completed (YES), the examination has been completed and the endoscopic observation support process is executed. finish. On the other hand, if it is determined that imaging is being continued (NO), the process returns to step S1 to continue the endoscopic observation support processing for the next endoscopic image.

As described above, the endoscopic observation support device 100 of the present embodiment includes the image information acquisition unit 31 that acquires the captured image of the hollow organ captured by the endoscope device 40 and displays it on the display unit 22. a lesion information acquiring unit 32 for detecting a predetermined lesion based on the captured image and acquiring lesion information about the lesion; and a notification unit 35 for notifying the determination result when the lesion disappearance determination unit 34 determines that the lesion has disappeared from the captured image. .

In addition, the endoscopic observation support method of the present embodiment includes the steps of acquiring a captured image of a hollow organ captured by the endoscope device 40, detecting a predetermined lesion based on the captured image, and obtaining lesion information; tracking the lesion based on the captured image and the lesion information; determining whether the lesion has disappeared from the captured image; and determining that the lesion has disappeared from the captured image. Sometimes, the step of notifying the determination result.

Further, the program for executing the endoscopic observation support method of the present embodiment includes means for acquiring a captured image of a hollow organ captured by the endoscope apparatus 40, and based on the captured image, Means for detecting a predetermined lesion and acquiring lesion information about the lesion, means for tracking the lesion based on the captured image and the lesion information, and determining whether or not the lesion has disappeared from the captured image, and This is a program for functioning as means for notifying the determination result when it is determined that the image has disappeared.

Therefore, in the present embodiment, an operator such as a doctor observes the hollow organ by visually recognizing the captured image (endoscopic image) displayed on the display unit 22, and performs pathological diagnosis, treatment, and the like. be able to. On the other hand, it is possible to automatically detect a lesion with high accuracy based on the captured image. While the lesion appears on the captured image, the automatic detection result is not notified prior to the doctor's decision. It can be carried out.

On the other hand, when the lesion disappears from the captured image, the doctor is notified that the lesion has been automatically detected and that the detected lesion has disappeared from the captured image. . At this time, if the doctor himself/herself confirms the lesion on the captured image, he or she will find that the result of his/her own diagnosis matches the result of the automatic detection. . In addition, this notification can prompt the doctor to observe the vicinity of the lesion again or to observe more carefully. Therefore, the endoscopic observation support apparatus 100, the endoscopic observation support process, and the endoscopic observation support apparatus 100 that can improve the performance of the observation support by the endoscope and enable the doctor to observe the lesion more easily and accurately. program can be provided.

Further, in the present embodiment, lesions can be automatically detected with high accuracy as described above, and when the automatically detected lesions disappear from the captured image, a notification is given. It can also be suitably used for training in pathological diagnosis, etc. In other words, even if an inexperienced doctor continues the examination while overlooking the lesion when making a pathological diagnosis based on the captured image, the doctor is promptly notified that the lesion has disappeared from the captured image. Therefore, the doctor can notice an oversight and facilitate reconfirmation. In addition, since no notification is given while the lesion appears on the captured image, the doctor can perform pathological diagnosis without being unnecessarily affected by the automatic detection results. As a result, a doctor's pathological diagnosis ability can be improved.

Further, in this embodiment, the notification unit 35 is configured to notify the determination result by outputting a predetermined notification sound from the sound output unit 23 . In addition, the notification unit 35 is configured to notify the determination result by vibrating the vibrating body (vibration unit 24). The notification sound and vibration can attract the doctor's attention more strongly, and can more clearly notify the doctor that the lesion has disappeared from the captured image.

Further, in the present embodiment, when the lesion disappearance determination unit 34 determines that the lesion has disappeared from the captured image, the captured image of the lesion before disappearance is output to a predetermined output destination (for example, the display unit 22, the storage unit 1, or the like).
7) is provided with a lesion information output unit 36 for outputting. As a result, the doctor can check the endoscopic image and lesion information at any time after the end of the examination and refer to the pathological diagnosis, thereby improving the support performance of the endoscopic observation.

Further, in the present embodiment, the input unit 21 is provided for receiving input of a lesion confirmation instruction from the operator. The determination result is notified when no confirmation instruction is input from the input unit 21 . As a result, even if the lesion disappears, if the doctor who is the operator has confirmed the lesion, no notification will be sent. can be done more efficiently.

(Second embodiment)
Next, an endoscope observation support device according to a second embodiment will be described. In the above-described first embodiment, when it is determined that the lesion has disappeared from the endoscopic image, an alarm is issued to notify that the lesion has disappeared. On the other hand, in the second embodiment, an alarm notification that a lesion has been detected is issued a short time after the lesion is detected on the endoscopic image. In other words, regardless of whether or not the lesion has disappeared from the endoscopic image, the alarm is notified with a delay from the time of detection.

The endoscopic observation support apparatus of the second embodiment has the same basic configuration as that of the first embodiment except that the lesion disappearance determination unit 34 is not provided, so detailed description thereof will be omitted. The confirmation information acquisition unit 33 may also be omitted or may be provided. In addition, the endoscopic observation support method of the second embodiment includes the steps of acquiring a captured image of a hollow organ captured by the endoscope device 40, detecting a predetermined lesion based on the captured image, Obtaining lesion information about the lesion; and notifying that the lesion has been detected a short time after the lesion was detected. Further, a program for executing the endoscopic observation support method of the second embodiment includes means for acquiring a captured image of a hollow organ captured by the endoscope apparatus 40 and a computer based on the captured image. A program for functioning as means for detecting a predetermined lesion and obtaining lesion information about the lesion, and as means for notifying that a lesion has been detected after a short period of time has passed since the lesion was detected. .

In the endoscopic observation support apparatus of the second embodiment, when the lesion information acquisition unit 32 detects a lesion such as a polyp, it notifies the notification unit 35 of it. A notification part 35 does not notify in real time when a lesion is detected, but notifies that the lesion is automatically detected after a short time has passed since the lesion was detected, that is, after a predetermined time has passed. As in the first embodiment, this notification is made by generating an alarm sound from the sound output unit 23, vibrating the vibrating unit 24, further changing the color of the alarm display area 22b as shown in FIG. can be done.

Here, "short time" means the time required for a lesion to appear in an endoscopic image and for a doctor or the like who visually recognizes it to determine that it is a lesion. More specifically, 30 ms to 2,000 ms is preferable as the “short time”. Moreover, it is more preferable to be able to arbitrarily set or change the "short period of time" within this range. As a result, even if the determination time varies depending on the doctor's degree of proficiency in endoscopic examination, age, etc., it is possible to set a suitable time according to the doctor or the like. Therefore, more efficient examination can be performed, and usability of the endoscope apparatus 40 is improved.

In this way, by notifying the detection of a lesion after a short period of time, it is possible to ensure time for a doctor or the like to determine that there is a lesion based on the endoscopic image. Therefore, the doctor or the like can determine whether or not there is a lesion by himself/herself without being influenced by the automatic detection result of the lesion by the lesion information acquisition unit 32 .

After a short period of time has passed, the lesion information acquisition unit 32 is notified that a lesion has been detected, so the doctor or the like can confirm whether his/her own judgment is the same as the result of automatic detection by artificial intelligence or the like. It is possible to improve the accuracy of judgment by a doctor or the like.

The second embodiment can also be suitably used when conducting pathological diagnosis training. For example, when an inexperienced doctor performs endoscopic observation at the beginning of training, setting a long "short time" can give the doctor a long time for judgment. By gradually shortening the “short time” every time the training is repeated, the doctor can gradually make a quick decision.

Furthermore, as described above, by analyzing a plurality of image frames of a captured image using artificial intelligence, it is possible to diagnose a lesion with higher accuracy than by analyzing one frame. In the conventional method of notifying detection of a lesion in real time, there is a limit to increase the number of image frames to be analyzed because speed is prioritized. On the other hand, when the notification is given after a predetermined time from detection of the lesion as in the present embodiment, or when the notification is given after the lesion disappears as in the first embodiment, a predetermined analysis time is required. Since it can be reserved, more frames can be analyzed. Therefore, it is possible to obtain more detailed lesion information such as the type and degree of the lesion, and to improve the accuracy of lesion detection by artificial intelligence and the accuracy of lesion information.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the above-described embodiments are merely examples of the present invention, and the present invention is not limited only to the configurations of the above-described embodiments. It goes without saying that the present invention includes any design changes that do not deviate from the gist of the present invention.

For example, in each of the above-described embodiments, the endoscope device 40 having the imaging unit 41, the insertion unit 42, and the operation unit 43 is connected to the support device main body 10, and the support device main body 10 captures an image captured by the imaging unit 41. The image is displayed on the display unit 22 after image processing. On the other hand, in addition to the imaging section 41, the insertion section 42, and the operation section 43, an endoscope unit having a function of displaying a captured image such as a video tuner or a display section may be used. Further, the display section of such an endoscope unit may also be used as the display section 22 of the endoscope observation support device 100, or the display section 22 may be provided separately.

Furthermore, the endoscope unit may be configured to include an information processing device having an image processing section that automatically detects a lesion based on a captured image. In this case, if the lesion information acquisition unit 32 of the support device main body 10 acquires the lesion information acquisition unit 32 of the support device main body 10 and uses it for endoscopic observation support processing, the endoscopic unit has The automatic detection result can be effectively used, the load of arithmetic processing in the support device main body 10 can be reduced, and the arithmetic speed can be increased.

Further, the endoscope observation support device 100 of each embodiment may be incorporated in an endoscope unit including an information processing device and the like. That is, the endoscope unit is provided with a sound output unit 23 and a vibration unit 24, and the information processing device of the endoscope unit is also used as the information processing unit 20, and the endoscope observation support program of each embodiment is installed. do. This makes it possible to provide an endoscope unit equipped with an endoscope observation support device. Therefore, it can be sold as an endoscope unit equipped with an endoscope observation support device, or the endoscope observation support program can be sold separately and installed in an existing endoscope unit for use. can also In addition, each device can be made compact and simplified, and handling can be simplified.

In each of the above-described embodiments, the doctor is notified that the lesion has disappeared from the captured image by a notification sound or vibration. The error may be transmitted by In this case as well, the weighting of lesions and the color of emitted light, blinking time, and the like may be appropriately adjusted according to the type of lesion, making it possible to provide more understandable notification.

In addition, a time measurement unit for measuring the time required for inserting the insertion portion 42 into the body is provided, and the insertion speed of the insertion portion 42 (may be measured with a speedometer, or an average insertion speed may be used). For example, in the case of a large intestine endoscope, the distance from the anal verge may be calculated and displayed on the display unit 22 based on the insertion time and the captured image. Further, when the lesion disappears from the captured image and the image of the lesion is displayed on the display unit 22 or the like, if the distance from the anal verge of the lesion is displayed, the lesion can be photographed by moving back several centimeters. A doctor can more easily and more accurately determine whether or not there is a lesion, and the performance of lesion observation support can be further improved.

Further, in the first embodiment, an alarm is notified when the lesion disappears, and in the second embodiment, the alarm is notified after a short time has passed since the lesion was detected. It is not limited. Depending on the purpose of use, etc., an alarm notification may be given both when the lesion disappears and when a short time has passed since the lesion was detected, so that the doctor's attention can be drawn more strongly. .

17 storage unit (output destination) 20 information processing unit 21 input unit 22 display unit (output destination) 23 sound output unit 24 vibration unit (vibrating body)
31 image information acquisition unit 32 lesion information acquisition unit 34 lesion disappearance determination unit (determination unit)
35 notification unit 36 lesion information output unit 40 endoscope device (endoscope)
100 Endoscope observation support device

Claims (5)

A training colonoscopy observation support device used for training in colonoscopy observation of the large intestine,
an image information acquisition unit that acquires a captured image of the large intestine captured by the colonoscope and displays it on a display unit;
a lesion information acquisition unit that detects a predetermined lesion based on the captured image and acquires lesion information about the lesion;
A notification unit that notifies that the lesion has been detected after a predetermined time has elapsed since the lesion was detected ,
The notification unit gradually sets the predetermined time within a range of 30 ms to 2,000 ms each time the operator of the colonoscope repeats training.
A colonoscopy observation support device for training characterized by: 2. The apparatus according to claim 1 , wherein the notification unit changes the predetermined time within a range of 30 ms to 2,000 ms according to the skill level and age of the operator of the colonoscope. colonoscopy observation support device for training . The notification unit is configured to change the color of an alarm display area of a predetermined color provided in an area outside the display area of the captured image of the display unit or on the outer peripheral edge from the predetermined color to another color. configured to notify an alarm that the lesion has been detected by performing at least one of blinking;
3. A lesion information output unit for displaying the captured image of the lesion and the lesion information on the display unit when the predetermined time has elapsed since the lesion was detected. 2. The colonoscopy observation support device for training according to . An operation method for operating the colonoscopy observation support device for training according to any one of claims 1 to 3 ,
a step in which the image information acquisition unit acquires a captured image of an internal organ of the large intestine captured by a colonoscope and displays the captured image on a display unit;
a step in which the lesion information acquisition unit detects a predetermined lesion based on the captured image and acquires lesion information about the lesion;
When the notification unit notifies that the lesion has been detected after a predetermined time has passed since the lesion was detected, every time the operator of the colonoscope repeats training, the predetermined time , a step of notifying while setting gradually shorter within the range of 30 ms to 2,000 ms ;
A method of operation , comprising: A program for causing a computer to execute each step according to claim 4 .

Images