JP7127779B2 - Diagnostic support system and diagnostic support program - Google Patents

Description

Article 30, Paragraph 2 of the Patent Act applies [Disclosure by presentation at a meeting] ・Meeting name: The 95th Annual General Meeting of the Japan Gastroenterological Endoscopy Society Date: May 11, 2018 Venue: Grand Prince New Takanawa International Hall Pamir ・Meeting name: Digestive Disease Week (DDW2018) Date: June 2, 2018 Venue: Walter E. Washington Convention Center ・Meeting name: The 96th Annual General Meeting of the Japan Gastroenterological Endoscopy Society Date: November 3, 2018 Venue: Kobe Convention Center [Published by publication] ・Publication name: PR TIMS Publication date: 2018 August 17, 2018 ・Publication name: IoT NEWS Publication date: August 17, 2018 ・Publication name: iza: Iza! Publication date: August 17, 2018 ・Publication name: Sankei News Publication date: August 17, 2018 ・Publication name: Excite Publication date: August 17, 2018 ・Publication name: Infoseek News Publication date: August 17, 2018 ・Publication name: CNET Japan Date of issue: August 17, 2018 ・Publication name: Yomiuri Shimbun Date of issue: January 16, 2019 ・Publication name: yomiDr. (Yomi Doctor) Publication date: January 16, 2019 [Published on website] Telecommunications line announcement: October 1, 2018 Posting address: http://www. jddw. jp/jddw2018/abstracts/index. php/login

The present invention relates to a diagnostic support system and a diagnostic support program.

The morbidity and mortality of colorectal cancer in Japan are increasing year by year. In order to prevent the disease and improve the results of treatment, it is important to detect and remove neoplastic polyps at an early stage by colonoscopy.

However, the current situation is that the accuracy of examination of neoplastic polyps depends on the skill of the endoscopist and the performance of the machine. Serrated lesions of the colon can be classified into hyperplastic polyps of the colon (HP), serrated adenomas (TSA), and broadly basal sessile serrated lesions (SSA/P). A TSA is a distinct neoplastic lesion. On the other hand, SSA/P was not considered as a lesion in the past, but in recent years it has attracted attention as a precursor lesion. Therefore, it is required to accurately detect SSA/P from endoscopic images and use it as a target for tissue diagnosis prediction (classification).

However, it is difficult even for a skilled endoscopist to accurately discriminate between HP and SSA/P, and it is also difficult for an image processing apparatus to perform accurate classification. Therefore, there is a demand for a diagnostic support system capable of improving examination accuracy regardless of the skill of the endoscopist or the performance of the machine. Such problems occur not only in colonoscopy but also in diagnosis of other human tissues.

In Patent Document 1, the cell nuclei contained in the endoscopic image obtained by magnifying the mucosal cells of the gastrointestinal tract with an endoscope are extracted, and texture analysis is performed to analyze nontumor, adenoma, cancer, and SSA/P. A diagnostic assistance system for identification is disclosed. However, the technique of Patent Document 1 uses an endoscope with a high-magnification imaging function to provide diagnostic support based on super-magnified images, and cannot be used for general purposes even in ordinary endoscopes. do not have. In addition, it is premised on the use of super-enlarged images, and does not provide effective means for solving the problem of overlooking lesions.

JP 2017-70609 A

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and provides a diagnostic support system capable of classifying lesions at an appropriate timing according to endoscopic skills while preventing lesions from being overlooked. It is intended to

In order to solve the above problems, the diagnostic support system according to the present invention includes a display unit for displaying an image obtained by an imaging device, and a specific site for detecting a specific site having characteristics of a lesion or a precursor lesion in the image. a detection unit, a display control unit for displaying a graphic surrounding the specific site specified by the specific site detection unit in the display unit, and a tissue diagnosis prediction unit for predicting a tissue diagnosis for the specific site according to a predetermined standard. and when the condition that the image is in a static state is satisfied while the detection algorithm for detecting the specific site is executed by the specific site detection unit, the tissue diagnosis prediction unit predicts the tissue diagnosis. and a controller for initiating a tissue diagnosis algorithm for performing the tissue diagnosis.

This system can further include an image state detector that determines the state of the image. In this case, the control section can execute the detection algorithm and tissue diagnosis algorithm in parallel according to the determination result of the image state detection section. Here, the image state detection unit can be configured to determine whether or not the image is in a static state.

Further, the tissue diagnosis prediction unit can be configured to calculate a certainty factor indicating the possibility that a lesion exists at the site based on the overlap rate of the graphics between a plurality of frames forming the image.

According to the diagnostic support system according to the present invention, it is possible to provide a diagnostic support system capable of preventing lesions from being overlooked and classifying lesions at appropriate timing according to endoscope skills.

1 is a schematic diagram of a diagnosis support system 1 of a first embodiment; FIG. It is a schematic diagram showing an outline of operation of diagnosis support system 1 of a 1st embodiment. 4 is a flow chart showing an overall procedure of operation of the diagnostic support system 1 of the first embodiment; FIG. 4 is a flowchart for explaining the details of the procedure of image acquisition (step S16) in FIG. 3; FIG. FIG. 4 is a flow chart illustrating the details of the freeze determination (step S17) procedure in FIG. 3; FIG. FIG. 4 is a flow chart illustrating details of a procedure for drawing (for detection) (step S18) of the user interface screen of FIG. 3; FIG. FIG. 4 is a flow chart illustrating the details of the procedure for drawing the user interface screen of FIG. 3 (for tissue diagnosis prediction (classification)) (step S18); FIG. FIG. 4 is a flow chart showing the details of the procedure of bounding box estimation (step S20) in FIG. 3; FIG. 1 is a schematic diagram of a diagnosis support system 1 of a second embodiment; FIG. It is a schematic diagram showing a modification of each embodiment.

Hereinafter, this embodiment will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be labeled with the same numbers. It should be noted that although the attached drawings show embodiments and implementation examples in accordance with the principles of the present disclosure, they are for the purpose of understanding the present disclosure and are in no way used to interpret the present disclosure in a restrictive manner. is not. The description herein is merely exemplary and is not intended to limit the scope or application of this disclosure in any way.

Although the present embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure, other implementations and configurations are possible without departing from the scope and spirit of the present disclosure. It is necessary to understand that it is possible to change the composition/structure and replace various elements. Therefore, the following description should not be construed as being limited to this.

First, a diagnosis support system according to the first embodiment of the present invention will be described in detail.

(Hardware configuration)
FIG. 1 is a schematic diagram for explaining the hardware configuration of a diagnostic support system 1 according to the first embodiment. This diagnostic support system 1 is roughly composed of an endoscope 100 (imaging device), a processor 200 , a display 300 and a light source section 400 .

The endoscope 100 is configured to be insertable into the body of a subject, for example, the large intestine, and has a function of capturing an image of the subject and transmitting an image signal of the captured image to the processor 200 . The processor 200 receives an image signal from the endoscope 100, performs predetermined signal processing, and performs detection and prediction of tissue diagnosis, which will be described later.

The endoscope 100 includes an insertion section 101 , a hand operation section 102 , a bending section 103 , a tip section 104 , a universal cable 105 and a connector section 106 . The insertion section 101 is flexible, has a light guide, various wirings, a water pipe, etc. along its longitudinal direction, and is inserted into the body of the subject. One end of the insertion portion 101 is connected to the handheld operation portion 102 . The hand operation unit 102 includes, for example, a curved operation knob 102A and an operation button 102B. The operation buttons 102B may include an air supply button for supplying airflow, a water supply button for supplying water flow, and a freeze button 102Bf for obtaining a still image from an image being captured.

A bending portion 103 configured to be bendable is provided at the distal end of the insertion portion 101 . The bending portion 103 is bent by pulling an operation wire (not shown) interlocked with the turning operation of the bending operation knob 102A provided on the hand operation portion 102 . Furthermore, a distal end portion 104 having an imaging device such as a CCD or CMOS sensor is connected to the distal end of the bent portion 103 . The distal end portion 104 is configured such that its orientation is changed according to the bending motion of the bending portion 103 by the rotating operation of the bending operation knob 102A.

Hand operation unit 102 and connector unit 106 are connected by universal cable 105 . The universal cable 105 includes a light guide and various wiring inside. The connector section 106 includes various connectors for connecting the endoscope 100 to the processor 200 .

The processor 200 performs predetermined image processing on the image captured by the endoscope 100 and causes the display 300 to display the image. In addition, the processor 200 analyzes the obtained image, detects a specific site from the image, and classifies the detected site according to its properties (histologic diagnosis prediction). As an example, the processor 200 can classify detected sites into colonic hyperplastic polyps (HP), serrated adenomas (TSA), broadly basal sessile serrated lesions (SSA/P), and the like. configured to

The display 300 displays, for example, a user interface (UI) 301, an observed image display section 302, and a captured image display section 303 on the display screen. The UI 301 is an interface for inputting or displaying, for example, display contents, a detection method, whether or not to collect data for learning, and various other commands and parameters relating to the operation of this diagnosis support system. The observation image display unit 302 displays an image captured by the endoscope 100 and also displays a bounding box BX indicating the position of the detected specific region. The bounding box BX is an example of a figure for indicating the position of a specific portion having predetermined characteristics, and although the illustrated example is a rectangle, it goes without saying that the bounding box BX is not limited to this. Any shape is acceptable as long as its existence can be recognized. The captured image display unit 303 is a part that captures (stores), for example, a frame that precedes the frame that is currently being captured, and then displays it, instead of the image that is currently being captured (in real time). The captured image display unit 303 can also function as a display unit that outputs the detection result as a report when a new lesion is detected.

The details of the configuration of processor 200 will now be described with continued reference to FIG. The processor 200 includes, for example, an image processing unit 201, an interface 202, a hard disk drive 203, a ROM 204, a RAM 205, an input unit 206, an interface 207, a CPU 208, a display control unit 209, a specific part detection unit 210, a tissue diagnosis prediction unit 211, an image A state detection unit 212 and a learning data generation unit 213 are provided.

The image processing unit 201 performs predetermined image processing on a video signal received from a solid-state imaging device (not shown) of the endoscope 100 and converts it into a digital signal. A video signal converted into a digital signal is taken into the CPU 208 via the interface 202 . The hard disk drive 203 stores a computer program for controlling the operation of the diagnostic support system in this system and a data group used for the operation. Colonoscopy data includes colonoscopy images and diagnostic data about the lesion.

The ROM 204 is a storage device for storing, for example, setting files for the computer program and other operating parameters of the device. etc. is temporarily stored. The input unit 206 is a device such as a well-known keyboard, mouse, numeric keypad, joystick, touch panel, etc., for the user to perform various inputs, and transmits data to the CPU 208 via the interface 207 . The input unit 206 also includes an operation button 102B provided on the endoscope 100. FIG. In addition, endoscope 100 and/or processor 200 may be provided with an end button (not shown) for instructing the end of the examination.

The display control unit 209 is a control unit that controls screen display on the display 300 . The display control unit 209 is configured to be able to change the display state of the display 300 according to the detection result and also change the display state according to the input from the input unit 206 .

The specific site detection unit 210 is a module that detects a specific site (lesion, precursor lesion, etc.) from the captured image according to the detection criteria (detection model) determined by the learning data described above. 200 can be implemented in software. When the specific site is detected, the specific site detection unit 210 outputs data including the position on the captured image to the CPU 208 . The display control unit 209 receives the information from the CPU 208 and displays the bounding box BX at the corresponding location. Since the bounding box BX is displayed on multiple displays 300 during the colonoscopy, the endoscopist can continue the examination while paying attention to the display of the bounding box BX. Note that the specific site detection unit 210, the tissue diagnosis prediction unit 211, and the image state detection unit 212 may be realized by hardware such as a DSP (Digital Signal Processor) instead of by software.

The tissue diagnosis prediction unit 211 has a function of performing tissue diagnosis prediction (classification) according to the tissue diagnosis prediction model for the specific site detected by the specific site detection unit 210 when a predetermined condition is satisfied, Similarly, it can be implemented in software within the processor 200 by the aforementioned computer program. Here, the predetermined condition is, for example, the case where the image state detection unit 212 detects a freeze state described below. A tissue diagnosis prediction model can be generated based on, for example, logistic regression, naive Bayes classifier, support vector machine, neural network, etc., but is not limited to any particular one. As the neural network, both a forward propagation neural network and a recurrent neural network can be used. In addition to the tissue diagnosis prediction unit 211, a lesion discrimination unit that discriminates the type of lesion may be further provided.

The image state detection unit 212 is a module that detects whether or not the video signal is in a predetermined state, and likewise can be implemented in software within the processor 200 by a computer program. For example, the image state detection unit 212 detects whether the change per unit time of the image being captured is less than a predetermined amount. As schematically shown in FIG. 2, when an endoscope 100 is used to image the large intestine, an endoscopist D, who is the user of this system, inserts and withdraws the insertion portion 101 of the endoscope 100 into and out of the large intestine LB. perform the action (arrow A). The movement of the distal end portion 104 in the large intestine LB due to this movement of putting in and out changes the picked-up image moment by moment.

However, when the endoscopist D stops moving the insertion portion 101 or presses the freeze button 102Bf, the captured image stops changing or changes very little. Hereinafter, a state in which the change in the captured image per unit time is less than a predetermined value due to stopping the movement of the insertion portion 101 or pressing the freeze button 102Bf will be referred to as a "freeze state." (static state)”. The image state detection unit 212 is a module having a function of detecting such a freeze state (still state). Specifically, the image state detection unit 212 determines whether the captured image is in a frozen state based on a comparison between a certain frame (first frame) and a frame (second frame) temporally before that frame. It is possible to determine whether Note that the image state detection unit 212 can also detect whether the image being captured is a white light image, an NBI (Narrow Band Imaging (registered trademark)) image, or a dyed image as well as the detection of the frozen state. ing.

In this embodiment, the specific part detection unit 210 always operates during execution of the examination, and the specific part detection unit 210 operates regardless of the user's input or detection result (examination end button including end button). Unless otherwise instructed), continue to detect specific parts. On the other hand, the tissue diagnosis prediction unit 211 starts operating only when a specific condition is satisfied, and performs tissue diagnosis prediction in parallel with the detection of the specific site by the specific site detection unit 210 . That is, the CPU 208 and the program are configured to concurrently execute the detection algorithm by the specific site detection unit 210 and the tissue diagnosis algorithm by the tissue diagnosis prediction unit 211 according to the determination result of the image state detection unit 212 . As described above, in this embodiment, while the specific site detection unit 210 is always operated, the tissue diagnosis prediction unit 211 can be operated in parallel with the specific site detection unit 210 only when a specific condition is satisfied. As a result, it is possible to predict the tissue diagnosis at an appropriate timing according to the endoscopist's skill, etc., while preventing the oversight of a specific site.

For example, an endoscopist whose skill is not necessarily high frequently stops (or slows down) the movement of the insertion portion 101 or presses the freeze button 102Bf frequently, so that the detected measurement site is frequently Tissue diagnostic prediction can be performed. As a result, the result of tissue diagnosis prediction can be displayed on the display 300 at a high frequency according to the skill of the endoscopist, while preventing the oversight of a specific site.

On the other hand, a highly skilled endoscopist operates the insertion unit 101 at a speed suitable for his/her own skill, stops the movement of the insertion unit 101 at a timing suitable for his/her own diagnosis skill, or identifies a point that catches the eye. By pressing the freeze button 102Bf at the position of the site, tissue diagnosis prediction of the specific site can be executed. Detection of a specific site by the specific site detection unit 210 is always performed as long as the image is captured by the endoscope 100 regardless of the skill of the endoscopist. Depends on medical skill. When a highly skilled endoscopist performs an examination in this system, the frequency of tissue diagnosis prediction by the tissue diagnosis prediction unit 211 is lower than when an endoscopist who is not necessarily highly skilled performs an examination. Become. That is, tissue diagnosis prediction can be performed at a timing that matches the skill of the endoscopist. In other words, according to the system of the present embodiment, it is possible to prevent unnecessary information from being displayed for a highly skilled endoscopist.

(motion)
Next, the operation of the diagnostic support system according to the embodiment will be described with reference to the flow charts of FIGS. 3 to 8. FIG. The flow chart of FIG. 3 shows the overall procedure of operation of the system, and FIGS. 4-8 explain some steps in the flow chart of FIG. 3 in more detail. The operations shown in these flowcharts are defined according to a computer program stored in hard disk drive 203 .

(overall overview of operation)
As shown in FIG. 3, when the examination by the diagnostic support system is started, first, the CPU 208 reads parameters necessary for determination and detection by software from the setting file stored in the ROM 204 (step S11). Then, the user interface is activated (step S12) and displayed on a part of the display 300. FIG.

After the parameters read in step S12 are input and stored in the RAM 205, the user performs an operation to instruct the start of the operation of the diagnostic support program (step S13). The operation for instructing the start may be an instruction by clicking an icon on the user interface 301 , or may be replaced by an instruction simply by detecting the start of the operation on the endoscope 100 .

After starting the operation, the diagnostic support program creates a child thread in step S14, and thereafter the parent thread and the child thread are executed in parallel. The parent thread (steps S15 to S19) is always continued unless the end button is pressed. On the other hand, the child thread (step S20: bounding box estimation) is also continuously executed with respect to the display processing of the bounding box BX. Executed only if affirmative (YES). If the freeze determination result is negative (NO), the child thread (step S20) only detects the specific site and displays the bounding box BX, but does not execute tissue diagnosis prediction.

In the parent thread (steps S15 to S19), it is determined whether or not the end button has been pressed (step S15). If not pressed, the process proceeds to step S16. After terminating the thread, the operation ends.

In step S<b>16 , the image captured by the endoscope 100 is acquired by the image processing section 201 . The acquired image is passed to the child thread (processing in the child thread will be described later). In the parent thread, in subsequent step S17, it is determined whether or not the captured image is in a frozen state. The judgment result is passed to the child thread.

In the following step S18, a bounding box BX, a "label" indicating the result of tissue diagnosis prediction (judgment regarding the type of lesion), and the degree of certainty of tissue diagnosis prediction (possibility of corresponding to a predetermined lesion) are displayed on the obtained image. A user interface screen on which information such as gender) is superimposed is drawn and displayed on the display 300 . The label information and confidence information are received and drawn only at the timing when tissue diagnosis prediction is executed by the tissue diagnosis prediction unit 211 . Only the bounding box BX is displayed at the timing when tissue diagnosis prediction is not executed. In addition to the bounding box BX, the label, and the degree of certainty, the display items may include information such as the site of the lesion, the number of lesions, and the benign/border/malignant determination.

The image data captured and drawn in this manner is collected by the learning data generation unit 213, and based on these image data, the learning data generation unit 213 generates learning data (step S19). The learning data is also stored in the hard disk drive 203 in advance, and such prior learning data and newly generated learning data can be combined to form a new learning data group. It is also possible to use only the learning data stored in advance in the hard disk drive 203 and omit this step S19.

(Image acquisition (step S16))
Next, with reference to FIG. 4, the details of image acquisition in step S16 will be described. First, in step S<b>31 , image data (eg, 1920×1980 pixels) is read from a capture board (not shown) of the image processing unit 201 . Subsequently, in step S32, a predetermined portion of the read image data is cut according to the size designated by the Config file or the like.

Next, the image cut out in step S32 is resized to a size suitable for detection and generated as an image for detection and classification, and resized to a size suitable for display on the display 300 as an image for display. generated (steps S33, S34).

(Freeze determination (step S17))
Next, details of the freeze determination in step S17 of FIG. 3 will be described with reference to FIG. Freeze determination is performed in the image state detection unit 212 in FIG. First, the display image obtained in step S16 (step S34 in FIG. 4) is resized to a smaller number of pixels (eg, 30×30 pixels) (step S41), and further converted into a grayscale image (step S42). .

Subsequently, a weighted sum of the pixel value of each pixel of the grayscale image and the pixel value of each pixel of the grayscale image in the temporally previous frame is calculated (step S43). Then, the difference between the weighted sum and the pixel value of the current frame is calculated (step S44).

In step S45, it is determined whether the difference calculated in step S44 has fallen below the threshold two times in a row, or whether it has been within 20 frames since the last fall below the threshold. In the case of YES, the image state detection unit 212 determines that the target image is in a frozen state (step S46), and outputs the freeze determination result. In the case of NO, it is determined that the state is not frozen (step S47). Then, it is determined whether or not 30 frames or more have passed since the difference calculated in step S44 last fell below the threshold value (step S48). If YES, it is determined that a location different from that of 30 frames before is being photographed, and after clearing (erasing) the accumulation result of the difference value, etc. (step S49), it is determined that the state is not in a frozen state. is output and the freeze determination ends. On the other hand, if the determination result in step S48 is NO, the determination result that the state is not frozen is output while the integration result remains unchanged, and the same freeze determination operation is continued.

(User interface drawing (for detection) (step S18))
Subsequently, details of the procedure for drawing the user interface screen in step S18 of FIG. 3 (when only the detection of the specific part is continued instead of the frozen state) will be described with reference to the flowchart of FIG. In other words, the flowchart of FIG. 6 describes the details of the operation of step S18 when it is determined in step S17 of the flowchart of FIG. 3 that the state is not frozen.

In step S17 of FIG. 1, when the image state detection unit 212 determines that the state is not frozen (step S51 of FIG. 6), the process proceeds to step S52. Note that when it is determined that the system is in a frozen state, a classification user interface screen for tissue diagnosis prediction is drawn. This will be discussed later.

In step S52, it is detected whether or not a specific portion where the bounding box BX should be drawn has been detected. At this time, the specific parts whose certainty is equal to or less than the threshold are excluded from detection targets, and the bounding box BX is not displayed either.

If the determination result in step S52 is YES, in step S53 a bounding box BX is drawn around the specific part. When there are multiple detected specific parts, multiple bounding boxes BX are also drawn. Note that the processor 200 can generate a sound effect from a speaker (not shown) in response to the display of the bounding box BX.

If the determination result in step S52 is NO, the process proceeds to step S61, and the drawn image is displayed on display 300 as a user interface screen.

After the bounding box BX is drawn in step S53, the area of the bounding box BX (if there are a plurality of bounding boxes BX, the total area) is calculated in step S54. Then, it is determined whether or not the ratio of the area to the display screen of display 300 is equal to or greater than a predetermined value, for example, whether or not it is equal to or less than 1% of the display screen area.

If the determination result in step S54 is YES, the process advances to step S55 to continue displaying the frame, for example, for five frames or more. In this way, when the number of bounding boxes BX to be displayed is small, the bounding boxes BX are displayed over a plurality of frames (for example, 5 frames) to make it easier for the user to perceive the bounding boxes BX.

On the other hand, when the determination result of step S54 is NO (when the area of the bounding box BX is larger than 1%), the process proceeds to step S56. In step S56, it is determined whether or not it is within 30 frames from the previous reproduction of the effect sound emitted with the display of the bounding box BX. If it is within 30 frames, it is unnecessary to emit the sound effect again for detection of a lesion or the like, and there is a possibility that the user may feel uncomfortable. Therefore, if the determination result in step S56 is YES, the process proceeds to step S61 described above, displays the UI-drawn image without producing sound effects, and terminates the operation of FIG.

On the other hand, if the determination result of step S56 is NO, it will transfer to step S57. In step S57, it is determined whether or not it is within 40 frames from the reproduction of the previous sound effect. That is, if 30 frames have passed since the previous reproduction of the sound effect, but it is within 40 frames (YES in step S57), the sound effect is reproduced, and then the count value of the timer that measures the timing of reproducing the sound effect. is reset (step S59), and the process proceeds to step S60. On the other hand, if 40 frames have passed since the previous reproduction of the sound effect (NO in step S57), then in step S58, the image is captured (saved) and displayed on the side of the image display unit 302 that displays the real-time image. is displayed on the captured image display unit 303, and the sound effect is reproduced. Then, the drawn image is displayed on the display 300 as a user interface screen (step S60). 30 frames and 40 frames, which are the determination criteria in steps S57 and S58, are merely examples, and it goes without saying that other numbers of frames may be adopted as the determination criteria.

(User interface drawing (for classification) (step S18))
Next, referring to the flowchart of FIG. 7, the procedure for drawing the user interface screen in step S18 of FIG. I will explain the details. In other words, the flowchart of FIG. 7 describes the details of the operation of step S18 when the frozen state is determined in step S17 of the flowchart of FIG.

In step S17 in FIG. 1, when the image state detection unit 212 determines that the image is in a frozen state (step S71 in FIG. 7), the process proceeds to step S72. Note that when it is determined that the state is not frozen, the user interface screen for detecting the specific part is drawn (the operation already described in FIG. 6).

In step S72, it is determined whether or not the specific part has been detected in the screen. If it is not detected (NO), the process proceeds to step S77, and the drawn image is displayed on the display 300 as a user interface screen.

On the other hand, if it is determined in step S72 that the specific site has been detected in the screen (YES), then in step S73 the bounding box BX(1) associated with the detected specific site and the specific site detected in the past frame are displayed. It is determined whether or not the overlapping rate Iou with the bounding box BX(0) related to the part is 0.3 or more. If the overlap rate is 0.3 or more, the confidence factor is accumulated according to the label (lesion type).

When the confidence factor is integrated in this way and the integrated value reaches 600% or more (YES in step S74), the bounding box BX, the label (lesion type), and the confidence factor are displayed on the display 300. (Step S75). On the other hand, when the integrated value is less than 600%, the message "classifying" is displayed on the display 300, and the same procedure is continued.

(Bounding box estimation (step S20))
Next, the details of the bounding box estimation (step S20) execution procedure will be described with reference to the flowchart of FIG. In this step S20, as described with reference to FIG. 3, according to the result of the freeze state determination, only the detection of the specific site and the display of the bounding box BX are performed, and in addition to the display of the binding box BX, the organization of the specific site is displayed. A state in which diagnostic prediction is also performed is toggled.

Specifically, first, in step S81, it is determined whether or not the freeze state is present according to the freeze determination result in step S17 (FIG. 3). If it is determined that the state is frozen (YES in step S81), a tissue diagnosis prediction model is applied, and tissue diagnosis prediction is executed for the specific site detected according to this model (step S82). On the other hand, if it is determined that the state is not frozen (NO in step S81), a model for detection of specific parts is applied, and only detection of specific parts is executed according to this model (step S83).

In step S84, the certainty factor is calculated according to the model described above for the detected specific part, and it is determined whether or not the certainty factor is equal to or less than a threshold. If it is equal to or less than the threshold, the bounding box BX for that specific part is deleted. Then, in subsequent step S85, the bounding boxes BX whose overlap rate Iou is less than the threshold in relation to y or more bounding boxes BX detected in the past x frames are deleted. This suppresses the display of the bounding box BX for a portion that is unlikely to be the specific site, thereby facilitating diagnosis. Then, when the tissue diagnosis prediction model is applied, the bounding box BX, label, and confidence are displayed on the display 300, and when the detection model is applied, only the bounding box BX is displayed, This step S20 ends.

(effect)
As described above, according to the diagnosis support system of the first embodiment, the image state detection unit 212 detects whether or not the image being captured is in a frozen state while the detection of the specific part is continued. , and the tissue diagnosis prediction unit 211 operates appropriately according to the result. Therefore, lesions can be classified at appropriate timing according to endoscope skills while preventing lesions from being overlooked.

[Second embodiment]
FIG. 9 is a schematic diagram for explaining the hardware configuration of the diagnostic support system 1 of the second embodiment. In FIG. 9, the same reference numerals are given to the same components as in FIG. 1, so redundant description will be omitted below. The diagnosis support system 1 of the second embodiment includes a freeze button detector 212A instead of (or in addition to) the image state detector 212 of the first embodiment (FIG. 1). there is In the first embodiment, the image state detection unit 212 determines whether or not the image is in a frozen state by comparing the captured images themselves. In the second embodiment, instead of (or in addition to) this, the operation signal of the freeze button 102Bf itself of the endoscope 100 is detected by the freeze button detection unit 212A, and according to the result, the tissue diagnosis prediction unit At 211, a tissue diagnosis algorithm is initiated that performs a tissue diagnosis prediction. This form can also provide the same effects as the first embodiment.

[Modification]
In the embodiment described above, an example in which the diagnosis support system 1 includes the endoscope 100, the processor 200, the display 300, and the light source unit 400 has been described. These are only examples, and various additions, changes, alterations, etc. are possible without departing from the spirit of the invention. (A) to (D) of FIG. 10 each show an example of a modification. 10A to 10D, illustration of the light source unit 400 is omitted for simplification.

FIG. 10A shows an example in which processor 200 is provided as a stand-alone server and connected to display 300 . Software according to the present embodiment is installed in processor 200, which is a stand-alone server.

In FIG. 10B, the processor 200 is configured as a medical office server connected to the cloud, the medical office server is further connected to an external server via the cloud, and the software according to this embodiment is supplied from the external server. . The medical office server (200) is communicably connected to a bedside PC 500A directly connected to the endoscope 100 via an in-hospital network (LAN or the like). The medical office server 200 and the software may be provided by the same company or may be provided by separate companies.

FIG. 10(C) shows an endoscope system provided by Company A, which includes an endoscope 100, a bedside PC 500B, and a server 200, and is provided by Company B, which is different from Company A, in the present embodiment. A case is shown in which the produced software is installed using the open platform of the endoscope system.

FIG. 10(D) shows an example of a system including a bedside PC 500C (including a display 300) and a stand-alone server 200 as a support diagnosis system that controls the endoscope 100. FIG. The software of this embodiment can be installed on the server 200 . Bedside PC 500C and server 200 may be provided by the same company, or may be jointly developed by a plurality of companies.

[others]
The present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. In addition, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.
It is possible to make substitutions.

DESCRIPTION OF SYMBOLS 1... Diagnosis support system 100... Endoscope 101... Insertion part 102... Hand operation part 102A... Bending operation knob 102B... Operation button 102Bf... Freeze button 103... Bending part 104... Tip part 105 ... Universal cable 106 ... Connector section 200 ... Processor 201 ... Image processing section 201 ... Image processing section 202 ... Interface 203 ... Hard disk drive 204 ... ROM 205 ... RAM 206 ... Input section 207 ... Interface , 208...CPU 209...Display control unit 210...Specific site detection unit 211...Tissue diagnosis prediction unit 212...Image state detection unit 213...Learning data generation unit 300...Display 301...User interface 302 ... observation image display section, 400 ... light source section, BX ... bounding box.

Claims (7)

a display unit that displays an image obtained by the imaging device;
a specific site detection unit that detects a specific site having characteristics of a lesion or a precursor lesion in the image;
a display control unit for displaying, in the display unit, a figure surrounding the specific part identified by the specific part detection unit;
a tissue diagnosis prediction unit that predicts a tissue diagnosis according to a predetermined criterion with respect to the specific site;
When a condition that the image is in a static state is satisfied while the detection algorithm for detecting the specific site is executed by the specific site detection unit, tissue diagnosis prediction is performed by the tissue diagnosis prediction unit. A diagnostic support system, comprising: a controller that initiates a diagnostic algorithm; further comprising an image state detection unit that determines the state of the image;
2. The diagnosis support system according to claim 1, wherein said control unit executes said detection algorithm and said tissue diagnosis algorithm in parallel according to the determination result of said image state detection unit. 3. The diagnostic support system according to claim 2, wherein said image state detector is configured to determine whether said image is in a static state. The image state detection unit determines whether or not the image is in a static state based on a comparison between a first frame constituting the image and a second frame that temporally precedes the first frame. 4. The diagnostic support system according to claim 3, which determines. The tissue diagnosis prediction unit
2. The diagnosis support system according to claim 1, configured to calculate a degree of certainty indicating the possibility that a lesion is present in the specific site based on an overlapping rate of the graphics between a plurality of frames forming the image. . The method further comprises a learning data generation unit that generates learning data based on at least one of a detection result of the specific region by the specific region detection unit and a tissue diagnosis prediction result by the tissue diagnosis prediction unit. 2. The diagnosis support system according to 1. a step of displaying an image obtained by the imaging device on a display unit;
detecting in the image specific sites characteristic of lesions or precursor lesions ;
displaying, on the display unit, a figure surrounding the identified specific part;
a step of predicting tissue diagnosis according to predetermined criteria for the specific site;
causing a computer to execute a step of starting a tissue diagnosis algorithm for predicting a tissue diagnosis when a condition is satisfied that the image is in a static state while executing the detection algorithm for detecting the specific site. A diagnostic support program configured to enable

Images