JP6872581B2 - Information processing equipment, endoscope processors, information processing methods and programs - Google Patents

Description

The present invention relates to an information processing device, an endoscope processor, an information processing method and a program.

An image processing device that analyzes textures of endoscopic images and the like and classifies them according to pathological diagnosis has been proposed. By using such a diagnosis support technique, even a doctor who does not have highly specialized knowledge and experience can make a diagnosis promptly.

JP-A-2017-70609

However, the classification by the image processing apparatus of Patent Document 1 is a black box for the user. Therefore, the user may not always understand and be convinced of the reason for the output classification.

For example, in ulcerative colitis (UC), it is known that judgments may differ between specialists who have seen the same endoscopic image. In the case of such a disease, the doctor who is the user may not be satisfied with the judgment result by the diagnosis assisting technology.

In one aspect, it is an object of the present invention to provide an information processing device or the like that presents a region that contributes to the determination together with the determination result regarding the diagnosis of the disease.

The information processing apparatus includes an image acquisition unit that acquires an endoscopic image, the plurality of first models to output a plurality of items of the diagnostic criteria predictive diagnostic criteria for the disease when the endoscopic image is input, enter the endoscope image by the image acquisition unit has acquired each a first obtaining unit for obtaining diagnostic criteria predictive of the respective items output from each of the first model, from the plurality of items, selected items The reception unit that receives the image, the extraction unit that extracts the region that affects the diagnostic standard prediction regarding the selection item received by the reception unit from the endoscopic image, and the diagnostic standard prediction acquired by the first acquisition unit. And an index indicating the region extracted by the extraction unit, and an output unit that outputs the diagnostic prediction regarding the state of the disease acquired based on the endoscopic image in association with each other.

It is possible to provide an information processing device or the like that presents a region that contributes to the determination together with the determination result regarding the diagnosis of the disease.

It is explanatory drawing explaining the outline of the diagnosis support system. It is explanatory drawing explaining the structure of the diagnosis support system. It is explanatory drawing explaining the structure of the 1st score learning model. It is explanatory drawing explaining the structure of the 2nd model. It is a time chart which schematically explains the operation of the diagnosis support system. It is a flowchart explaining the process flow of a program. It is explanatory drawing explaining the outline of the diagnosis support system of the 1st modification. It is explanatory drawing explaining the screen display of the 2nd modification. It is explanatory drawing explaining the screen display of the 3rd modification. It is a time chart schematically explaining the operation of the 4th modification. It is explanatory drawing explaining the outline of the process of generating a model. It is explanatory drawing explaining the structure of the model generation system. It is explanatory drawing explaining the record layout of a teacher data DB. It is explanatory drawing explaining the teacher data input screen. It is explanatory drawing explaining the teacher data input screen. It is a flowchart explaining the process flow of the program which generates a learning model. It is a flowchart explaining the process flow of the program which updates a learning model. It is a flowchart explaining the process flow of the program which collects teacher data. It is explanatory drawing explaining the outline of the diagnosis support system of Embodiment 3. It is explanatory drawing explaining the feature quantity acquired from the 2nd model. It is explanatory drawing explaining the conversion between a feature amount and a score. It is explanatory drawing explaining the record layout of the feature amount DB. It is a flowchart explaining the process flow of the program which creates a converter. It is a flowchart explaining the process flow of the program at the time of endoscopy of Embodiment 3. It is explanatory drawing explaining the outline of the diagnosis support system of Embodiment 4. It is explanatory drawing explaining the conversion of the endoscopic image and the score of Embodiment 4. It is a flowchart explaining the process flow of the program which creates the converter of Embodiment 4. It is a flowchart explaining the process flow of the program at the time of endoscopy of Embodiment 4. It is explanatory drawing explaining the outline of the diagnosis support system of Embodiment 5. It is explanatory drawing explaining the structure of the 1st score learning model of Embodiment 6. It is explanatory drawing explaining the screen display of Embodiment 6. It is explanatory drawing explaining the screen display of Embodiment 7. It is explanatory drawing explaining the outline of the diagnosis support system of Embodiment 8. It is explanatory drawing explaining the outline of the diagnosis support system of Embodiment 9. It is explanatory drawing explaining the structure of 1st model. It is explanatory drawing explaining the structure of the extraction part. It is a flowchart explaining the process flow of the program of Embodiment 9. It is a flowchart explaining the process flow of the subroutine of attention area extraction. It is explanatory drawing explaining the screen display of the 1st modification of Embodiment 9. It is explanatory drawing explaining the screen display of the 2nd modification of Embodiment 9. It is explanatory drawing explaining the screen display of the 3rd modification of Embodiment 9. It is a flowchart explaining the process flow of the subroutine of attention area extraction of Embodiment 10. It is a functional block diagram of the information processing apparatus of Embodiment 11. It is explanatory drawing explaining the structure of the diagnosis support system of Embodiment 12.

[Embodiment 1]
In the present embodiment, a diagnostic support system 10 that supports the diagnosis of ulcerative colitis will be described as an example. Ulcerative colitis is one of the inflammatory bowel diseases in which the mucous membrane of the large intestine is inflamed. It is known that the affected area develops from the rectum to the entire circumference of the large intestine and progresses toward the oral side.

Since the active period in which the symptoms appear strongly and the remission period in which the symptoms subside may be repeated, and the risk of developing colorectal cancer increases if inflammation continues, the large intestine is regularly developed after the onset. Follow-up by colonoscopy is recommended.

The doctor inserts the tip of a colonoscope into, for example, the cecum, and then observes the endoscopic image while removing it. In the affected area, that is, the inflamed area, the inflammation appears to spread over the entire endoscopic image.

Public institutions such as the WHO (World Hearth Organization), medical societies, and medical institutions have established diagnostic criteria to be used when diagnosing various diseases. For example, in ulcerative colitis, a plurality of items such as the degree of redness of the affected area, the degree of vascular see-through, which means the appearance of blood vessels, and the degree of ulcer are listed as diagnostic criteria.

After examining each item of the diagnostic criteria, the doctor makes a comprehensive judgment and diagnoses the part under observation with the endoscope 14. Diagnosis includes determination of whether or not the site under observation is an affected area of ulcerative colitis, and if so, determination of severity such as severe or mild. A skilled doctor examines each item of the diagnostic criteria while removing the colonoscope, and makes a diagnosis of the position under observation in real time. The doctor will comprehensively diagnose the process of removing the colonoscope to determine the extent of the affected area that is inflamed by ulcerative colitis.

FIG. 1 is an explanatory diagram illustrating an outline of the diagnosis support system 10. The endoscope image 49 taken with the endoscope 14 (see FIG. 2) is input to the first model 61 and the second model 62. The second model 62 outputs a diagnostic prediction regarding the state of ulcerative colitis when the endoscopic image 49 is input. In the example shown in FIG. 1, a diagnostic prediction is output that the probability of not being normal, that is, the affected part of ulcerative colitis is 70%, and the probability of having mild ulcerative colitis is 20%. The details of the second model 62 will be described later.

The first model 61 includes a first score learning model 611, a second score learning model 612, and a third score learning model 613. In the following description, when it is not necessary to particularly distinguish the first score learning model 611 to the third score learning model 613, it may be simply described as the first model 61.

The first score learning model 611 outputs a predicted value of the first score that quantifies the evaluation regarding the degree of redness when the endoscopic image 49 is input. The second score learning model 612 outputs a predicted value of the second score, which is a numerical value of the evaluation regarding the degree of blood vessel see-through when the endoscopic image 49 is input. The third score learning model 613 outputs a predicted value of the third score, which is a numerical value of the evaluation regarding the degree of ulcer when the endoscopic image 49 is input.

The degree of redness, the degree of vascular see-through, and the degree of ulcer are examples of diagnostic criteria included in the diagnostic criteria used by physicians when diagnosing the condition of ulcerative colitis. The predicted values of the first score to the third score are examples of the diagnostic criteria prediction regarding the diagnostic criteria of ulcerative colitis.

In the example shown in FIG. 1, the predicted values that the first score is 10, the second score is 50, and the third score is 5 are output. The first model 61 is a score learning model that outputs a predicted value of a score that quantifies evaluations of various diagnostic criteria items related to ulcerative colitis, such as the degree of easy bleeding and the degree of secretion adhesion. May include. Details of the first model 61 will be described later.

The outputs of the first model 61 and the second model 62 are acquired by the first acquisition unit and the second acquisition unit, respectively. Based on the outputs acquired by the first acquisition unit and the second acquisition unit, the screen shown at the bottom of FIG. 1 is displayed on the display device 16 (see FIG. 2). The displayed screen includes an endoscopic image column 73, a first result column 71, a first stop button 711, a second result column 72, and a second stop button 722.

In the endoscope image column 73, an endoscope image 49 taken by using the endoscope 14 is displayed in real time. In the first result column 71, the diagnostic criteria predictions output from the first model 61 are listed. The diagnostic prediction output from the second model 62 is displayed in the second result column 72.

The first stop button 711 is an example of a first reception unit that receives an operation stop instruction of the first model 61. That is, when the first stop button 711 is selected, the output of the predicted value of the score using the first model 61 is stopped. The second stop button 722 is an example of a second reception unit that receives an operation stop instruction of the second model 62. That is, when the second stop button 722 is selected, the output of the predicted value of the score using the second model 62 is stopped.

By referring to the diagnostic criteria prediction displayed in the first result column 71, the doctor confirms the rationale for whether or not the diagnostic criterion displayed in the second result column 72 is appropriate in light of the diagnostic criteria. It is determined whether or not to adopt the diagnostic prediction displayed in the first result column 71.

FIG. 2 is an explanatory diagram illustrating the configuration of the diagnosis support system 10. The diagnostic support system 10 includes an endoscope 14, an endoscope processor 11, and an information processing device 20. The information processing device 20 includes a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, a display device I / F (Interface) 26, an input device I / F 27, and a bus.

The endoscope 14 includes a long insertion portion 142 having an image pickup device 141 at the tip portion thereof. The endoscope 14 is connected to the endoscope processor 11 via the endoscope connector 15. The endoscope processor 11 receives an image signal from the image sensor 141 and performs various image processing to generate an endoscope image 49 suitable for observation by a doctor. That is, the endoscope processor 11 functions as an image generation unit that generates an endoscope image 49 based on a video signal acquired from the endoscope 14.

The control unit 21 is an arithmetic control device that executes the program of the present embodiment. One or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), multi-core CPUs, and the like are used for the control unit 21. The control unit 21 is connected to each hardware unit constituting the information processing device 20 via a bus.

The main storage device 22 is a storage device such as a SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), and a flash memory. The main storage device 22 temporarily stores information necessary in the middle of processing performed by the control unit 21 and a program being executed by the control unit 21.

The auxiliary storage device 23 is a storage device such as a SRAM, a flash memory, or a hard disk. The auxiliary storage device 23 stores the first model 61, the second model 62, the program to be executed by the control unit 21, and various data necessary for executing the program. As described above, the first model 61 includes the first score learning model 611, the second score learning model 612, and the third score learning model 613. The first model 61 and the second model 62 may be stored in an external large-capacity storage device connected to the information processing device 20.

The communication unit 24 is an interface for performing data communication between the information processing device 20 and the network. The display device I / F 26 is an interface that connects the information processing device 20 and the display device 16. The display device 16 is an example of an output unit that outputs the diagnostic reference prediction acquired from the first model 61 and the diagnostic prediction acquired from the second model 62.

The input device I / F 27 is an interface for connecting the information processing device 20 and an input device such as a keyboard 17. The information processing device 20 is an information device such as a general-purpose personal computer, tablet, or smartphone.

FIG. 3 is an explanatory diagram illustrating the configuration of the first score learning model 611. The first score learning model 611 outputs the predicted value of the first score when the endoscopic image 49 is input.

The first score is a numerical value of the degree of redness determined based on the diagnostic criteria for ulcerative colitis when a skilled doctor looks at the endoscopic image 49. For example, the doctor determines the score out of 100 points, such as 0 points for "deemed red" and 100 points for "severe redness".

In addition, the doctor judges in 4 stages such as "deemed red", "mild", "moderate", and "severe", and "deemed red" is 0 points, "mild" is 1 point, and "moderate" is given. 2 points and "severe" may be scored as 3 points. The score may be set so that "severe" is a smaller number.

The first score learning model 611 of the present embodiment is a learning model generated by machine learning using, for example, a CNN (Convolutional Neural Network). The first score learning model 611 is composed of an input layer 531, an intermediate layer 532, an output layer 533, and a neural network model 53 having a convolution layer and a pooling layer (not shown). The method of generating the first score learning model 611 will be described later.

The endoscopic image 49 is input to the first score learning model 611. The input image is input to the fully connected layer after being repeatedly processed by the convolution layer and the pooling layer. The predicted value of the first score is output to the output layer 533.

Similarly, the second score is a numerical value for determining the degree of vascular see-through based on the diagnostic criteria for ulcerative colitis when a skilled specialist looks at the endoscopic image 49. The third score is a numerical value for determining the degree of ulcer based on the diagnostic criteria for ulcerative colitis when a skilled specialist looks at the endoscopic image 49. Since the configurations of the second score learning model 612 and the third score learning model 613 are the same as those of the first score learning model 611, illustration and description are omitted.

FIG. 4 is an explanatory diagram illustrating the configuration of the second model 62. The second model 62 outputs a diagnostic prediction of ulcerative colitis when the endoscopic image 49 is input. The diagnostic prediction is a prediction of how a skilled specialist would diagnose ulcerative colitis when looking at the endoscopic image 49.

The second model 62 of the present embodiment is a learning model generated by machine learning using, for example, CNN. The second model 62 is composed of an input layer 531, an intermediate layer 532, an output layer 533, and a neural network model 53 having a convolution layer and a pooling layer (not shown). The method of generating the second model 62 will be described later.

The endoscopic image 49 is input to the second model 62. The input image is input to the fully connected layer after being repeatedly processed by the convolution layer and the pooling layer. The diagnostic prediction is output to the output layer 533.

In FIG. 4, the output layer 533 is judged by a skilled specialist when the endoscopic image 49 is viewed, with a probability of severe ulcerative colitis, a probability of moderate ulcerative colitis, and a mild ulcerative colitis. It has four output nodes that each output the probability of having ulcerative colitis and the probability of being normal, i.e. not affected by ulcerative colitis.

FIG. 5 is a time chart schematically explaining the operation of the diagnosis support system 10. FIG. 5A shows the timing of shooting by the image sensor 141. FIG. 5B shows the timing of generating the endoscope image 49 by the image processing in the endoscope processor 11. FIG. 5C shows the timing at which the first model 61 and the second model 62 output the prediction based on the endoscopic image 49. FIG. 5D shows the timing of display on the display device 16. The horizontal axis from FIG. 5A to FIG. 5D all indicates time.

At time t ₀ , the frame of “a” is photographed by the image sensor 141. The video signal is transmitted to the endoscope processor 11. The endoscope processor 11 performs image processing to generate an endoscope image 49 of "a" at time t _1. The control unit 21 acquires the endoscope image 49 generated by the endoscope processor 11 and inputs it to the first model 61 and the second model 62. At time t ₂ , the control unit 21 acquires the predictions output from the first model 61 and the second model 62, respectively.

At time t _3, the control unit 21 outputs the endoscopic image 49 and display the predictive device 16 of the frame "a". This completes the processing of the image for one frame taken by the image sensor 141. Similarly, frame "b" is captured by the imaging element 141 at time t _6. Endoscopic image 49 of "b" is generated at time t _7. The control unit 21 obtains the predicted time t _8, and outputs an endoscope image 49 and predictive frames "b" on the display device 16 at time t _9. Since the operation after the frame of "c" is the same, the description thereof will be omitted. As described above, the endoscopic image 49 and the predictions made by the first model 61 and the second model 62 are displayed in synchronization with each other.

FIG. 6 is a flowchart illustrating a process flow of the program. The program described with reference to FIG. 6 is executed every time the control unit 21 acquires an endoscope image 49 of one frame from the endoscope processor 11.

The control unit 21 acquires the endoscope image 49 from the endoscope processor 11 (step S501). The control unit 21 inputs the acquired endoscopic image 49 into the second model 62 and acquires a diagnostic prediction output from the output layer 533 (step S502). The control unit 21 inputs the acquired endoscopic image 49 into one of the score learning models constituting the first model 61, and acquires a predicted value of the score output from the output layer 533 (step S503). ..

The control unit 21 determines whether or not the processing of the score learning model constituting the first model 61 is completed (step S504). If it is determined that the process has not been completed (NO in step S504), the control unit 21 returns to step S503.

When it is determined that the process is completed (YES in step S504), the control unit 21 generates an image described using the lower part of FIG. 1 and outputs it to the display device 16 (step S505). The control unit 21 ends the process.

According to the present embodiment, it is possible to provide the diagnostic support system 10 that displays the diagnostic reference prediction output from the first model 61 and the diagnostic prediction output from the second model 62 together with the endoscopic image 49. While observing the endoscopic image 49, the doctor can confirm the diagnostic prediction that predicts the diagnosis when a skilled specialist looks at the same endoscopic image 49, and the diagnostic criterion prediction.

By referring to the diagnostic criteria prediction displayed in the first result column 71, the doctor confirms the rationale for whether or not the diagnostic criterion displayed in the second result column 72 is appropriate in light of the diagnostic criteria. It is possible to determine whether or not to adopt the diagnostic prediction displayed in the first result column 71.

In the second result column 72, only the item having the highest probability and the probability may be displayed. The character size can be increased by reducing the number of characters to be displayed. The doctor can detect the change in the display of the second result column 72 while gazing at the endoscopic image column 73.

The doctor can stop predicting and displaying the score by selecting the first stop button 711. The doctor can stop the diagnosis prediction and the display of the diagnosis prediction by selecting the second stop button 722. The doctor can resume the display of the diagnostic prediction and the diagnostic criterion prediction by reselecting the first stop button 711 or the second stop button 722.

The first stop button 711 and the second stop button 722 can be operated from any input device such as a keyboard 17, a mouse, a touch panel, or voice input. The first stop button 711 and the second stop button 722 may be operable by using a control button or the like provided on the operation unit of the endoscope 14.

For example, when performing endoscopic treatment such as resection of a polyp or EMR (Endoscopic Mucosal Resection), the time lag from imaging by the image sensor 141 to display on the display device 16 is It is desirable to be as short as possible. The doctor can shorten the time lag by selecting the first stop button 711 and the second stop button 722 to stop the diagnostic prediction and the diagnostic criterion prediction.

The diagnostic reference prediction using each score learning model constituting the first model 61 and the diagnostic prediction using the second model 62 may be performed by parallel processing. By using parallel processing, the real-time property of display on the display device 16 can be improved.

According to the present embodiment, it is possible to provide an information processing device 20 or the like that presents a determination reason together with a determination result regarding a predetermined disease such as ulcerative colitis. By looking at both the diagnostic probability of the disease output by the second model 62 and the score related to the diagnostic criteria output by the first model 61, the doctor can output the correct result based on the diagnostic criteria. Can be confirmed.

If there is a discrepancy between the output of the second model 62 and the output of the first model 61, the doctor suspects a disease other than ulcerative colitis, consults with an instructor, or adds necessary tests. Etc. can be dealt with. From the above, it is possible to avoid oversight of rare diseases.

The diagnostic criterion prediction using the first model 61 and the diagnostic prediction using the second model 62 may be executed by different hardware.

The endoscopic image 49 may be an image recorded in an electronic medical record system or the like. For example, by inputting each image taken at the time of follow-up into the first model 61, it is possible to provide a diagnostic support system 10 capable of comparing temporal changes of each score.

[First modification]
FIG. 7 is an explanatory diagram illustrating an outline of the diagnosis support system 10 of the first modification. The description will be omitted except for the differences from FIG. The display device 16 includes a first display device 161 and a second display device 162. The first display device 161 is connected to the display device I / F26. The second display device 162 is connected to the endoscope processor 11. It is desirable that the first display device 161 and the second display device 162 are arranged adjacent to each other.

The endoscope image 49 generated by the endoscope processor 11 is displayed in real time on the first display device 161. The second display device 162 displays the diagnostic prediction and the diagnostic standard prediction acquired by the control unit 21.

According to this modification, it is possible to provide a diagnostic support system 10 that displays diagnostic predictions and diagnostic reference predictions while reducing the time lag in displaying the endoscopic image 49.

The diagnosis support system 10 may have three or more display devices 16. For example, the endoscopic image 49, the first result column 71, and the second result column 72 may be displayed on different display devices 16.

[Second modification]
FIG. 8 is an explanatory diagram illustrating a screen display of the second modification. The description will be omitted except for the differences from the lower part of FIG. In this modification, the CPU 21 outputs the first result column 71 and the second result column 72 in a graph format.

In the first result column 71, three diagnostic criterion predictions are displayed in a three-axis graph format. In FIG. 8, the upward axis indicates the predicted value of the first score, that is, the score regarding redness. The downward-right axis indicates the second score, that is, the predicted value of the score for vascular fluoroscopy. The downward left axis shows the predicted value of the third score, that is, the score for ulcer.

The predicted values of the first score, the second score, and the third score are displayed by the inner triangle. In the second result column 72, the diagnostic prediction output from the second model 62 is displayed by a bar graph. According to this variant, the doctor can intuitively grasp the diagnostic criteria prediction by looking at the triangle and bar graphs.

[Third variant]
FIG. 9 is an explanatory diagram illustrating a screen display of the third modification. FIG. 9 is a screen displayed by the diagnosis support system 10 that supports the diagnosis of Crohn's disease. Crohn's disease, like ulcerative colitis, is a type of inflammatory bowel disease. In FIG. 9, the first score is the degree of longitudinal ulcer extending in the length direction of the intestinal tract, the second score is the degree of paving stones which are dense mucosal ridges, and the third score is the degree of aphthae of red spots. Shown.

The diseases for which the diagnosis support system 10 supports the diagnosis are not limited to ulcerative colitis and Crohn's disease. It is possible to provide a diagnostic support system 10 that supports the diagnosis of any disease capable of creating appropriate first model 61 and second model 62. The user may be able to switch which disease to assist in diagnosing during endoscopy. Information that supports the diagnosis of each disease may be displayed on the plurality of display devices 16.

[Fourth variant]
FIG. 10 is a time chart schematically explaining the operation of the fourth modification. The description of the parts common to FIG. 5 will be omitted. FIG. 10 shows an example of a time chart when the processing using the first model 61 and the second model 62 takes a long time.

At time t ₀ , the frame of “a” is photographed by the image sensor 141. The endoscope processor 11 performs image processing to generate an endoscope image 49 of "a" at time t _1. The control unit 21 acquires the endoscope image 49 generated by the endoscope processor 11 and inputs it to the first model 61 and the second model 62. At time t ₂ , the control unit 21 outputs the endoscopic image 49 of “a” to the display device 16.

At time t ₆ , the image sensor 141 takes a picture of the frame “b”. The endoscope processor 11 performs image processing, the time t ₇ to generate an endoscope image 49 of "b". The endoscopic image 49 of "b" is not input to the first model 61 and the second model 62. At time t ₈ , the control unit 21 outputs the endoscopic image 49 of “b” to the display device 16.

At time t ₉ , the control unit 21 acquires a prediction based on the endoscopic image 49 of “a” output from the first model 61 and the second model 62, respectively. At time t _10, the control unit 21 outputs to the display device 16 the prediction based on the endoscopic image 49 of "a". At time t _12, the frame of the "c" is captured by the imaging element 141. Since the subsequent processing is the same as from time t ₀ to time t ₁₀ , the description thereof will be omitted. As described above, the endoscopic image 49 and the predictions made by the first model 61 and the second model 62 are displayed in synchronization with each other.

According to this modification, even if it takes time to process using the first model 61 and the second model 62 by thinning out the endoscopic images 49 input to the first model 61 and the second model 62. , Real-time display can be realized.

[Embodiment 2]
The present embodiment relates to a model generation system 19 that generates the first model 61 and the second model 62. The description of the parts common to the first embodiment will be omitted.

FIG. 11 is an explanatory diagram illustrating an outline of a process for generating a model. In the teacher data DB 64 (see FIG. 12), a plurality of sets of teacher data are recorded in which the endoscopic image 49 is associated with the judgment result by an expert such as a skilled specialist. The judgment results by experts are the diagnosis of ulcerative colitis based on the endoscopic image 49, the first score, the second score, and the third score.

The second model 62 is generated by machine learning using the pair of the endoscopic image 49 and the diagnosis result as teacher data. The first score learning model 611 is generated by machine learning using the pair of the endoscopic image 49 and the first score as teacher data. The second score learning model 612 is generated by machine learning using the pair of the endoscopic image 49 and the second score as teacher data. The third score learning model 613 is generated by machine learning using the pair of the endoscopic image 49 and the third score as teacher data.

FIG. 12 is an explanatory diagram illustrating the configuration of the model generation system 19. The model generation system 19 includes a server 30 and a client 40. The server 30 includes a control unit 31, a main storage device 32, an auxiliary storage device 33, a communication unit 34, and a bus. The client 40 includes a control unit 41, a main storage device 42, an auxiliary storage device 43, a communication unit 44, a display unit 46, an input unit 47, and a bus.

The control unit 31 is an arithmetic control device that executes the program of the present embodiment. One or more CPUs, a multi-core CPU, a GPU, or the like is used for the control unit 31. The control unit 31 is connected to each hardware unit constituting the server 30 via a bus.

The main storage device 32 is a storage device for SRAM, DRAM, flash memory, and the like. The main storage device 32 temporarily stores information necessary during the processing performed by the control unit 31 and the program being executed by the control unit 31.

The auxiliary storage device 33 is a storage device such as a SRAM, a flash memory, a hard disk, or a magnetic tape. The auxiliary storage device 33 stores a program to be executed by the control unit 31, a teacher data DB 64, and various data necessary for executing the program. Further, the first model 61 and the second model 62 generated by the control unit 31 are also stored in the auxiliary storage device 33. The teacher data DB 64, the first model 61, and the second model 62 may be stored in an external large-capacity storage device or the like connected to the server 30.

The server 30 is a general-purpose personal computer, a tablet, a large computer, a virtual machine running on the large computer, a cloud computing system, or a quantum computer. The server 30 may be a plurality of personal computers or the like that perform distributed processing.

The control unit 41 is an arithmetic control device that executes the program of the present embodiment. The control unit 41 is an arithmetic control device that executes the program of the present embodiment. For the control unit 41, one or more CPUs, a multi-core CPU, a GPU, or the like is used. The control unit 41 is connected to each hardware unit constituting the client 40 via a bus.

The main storage device 42 is a storage device for SRAM, DRAM, flash memory, and the like. The main storage device 42 temporarily stores information necessary during the processing performed by the control unit 41 and the program being executed by the control unit 41.

The auxiliary storage device 43 is a storage device such as a SRAM, a flash memory, or a hard disk. The auxiliary storage device 43 stores a program to be executed by the control unit 41 and various data necessary for executing the program.

The communication unit 44 is an interface for performing data communication between the client 40 and the network. The display unit 46 is, for example, a liquid crystal display panel, an organic EL (Electro Luminescence) display panel, or the like. The input unit 47 is, for example, a keyboard 17 and a mouse. The client 40 may have a touch panel in which the display unit 46 and the input unit 47 are laminated.

The client 40 is an information device such as a general-purpose personal computer, tablet, or smartphone used by a specialist or the like who creates teacher data. The client 40 may be a so-called thin client that realizes a user interface based on control by the control unit 31. When a thin client is used, most of the processing described later by the client 40 is executed by the control unit 31 instead of the control unit 41.

FIG. 13 is an explanatory diagram illustrating a record layout of the teacher data DB 64. The teacher data DB 64 is a DB for recording teacher data used for generating the first model 61 and the second model 62. The teacher data DB 64 has a site field, a disease field, an endoscopic image field, an endoscopic finding field, and a score field. The score field has a redness field, a vascular fluoroscopy field and an ulcer field.

In the site field, the site where the endoscopic image 49 was taken is recorded. In the disease field, the name of the disease judged by a specialist or the like when creating the teacher data is recorded. An endoscopic image 49 is recorded in the endoscopic image field. In the endoscopic findings field, the state of the disease judged by a specialist or the like by looking at the endoscopic image 49, that is, the endoscopic findings is recorded.

In the redness field, the first score regarding redness, which is judged by a specialist or the like by looking at the endoscopic image 49, is recorded. In the blood vessel see-through field, a second score regarding blood vessel see-through, which is judged by a specialist or the like by looking at the endoscopic image 49, is recorded. In the ulcer field, a third score regarding blood vessels, which is judged by a specialist or the like by looking at the endoscopic image 49, is recorded. The teacher data DB 64 has one record for one endoscopic image 49.

14 and 15 are explanatory views for explaining the teacher data input screen. FIG. 14 shows an example of a screen displayed on the display unit 46 by the control unit 41 when the teacher data is created without using the existing first model 61 and the second model 62.

The screen shown in FIG. 14 includes an endoscopic image field 73, a first input field 81, a second input field 82, a next button 89, a patient ID field 86, a disease name field 87, and a model button 88. The first input field 81 includes a first score input field 811, a second score input field 812, and a third score input field 813. In FIG. 14, the model button 88 is set to the “model not used” state.

The endoscope image 49 is displayed in the endoscope image column 73. The endoscopic image 49 may be an image taken by an endoscopic examination performed by a specialist or the like who inputs teacher data, or may be an image delivered from the server 30. A specialist or the like makes a diagnosis regarding "ulcerative colitis" displayed in the disease name field 87 based on the endoscopic image 49, and selects the check box provided at the left end of the second input field 82.

The "inappropriate image" means that a specialist or the like has determined that it is inappropriate for use in diagnosis due to circumstances such as a large amount of residue or blurring. The endoscopic image 49 determined to be an “inappropriate image” is not recorded in the teacher data DB 64.

The specialist or the like determines the third score from the first score based on the endoscopic image 49, and inputs it into the first score input field 811 to the third score input field 813, respectively. After completing the input, the specialist or the like selects the next button 89. The control unit 41 transmits the endoscopic image 49, the input to the first input field 81, and the input to the second input field 82 to the server 30. The control unit 31 adds a new record to the teacher data DB 64 and records the endoscopic image 49, the endoscopic findings, and each score.

FIG. 15 shows an example of a screen displayed on the display unit 46 by the control unit 41 when the teacher data is created with reference to the existing first model 61 and the second model 62. In FIG. 15, the model button 88 is set to the “model in use” state. When the existing first model 61 and second model 62 are not generated, the model button 88 is set so that the "model in use" state is not selected.

The result of inputting the endoscopic image 49 into the first model 61 and the second model 62 is displayed in the first input field 81 and the second input field 82. In the second input field 82, the check box at the left end of the item with the highest probability is checked by default.

The specialist or the like determines whether or not each score in the first input field 81 is correct based on the endoscopic image 49, and changes the score as necessary. The specialist or the like determines whether or not the check in the second input field 82 is correct based on the endoscopic image 49, and reselects the check box if necessary. After making the first input field 81 and the second input field 82 in an appropriate state, the specialist or the like selects the next button 89. The subsequent processing is the same as the case of "model not used" described with reference to FIG. 14, and therefore the description thereof will be omitted.

FIG. 16 is a flowchart illustrating a processing flow of a program that generates a learning model. The program described with reference to FIG. 16 is used to generate each learning model constituting the first model 61 and the second model 62.

The control unit 31 selects the learning model to be created (step S522). The learning model to be created is any one of the learning models constituting the first model 61, or the second model 62. The control unit 31 extracts necessary fields from the teacher data DB 64 and creates teacher data composed of a pair of the endoscope image 49 and the output data (step S523).

For example, when generating the first score learning model 611, the output data is a score for redness. The control unit 31 extracts the endoscopic image field and the redness field from the teacher data DB 64. Similarly, when generating the second model 62, the output data is endoscopic findings. The control unit 31 extracts the endoscopic image field and the endoscopic finding field from the teacher data DB 64.

The control unit 31 separates the teacher data created in step S523 into training data and test data (step S524). The control unit 31 performs supervised machine learning by using the training data and adjusting the parameters of the intermediate layer 532 by using an error backpropagation method or the like, and generates a learning model (step S525).

The control unit 31 verifies the accuracy of the learning model using the training data (step S526). The verification is performed by calculating the probability that the output matches the output data corresponding to the endoscope image 49 when the endoscope image 49 in the training data is input to the learning model.

The control unit 31 determines whether or not the accuracy of the learning model generated in step S525 is acceptable (step S527). If it is determined to pass (YES in step S527), the control unit 31 records the learning model in the auxiliary storage device 33. (Step S528).

When it is determined that the result is not passed (NO in step S527), the control unit 31 determines whether or not to end the process (step S529). For example, when the processing from step S524 to step S529 is repeated a predetermined number of times, the control unit 31 determines that the processing is completed. If it is determined that the process is not completed (NO in step S529), the control unit 31 returns to step S524.

When it is determined that the process is to be completed (YES in step S529), or after the end of step S528, the control unit 31 determines whether or not to end the process (step S531). If it is determined that the process is not completed (NO in step S531), the control unit 31 returns to step S522. If it is determined that the process is to be completed (YES in step S531), the control unit 31 ends the process.

If the learning model determined to pass is not generated, the program described with reference to FIG. 16 after reviewing each record recorded in the teacher data DB 64 and adding the record. Is executed again.

The first model 61 and the second model 62 updated by the program described with reference to FIG. 16 are, for example, after the procedures such as approval under the Pharmaceuticals and Medical Devices Act are completed, via a network or as a recording medium. Is delivered to the information processing apparatus 20 via.

FIG. 17 is a flowchart illustrating a processing flow of a program that updates the learning model. The program described with reference to FIG. 17 is appropriately executed when additional records are recorded in the teacher data DB 64. The additional teacher data may be recorded in a database different from the teacher data DB 64.

The control unit 31 acquires the learning model to be updated (step S541). The control unit 31 acquires additional teacher data (step S542). Specifically, the control unit 31 obtains the endoscope image 49 recorded in the endoscope image field and the output data corresponding to the learning model acquired in step S541 from the record added to the teacher data DB 64. get.

The control unit 31 sets the endoscope image 49 as the input data of the learning model and the output data associated with the endoscope image 49 as the output of the learning model (step S543). The control unit 31 updates the parameters of the learning model by the back-propagation method (step S544). The control unit 31 records the updated parameters (step S545).

The control unit 31 determines whether or not the processing of the record added to the teacher data DB 64 is completed (step S546). If it is determined that the process has not been completed (NO in step S546), the control unit 31 returns to step S542. When it is determined that the process is completed (YES in step S546), the control unit 31 ends the process.

The first model 61 and the second model 62 updated by the program described with reference to FIG. 17 are, for example, via a network or a recording medium after the procedures such as approval under the Pharmaceuticals and Medical Devices Act are completed. Is delivered to the information processing apparatus 20 via. As a result, the first model 61 and the second model 62 are updated. The learning models constituting the first model 61 and the second model 62 may be updated at the same time or individually.

FIG. 18 is a flowchart illustrating a processing flow of a program for collecting teacher data. The control unit 41 acquires the endoscope image 49 from an electronic medical record system (not shown), a hard disk mounted on the endoscope processor 11, or the like (step S551). The control unit 41 determines whether or not it is selected to use the model via the model button 88 described with reference to FIG. 14 (step S552).

When it is determined that the intention to use the model is not selected (NO in step S552), the control unit 41 displays the screen described with reference to FIG. 14 on the display unit 46 (step S553). When it is determined that the intention to use the model is selected (YES in step S552), the control unit 41 acquires the first model 61 and the second model 62 from the server 30 (step S561).

The control unit 41 may temporarily store the acquired first model 61 and second model 62 in the auxiliary storage device 43. By doing so, the control unit 41 can omit the processing of the second and subsequent steps S561.

The control unit 41 inputs the endoscope image 49 acquired in step S551 into the first model 61 and the second model 62 acquired in step S561, respectively, and acquires the estimation result output from the output layer 533 (step). S562). The control unit 41 displays the screen described with reference to FIG. 15 on the display unit 46 (step S563).

After the end of step S553 or step S563, the control unit 41 acquires the input of the judgment result by the user via the input unit 47 (step S564). The control unit 41 determines whether or not the “inappropriate image” is selected in the second input field 82 (step S565). When it is determined that the "inappropriate image" is selected (YES in step S565), the control unit 41 ends the process.

When it is determined that the "inappropriate image" is not selected (NO in step S565), the control unit 41 transmits a teacher record associated with the endoscope image 49 and the input result by the user to the server 30 (step). S566). The teacher record may be recorded in the teacher data DB 64 via a portable recording medium such as a USB (Universal Serial Bus) memory.

The control unit 31 creates a new record in the teacher data DB 64 and records the received teacher record. It should be noted that, for example, when a plurality of experts make a judgment on the same endoscopic image 49 and the judgments by a predetermined number of experts match, the data may be recorded in the teacher data DB 64. By doing so, the accuracy of the teacher data DB 64 can be improved.

According to this embodiment, teacher data can be collected, and the first model 61 and the second model 62 can be generated and updated.

[Embodiment 3]
The present embodiment relates to a diagnostic support system 10 that outputs a score according to a diagnostic criterion based on a feature amount extracted from the intermediate layer 532 of the second model 62. The description of the parts common to the first embodiment or the second embodiment will be omitted.

FIG. 19 is an explanatory diagram illustrating an outline of the diagnosis support system 10 of the third embodiment. The endoscope image 49 taken by the endoscope 14 is input to the second model 62. The second model 62 outputs a diagnostic prediction of ulcerative colitis when the endoscopic image 49 is input. As will be described later, feature quantities 65 such as the first feature quantity 651, the second feature quantity 652, and the third feature quantity 653 are acquired from the nodes constituting the intermediate layer 532 of the second model 62.

The first model 61 includes a first converter 631, a second converter 632 and a third converter 633. The first feature amount 651 is converted by the first converter 631 into a predicted value of the first score indicating the degree of redness. The second feature amount 652 is converted by the second converter 632 into a predicted value of the second score indicating the degree of blood vessel see-through. The third feature amount 653 is converted by the third converter 633 into a predicted value of the third score indicating the degree of ulcer. In the following description, when the first converter 631 to the third converter 633 are not particularly distinguished, they are referred to as the converter 63.

The outputs of the first model 61 and the second model 62 are acquired by the first acquisition unit and the second acquisition unit, respectively. Based on the outputs acquired by the first acquisition unit and the second acquisition unit, the screen shown at the bottom of FIG. 19 is displayed on the display device 16. Since the displayed screen is the same as the screen described in the first embodiment, the description thereof will be omitted.

FIG. 20 is an explanatory diagram for explaining the feature amount acquired from the second model 62. Intermediate layer 532 includes a plurality of interconnected nodes. When the endoscopic image 49 is input to the second model 62, various features of the endoscopic image 49 appear in each node. As an example, each feature amount appearing in the five nodes is indicated by a symbol from the feature amount A65A to the feature amount E65E.

The feature amount may be acquired from the node immediately before being input to the fully connected layer after being repeatedly processed by the convolution layer and the pooling layer, or may be acquired from the node included in the fully connected layer.

FIG. 21 is an explanatory diagram illustrating the conversion between the feature amount and the score. The teacher data included in the teacher data DB 64 is schematically shown in the upper part of FIG. 21. In the teacher data DB 64, teacher data in which the endoscopic image 49 is associated with the judgment result by an expert such as a specialist is recorded. Since the record layout of the teacher data DB 64 is the same as that of the teacher data DB 64 of the first embodiment described with reference to FIG. 13, the description thereof will be omitted.

As described above, the endoscopic image 49 is input to the second model 62, and a plurality of feature quantities such as the feature quantity A65A are acquired. Correlation analysis is performed between the acquired feature amount and the first to third scores associated with the endoscopic image 49, and a feature amount having a high correlation with each score is selected. FIG. 21 shows a case where the correlation between the first score and the feature amount A65A, the correlation between the second score and the feature amount C65C, and the correlation between the third score and the feature amount D65D are high.

The first converter 631 is obtained by performing a regression analysis between the first score and the feature amount A65A. Similarly, the second converter 632 is obtained by the regression analysis of the second score and the feature amount C65C, and the third converter 633 is obtained by the regression analysis of the third score and the feature amount D65D. Linear regression or non-linear regression may be used for regression analysis. Regression analysis may be performed using a neural network.

FIG. 22 is an explanatory diagram illustrating a record layout of the feature amount DB. The feature amount DB is a DB that records the teacher data in association with the feature amount acquired from the endoscopic image 49. The feature amount DB has a site field, a disease field, an endoscopic image field, an endoscopic finding field, a score field, and a feature amount field. The score field has a redness field, a vascular fluoroscopy field and an ulcer field. The feature field has a large number of subfields such as an A field and a B field.

In the site field, the site where the endoscopic image 49 was taken is recorded. In the disease field, the name of the disease judged by a specialist or the like when creating the teacher data is recorded. An endoscopic image 49 is recorded in the endoscopic image field. In the endoscopic findings field, the state of the disease judged by a specialist or the like by looking at the endoscopic image 49, that is, the endoscopic findings is recorded.

In the redness field, the first score regarding redness, which is judged by a specialist or the like by looking at the endoscopic image 49, is recorded. In the blood vessel see-through field, a second score regarding blood vessel see-through, which is judged by a specialist or the like by looking at the endoscopic image 49, is recorded. In the ulcer field, a third score regarding ulcer, which is judged by a specialist or the like by looking at the endoscopic image 49, is recorded. In each subfield of the feature amount field, a feature amount such as the feature amount A64A acquired from each node of the intermediate layer 532 is recorded.

The feature amount DB has one record for one endoscopic image 49. The feature amount DB is stored in the auxiliary storage device 33. The feature amount DB may be stored in an external large-capacity storage device or the like connected to the server 30.

FIG. 23 is a flowchart illustrating a processing flow of a program for creating the converter 63. The control unit 31 selects one record from the teacher data DB 64 (step S571). The control unit 31 inputs the endoscope image 49 recorded in the endoscope image field into the second model 62, and acquires the feature amount from each node of the intermediate layer 532 (step S572). The control unit 31 creates a new record in the feature amount DB, and records the data recorded in the record acquired in step S571 and the feature amount acquired in step S572 (step S573).

The control unit 31 determines whether or not to end the process (step S574). For example, when the processing of a predetermined number of teacher data records is completed, the control unit 31 determines that the processing is completed. If it is determined that the process is not completed (NO in step S574), the control unit 31 returns to step S571.

When it is determined that the processing is completed (YES in step S574), the control unit 31 selects one subfield from the score field of the feature amount DB (step S575). The control unit 31 selects one subfield from the feature amount field of the feature amount DB (step S576).

The control unit 31 performs a correlation analysis between the score selected in step S575 and the feature amount selected in step S576, and calculates the correlation coefficient (step S577). The control unit 31 temporarily records the calculated correlation coefficient in the main storage device 32 or the auxiliary storage device 33 (step S578).

The control unit 31 determines whether or not to end the process (step S579). For example, the control unit 31 determines that the process is completed when the correlation analysis of all the combinations of the score and the feature amount is completed. The control unit 31 may determine that the process ends when the correlation coefficient calculated in step S577 is equal to or greater than a predetermined threshold value.

If it is determined that the process is not completed (NO in step S579), the control unit 31 returns to step S576. When it is determined that the processing is completed (YES in step S579), the control unit 31 selects the feature amount having the highest correlation with the score selected in step S575 (step S580).

The control unit 31 performs regression analysis using the score selected in step S575 as the objective variable and the feature amount selected in step S580 as the explanatory variable, and calculates a parameter for specifying the converter 63 that converts the feature amount into the score ( Step S581). For example, if the score selected in step S575 is the first score, the converter 63 identified in step S581 is the first converter 631, and the score selected in step S575 is the second score. The converter 63 identified in step S581 is the second converter 632. The control unit 31 stores the calculated converter 63 in the auxiliary storage device 33 (step S582).

The control unit 31 determines whether or not the processing of all the score fields of the feature amount DB has been completed (step S583). If it is determined that the process has not been completed (NO in step S583), the control unit 31 returns to step S575. When it is determined that the process is completed (YES in step S583), the control unit 31 ends the process. As a result, each converter 63 constituting the first model 61 is generated.

The first model 61 including the converter 63 created by the program described with reference to FIG. 23 is, for example, via a network or a recording medium after the procedures such as approval under the Pharmaceuticals and Medical Devices Act are completed. Is delivered to the information processing apparatus 20 via.

FIG. 24 is a flowchart illustrating a flow of processing of the program at the time of endoscopy according to the third embodiment. The program of FIG. 24 is executed by the control unit 21 instead of the program described with reference to FIG.

The control unit 21 acquires the endoscope image 49 from the endoscope processor 11 (step S501). The control unit 21 inputs the acquired endoscopic image 49 into the second model 62 and acquires a diagnostic prediction output from the output layer 533 (step S502).

The control unit 21 acquires a feature amount from a predetermined node included in the intermediate layer 532 of the second model 62 (step S601). The predetermined node is a node from which the feature amount selected in step S580 described with reference to FIG. 23 has been acquired. The control unit 21 converts the acquired feature amount by the converter 63 and calculates the score (step S602).

The control unit 21 determines whether or not all the scores have been calculated (step S603). If it is determined that the process has not been completed (NO in step S603), the control unit 21 returns to step S601. When it is determined that the process is completed (YES in step S603), the control unit 21 generates an image described using the lower part of FIG. 19 and outputs the image to the display device 16 (step S604). The control unit 21 ends the process.

According to this embodiment, since the learning model generated by deep learning is only the second model 62, the diagnosis support system 10 can be realized with a relatively small amount of calculation.

By acquiring the feature amount from the intermediate layer 532 of the second model 62, it is possible to obtain the feature amount having a high correlation with the score without being limited to the feature amount within the range that humans can usually think of. Therefore, each diagnostic criterion prediction can be calculated accurately based on the endoscopic image 49.

A part of the first score, the second score, and the third score may be calculated by the same method as in the first embodiment.

[Embodiment 4]
The present embodiment relates to an information processing system that calculates a diagnostic criterion prediction based on a method other than deep learning. The description of the parts common to the first embodiment or the second embodiment will be omitted.

FIG. 25 is an explanatory diagram illustrating an outline of the diagnosis support system 10 of the fourth embodiment. The endoscope image 49 taken by the endoscope 14 is input to the second model 62. The second model 62 outputs a diagnostic prediction of ulcerative colitis when the endoscopic image 49 is input.

The first model 61 includes a first converter 631, a second converter 632 and a third converter 633. The first converter 631 outputs a predicted value of the first score indicating the degree of redness when the endoscopic image 49 is input. The second transducer 632 outputs a predicted value of the second score indicating the degree of blood vessel see-through when the endoscopic image 49 is input. The third transducer 633 outputs a predicted value of a third score indicating the degree of ulcer when the endoscopic image 49 is input.

The outputs of the first model 61 and the second model 62 are acquired by the first acquisition unit and the second acquisition unit, respectively. Based on the outputs acquired by the first acquisition unit and the second acquisition unit, the screen shown at the bottom of FIG. 25 is displayed on the display device 16. Since the displayed screen is the same as the screen described in the first embodiment, the description thereof will be omitted.

FIG. 26 is an explanatory diagram illustrating the conversion between the endoscopic image 49 of the fourth embodiment and the score. In FIG. 26, the illustration of the second model 62 is omitted.

In this embodiment, various converters 63 such as a converter A63A and a converter B63B are used to output a feature amount when an endoscopic image 49 is input. For example, the converter A63A converts the endoscopic image 49 into the feature amount A65A.

The converter 63 converts the endoscopic image 49 into a feature amount, for example, based on the number or ratio of pixels satisfying a predetermined condition. The converter 63 may convert the endoscopic image 49 into a feature amount by classification using SVM (Support Vector Machine), random forest, or the like.

Correlation analysis is performed between the feature amount converted by the converter 63 and the first to third scores associated with the endoscopic image 49, and the feature amount having a high correlation with each score is obtained. Be selected. FIG. 26 shows a case where the correlation between the first score and the feature amount A65A, the correlation between the second score and the feature amount C65C, and the correlation between the third score and the feature amount D65D are high.

Regression analysis of the first score and the feature amount A65A is performed, and the first converter 631 is obtained by combining with the converter A63A. Similarly, a regression analysis of the second score and the feature amount C65C is performed, and the second converter 632 is obtained by combining with the converter C63C.

FIG. 27 is a flowchart illustrating a processing flow of a program for creating the converter 63 of the fourth embodiment. The control unit 31 selects one record from the teacher data DB 64 (step S611). The control unit 31 uses a plurality of converters 63 such as the converter A63A and the converter B63B to convert the endoscopic image 49 recorded in the endoscopic image field into a feature amount (step S612). The control unit 31 creates a new record in the feature amount DB, and records the data recorded in the record acquired in step S611 and the feature amount acquired in step S612 (step S613).

The control unit 31 determines whether or not to end the process (step S614). For example, when the processing of a predetermined number of teacher data records is completed, the control unit 31 determines that the processing is completed. When it is determined that the process is not completed (NO in step S614), the control unit 31 returns to step S611.

When it is determined that the processing is completed (YES in step S614), the control unit 31 selects one subfield from the score field of the feature amount DB (step S575). Since the processing from step S575 to step S581 is the same as the processing flow of the program described with reference to FIG. 23, the description thereof will be omitted.

The control unit 31 combines the result obtained by the regression analysis with the converter 63 that has converted the endoscopic image 49 into a feature amount in step S612 to calculate a new converter 63 (step S620). The control unit 31 stores the calculated converter 63 in the auxiliary storage device 33 (step S621).

The control unit 31 determines whether or not the processing of all the score fields of the feature amount DB has been completed (step S622). If it is determined that the process has not been completed (NO in step S622), the control unit 31 returns to step S575. When it is determined that the process is completed (YES in step S622), the control unit 31 ends the process. As a result, each converter 63 constituting the first model 61 is generated.

The first model 61 including the converter 63 created by the program described with reference to FIG. 27 is, for example, via a network or a recording medium after the procedures such as approval under the Pharmaceuticals and Medical Devices Act are completed. Is delivered to the information processing apparatus 20 via.

FIG. 28 is a flowchart illustrating a flow of processing of the program at the time of endoscopy according to the fourth embodiment. The program of FIG. 28 is executed by the control unit 21 instead of the program described with reference to FIG.

The control unit 21 acquires the endoscope image 49 from the endoscope processor 11 (step S501). The control unit 21 inputs the acquired endoscopic image 49 into the second model 62 and acquires a diagnostic prediction output from the output layer 533 (step S502).

The control unit 21 inputs the acquired endoscopic image 49 into the converter 63 included in the first model 61, and calculates the score (step S631).

The control unit 21 determines whether or not all the scores have been calculated (step S632). If it is determined that the process has not been completed (NO in step S632), the control unit 21 returns to step S631. When it is determined that the process is completed (YES in step S632), the control unit 21 generates an image described using the lower part of FIG. 25 and outputs the image to the display device 16 (step S633). The control unit 21 ends the process.

According to this embodiment, since the learning model generated by deep learning is only the second model 62, the diagnosis support system 10 can be realized with a relatively small amount of calculation.

A part of the first score, the second score, and the third score may be calculated by the same method as in the first embodiment or the third embodiment.

[Embodiment 5]
The present embodiment relates to a diagnostic support system 10 that supports the diagnosis of locally occurring diseases such as cancer or polyps. The description of the parts common to the first embodiment or the second embodiment will be omitted.

FIG. 29 is an explanatory diagram illustrating an outline of the diagnosis support system 10 of the fifth embodiment. The endoscope image 49 taken by the endoscope 14 is input to the second model 62. In the second model 62, when the endoscopic image 49 is input, the area prediction that predicts the range of the lesion area 74 that is predicted to have a lesion such as a polyp or cancer, and the area prediction that the lesion is Outputs diagnostic predictions such as benign or malignant. In FIG. 29, the probability that the polyp in the lesion area 74 is "malignant" is 5% and the probability that it is "benign" is 95%.

The second model 62 uses an arbitrary object detection algorithm such as RCNN (Regions with Convolutional Neural Network), Fast RCNN, Faster RCNN, SSD (Single Shot Multibook Detector), or YOLO (You Only Look Once). This is the generated learning model. Since a learning model that accepts the input of a medical image and outputs the region where the lesion exists and the diagnostic prediction has been conventionally used, the description thereof will be omitted in detail.

The first model 61 includes a first score learning model 611, a second score learning model 612, and a third score learning model 613. The first score learning model 611 outputs a predicted value of the first score indicating the degree of boundary clarity when an image in the lesion area 74 is input. The second score learning model 612 outputs a predicted value of the second score indicating the degree of unevenness of the surface when an image in the lesion region 74 is input. The third score learning model 613 outputs a predicted value of the third score indicating the degree of redness when an image in the lesion area 74 is input.

In the example shown in FIG. 29, the predicted values that the first score is 50, the second score is 5, and the third score is 20 are output. The first model 61 may include a score learning model that outputs diagnostic criteria predictions related to various diagnostic criteria items related to polyps, such as a shape such as whether it is pedunculated or not, and the degree of secretion adhesion.

The outputs of the first model 61 and the second model 62 are acquired by the first acquisition unit and the second acquisition unit, respectively. Based on the outputs acquired by the first acquisition unit and the second acquisition unit, the screen shown at the bottom of FIG. 29 is displayed on the display device 16. Since the displayed screen is the same as the screen described in the first embodiment, the description thereof will be omitted.

When a plurality of lesion regions 74 are detected in the endoscopic image 49, each lesion region 74 is input to the first model 61, and a diagnostic criterion prediction is output. By selecting the lesion area 74 displayed in the endoscopic image field 73, the user can view the diagnostic prediction and the score regarding the lesion area 74. The diagnostic predictions and scores for the plurality of lesion areas 74 may be displayed in a list on the screen.

The lesion area 74 may be surrounded by a circle, an ellipse, or any closed curve. In such a case, by masking the peripheral region with black or white, an image corrected to a shape suitable for input to the first model 61 is input to the first model 61. For example, when a plurality of polyps are close to each other, a region including one polyp can be cut out and a score can be calculated by the first model 61.

[Embodiment 6]
The present embodiment relates to a diagnostic support system 10 that outputs the probability that the first model 61 is in each category defined in the diagnostic criteria for diseases. The description of the parts common to the first embodiment will be omitted.

FIG. 30 is an explanatory diagram illustrating the configuration of the first score learning model 611 of the sixth embodiment. The first score learning model 611 described with reference to FIG. 30 is used in place of the first score learning model 611 described with reference to FIG.

In the first score learning model 611, when the endoscopic image 49 is input, the degree of redness is "judgment 1", "judgment 2", and "judgment 3" based on the diagnostic criteria for ulcerative colitis. The output layer 533 has three output nodes that output the probabilities of each of the three stages. "Judgment 1" means that the degree of redness is "normal", "judgment 2" means "erythema", and "judgment 3" means "strong erythema".

Similarly, in the second score learning model 612, "judgment 1" indicates that the degree of blood vessel see-through is "normal", and "judgment 2" indicates that the blood vessel see-through is "mottled". “Judgment 3” means that they have “disappeared” over almost the entire area.

The number of nodes in the output layer 533 of the score learning model is arbitrary. In the present embodiment, the third score learning model 613 has four output nodes from "determination 1" to "determination 4" in the output layer 533. "Judgment 1" indicates that the degree of ulcer is "none", "judgment 2" indicates "erosion", and "judgment 3" indicates that the ulcer has a "medium" depth. "Judgment 4" means that it is a "deep" erosion, respectively.

FIG. 31 is an explanatory diagram illustrating the screen display of the sixth embodiment. The endoscope image field 73 is displayed in the upper left part of the screen. The first result column 71 and the first stop button 711 are displayed on the right side of the screen. A second result column 72 and a second stop button 722 are displayed below the endoscope image column 73.

According to the present embodiment, it is possible to provide the diagnostic support system 10 that displays the first result column 71 in an expression according to the definition defined in the diagnostic criteria.

[Embodiment 7]
The present embodiment relates to a diagnostic support system 10 that displays a warning when there is a discrepancy between the output of the first model 61 and the output of the second model 62. The description of the parts common to the first embodiment will be omitted.

FIG. 32 is an explanatory diagram illustrating the screen display of the seventh embodiment. In the example shown in FIG. 32, the probability of being normal is 70%, the first score indicating the degree of redness is 70, the second score indicating the degree of vascular see-through is 50, and the third score indicating the degree of ulcer is 5. The diagnostic criterion prediction that there is is output.

A warning column 75 is displayed at the bottom of the screen. The warning column 75 should be judged not to be "normal" when the first score, which is the degree of "redness" according to the diagnostic criteria, is high. Therefore, the first result column 71 and the second result Indicates that there is a discrepancy with column 72. The presence or absence of discrepancy is determined on a rule basis based on diagnostic criteria.

As described above, when there is a discrepancy between the output by the first model 61 and the output by the second model 62, the warning column 75 is displayed to call the attention of the doctor who is the user.

[Embodiment 8]
The present embodiment relates to a diagnostic support system 10 in which an endoscope processor 11 and an information processing device 20 are integrated. The description of the parts common to the first embodiment will be omitted.

FIG. 33 is an explanatory diagram illustrating an outline of the diagnosis support system 10 of the eighth embodiment. In FIG. 33, illustration and description of a configuration for realizing the basic functions of the endoscope processor 11, such as a light source, an air supply / water supply pump, and a control unit of an image sensor 141, will be omitted.

The diagnostic support system 10 includes an endoscope 14 and an endoscope processor 11. The endoscope processor 11 includes an endoscope connection unit 12, a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, a display device I / F26, an input device I / F27, and a bus.

Since the control unit 21, the main storage device 22, the auxiliary storage device 23, the communication unit 24, the display device I / F26, and the input device I / F27 are the same as those in the first embodiment, the description thereof will be omitted. The endoscope 14 is connected to the endoscope connecting portion 12 via the endoscope connector 15.

According to the present embodiment, the control unit 21 receives a video signal from the endoscope 14 via the endoscope connection unit 12 and performs various image processing to obtain an endoscope image 49 suitable for observation by a doctor. Generate. The control unit 21 inputs the generated endoscopic image 49 into the first model 61 and acquires the diagnostic standard prediction of each item according to the diagnostic standard. The control unit 21 inputs the generated endoscopic image 49 into the second model 62 to acquire a diagnosis prediction of the disease.

Even if the first model 61 and the second model 62 are configured to accept a video signal acquired from the endoscope 14 or an image in the process of generating an endoscope image 49 based on the video signal. good. By doing so, it is possible to provide the diagnostic support system 10 that can use the information lost in the process of generating an image suitable for observation by a doctor.

[Embodiment 9]
The present embodiment relates to a diagnostic support system 10 that displays a region of the endoscopic image 49 that has influenced the diagnostic reference prediction output from the first model 61. The description of the parts common to the first embodiment will be omitted.

FIG. 34 is an explanatory diagram illustrating an outline of the diagnosis support system 10 of the ninth embodiment. FIG. 34 shows a diagnostic support system 10 in which an extraction unit 66 for extracting a region affecting the second score is added to the diagnostic support system 10 of the first embodiment described with reference to FIG.

Similar to the first embodiment, the endoscopic image 49 is input to the first model 61 and the second model 62, and their respective outputs are acquired by the first acquisition unit and the second acquisition unit. Of the endoscopic images 49, the region of interest that affects the second score is extracted by the extraction unit 66.

The extraction unit 66 can be realized by an algorithm of a known region of interest visualization method such as CAM (Class Activation Mapping), Grad-CAM (Gradient-weighted Class Activation Mapping), or Grad-CAM ++.

The extraction unit 66 may be realized by software executed by the control unit 21 or by hardware such as an image processing chip. In the following description, a case where the extraction unit 66 is realized by software will be described as an example.

The control unit 21 displays the screen shown at the bottom of FIG. 34 on the display device 16 based on the output acquired by the first acquisition unit and the second acquisition unit and the region of interest extracted by the extraction unit 66. The screen to be displayed includes an endoscopic image column 73, a first result column 71, a second result column 72, and a region of interest column 78.

In the endoscope image column 73, an endoscope image 49 taken by using the endoscope 14 is displayed in real time. In the first result column 71, the diagnostic criterion prediction output from the first model 61 is displayed. The diagnostic prediction output from the second model 62 is displayed in the second result column 72.

In the example shown in FIG. 34, the selection cursor 76 indicates that the item of "blood vessel see-through" indicating the second score in the first result column 71 is selected by the user.

In the attention area column 78, the attention area extracted by the extraction unit 66 is displayed by the attention area index 781. The region of interest index 781 expresses the magnitude of the influence on the second score by a heat map or contour line display. In FIG. 34, the attention area index 781 is displayed by using finer hatching in a place having a stronger influence on the diagnostic criterion prediction. The attention area index 781 surrounds a region having a stronger influence on the second score than a predetermined threshold value. It may be expressed by a frame or the like.

When the "redness" item indicating the first score is selected by the user, the selection cursor 76 is displayed in the "redness" item. The extraction unit 66 extracts a region that affects the first score. Similarly, when the "ulcer" item indicating the third score is selected by the user, the selection cursor 76 is displayed in the "ulcer" item. The extraction unit 66 extracts a region that affects the third score. If none of the diagnostic criteria items is selected by the user, the selection cursor 76 is not displayed and the attention area index 781 is not displayed in the attention area column 78.

The diagnosis support system 10 may be able to simultaneously accept selection of a plurality of diagnostic reference items. In such a case, the diagnostic support system 10 has a plurality of extraction units 66 that extract regions that have influenced the diagnostic criterion prediction for each diagnostic criterion item for which selection has been accepted.

FIG. 35 is an explanatory diagram illustrating the configuration of the first model 61. In the present embodiment, the configuration of the first model 61, which has been outlined with reference to FIG. 3, will be described in more detail.

The endoscopic image 49 is input to the feature amount extraction unit 551. The feature amount extraction unit 551 is composed of repeating a convolution layer and a pooling layer. In the convolution layer, the convolution process of each of the plurality of filters and the input image is performed. In FIG. 35, the stacked quadrangles schematically show an image that has undergone convolution processing with different filters.

In the pooling layer, the input image is reduced. In the final layer of the feature amount extraction unit 551, a plurality of small images reflecting various features of the original endoscopic image 49 are generated. Data in which each pixel of these images is arranged in one dimension is input to the fully connected layer 552. The parameters of the feature amount extraction unit 551 and the fully connected layer 552 are adjusted by machine learning.

The output of the fully connected layer 552 is adjusted by the softmax layer 553 so that the total becomes 1, and the prediction probability of each node is output from the softmax layer 553. Table 1 shows an example of the output of the softmax layer 553.

For example, the probability that the value of the first score is 0 or more and less than 20 is output from the first node of the softmax layer 553. The probability that the value of the first score is 20 or more and less than 40 is output from the second node of the softmax layer 553. The total probability of all nodes is 1.

The representative value calculation unit 554 calculates and outputs a score that is a representative value of the output of the softmax layer 553. The representative value is, for example, the expected value of the score, the median value, or the like.

FIG. 36 is an explanatory diagram illustrating the configuration of the extraction unit 66. The control unit 21 sets “1” for the output node of the softmax layer 553 corresponding to the score calculated by the representative value calculation unit 554, and “0” for the other output nodes. The control unit 21 calculates the back propagation of the fully connected layer 552.

The control unit 21 generates a heat map based on the image of the final layer of the feature amount extraction unit 551 obtained by the back propagation. As described above, the region of interest index 781 is determined.

The heat map can be generated by a known method such as CAM (Class Activation Mapping), Grad-CAM (Gradient-weighted Class Activation Mapping), or Grad-CAM ++.

The control unit 21 may perform back propagation of the feature amount extraction unit 551 and generate a heat map based on an image other than the final layer.

For example, when using Grad-CAM, specifically, the control unit 21 has a model type of the first score learning model 611, the second score learning model 612, or the third score learning model 613, and a plurality of convolution layers. Accepts one of our layer names. The control unit 21 inputs the received model type and layer name into the Grad-CAM code, obtains the gradient, and generates a heat map. The control unit 21 displays the generated heat map, the model name and the layer name corresponding to the heat map on the display device 16.

FIG. 37 is a flowchart illustrating a processing flow of the program of the ninth embodiment. The program of FIG. 37 is executed by the control unit 21 instead of the program described with reference to FIG. Since the processing from step S501 to step S504 is the same as the processing flow of the program described with reference to FIG. 6, the description thereof will be omitted.

The control unit 21 determines whether or not the selection of the display relating to the region of interest is accepted (step S651). When it is determined that the selection is accepted (YES in step S651), the control unit 21 activates the subroutine of extracting the region of interest (step S652). The region of interest extraction subroutine is a subroutine that extracts the region of interest that has influenced the prediction of a predetermined diagnostic criterion from the endoscopic image 49. The processing flow of the subroutine for extracting the region of interest will be described later.

When it is determined that the selection is not accepted (NO in step S651), or after the end of step S652, the control unit 21 generates an image described using the lower part of FIG. 34 and outputs it to the display device 16. (Step S653). After that, the control unit 21 ends the process.

FIG. 38 is a flowchart illustrating a processing flow of the subroutine of extracting the region of interest. The region of interest extraction subroutine is a subroutine that extracts the region of interest that has influenced the prediction of a predetermined diagnostic criterion from the endoscopic image 49. The subroutine of extracting the region of interest realizes the function of the extraction unit 66 by software.

The control unit 21 determines the output node of the softmax layer 553 corresponding to the score calculated by the representative value calculation unit 554 (step S681). The control unit 21 sets "1" for the node determined in step S681 and "0" for the other nodes of the softmax layer. The control unit 21 calculates the back propagation of the fully connected layer 552 (step S682).

The control unit 21 generates an image corresponding to the final layer of the feature amount extraction unit 551. The control unit 21 performs a predetermined weighting on the generated plurality of images, and calculates the weight given to the softmax layer 553 by each part on the image. The control unit 21 determines the shape and position of the region of interest index 781 based on the portion having a large weight (step S683).

According to the present embodiment, it is possible to provide a diagnostic support system 10 that displays which part of the endoscopic image 49 affects the prediction of diagnostic criteria. By comparing the region of interest index 781 with the endoscopic image 49 displayed in the endoscopic image column 73, the user can understand which part of the endoscopic image 49 contributed to the prediction of the diagnostic criteria. For example, if a part that is not photographed normally, such as a part with residue or a part with flare, contributes to the diagnostic criterion prediction, the user should ignore the displayed diagnostic criterion prediction. I can judge.

By displaying the endoscopic image column 73 and the attention area column 78 separately, the user can observe the color, texture, and the like of the endoscopic image 49 without being hindered by the attention area index 781. By displaying the endoscope image column 73 and the attention area column 78 on the same scale, the user can more intuitively grasp the positional relationship between the endoscope image 49 and the attention area index 781.

[First modification]
FIG. 39 is an explanatory diagram illustrating a screen display of the first modification of the ninth embodiment. In this modification, the endoscopic image 49 and the attention area index 781 are superimposed and displayed in the attention area column 78. That is, the CPU 21 displays the same endoscope image 49 in the endoscope image column 73 and the attention area column 78.

According to this embodiment, the user can intuitively grasp the positional relationship between the endoscopic image 49 and the region of interest index 781. Further, by looking at the endoscopic image column 73, the endoscopic image 49 can be observed without being hindered by the region of interest index 781.

[Second modification]
This modification adds a function of displaying the region of interest index 781 to the diagnostic support system 10 of the sixth embodiment. Table 2 shows an example of the softmax layer 553 of the first score learning model 611.

For example, the probability that the reddish state is "normal" is output from the first node of the softmax layer 553. The probability of "with erythema" is output from the second node of the softmax layer 553. The probability of "with strong erythema" is output from the third node of the softmax layer 553.

The operation in the representative value calculation unit 554 is not performed, and the output node of the softmax layer 553 is output as it is from the first model 61.

FIG. 40 is an explanatory diagram illustrating a screen display of the second modification of the ninth embodiment. In this modified example, the probabilities of each category defined in the diagnostic criteria for diseases are output in the first result column 71.

In the example shown in FIG. 40, it is determined that the "normal" item in the first score of the first result column 71 and the "mottled disappearance" item in the second score are selected by the user. It is displayed by the selection cursor 76. Two attention area columns 78 arranged vertically are displayed in the center of FIG. 40.

The first score will be described as an example. The control unit 21 sets "1" for the output node of the softmax layer 553 corresponding to "normal" selected by the user, and "0" for the other output nodes. The control unit 21 performs back propagation of the fully connected layer 552 and generates a region of interest index 781 indicating a portion that influences the determination that the probability of being “normal” is 90%.

The control unit 21 displays the attention area index 781 regarding the probability that "redness" is "normal" in the attention area column 78 on the upper side. When the user changes the selection to the item of "erythema", the control unit 21 sets "1" for the output node of the softmax layer 553 corresponding to "erythema" and "0" for the other output nodes. The control unit 21 performs back propagation of the fully connected layer 552, generates an attention region index 781 indicating a portion that influences the determination that the probability of “erythema” is 10%, and updates the screen.

The user may operate the selection cursor 76 to select, for example, the "normal" item and the "erythema" item among the "redness" items. The user can confirm in the attention area column 78 a portion that affects the probability that "redness" is "normal" and a portion that affects the probability that "redness" is "erythema".

[Third variant]
This modification adds a function of displaying the region of interest index 781 for the item of diagnostic prediction. FIG. 41 is an explanatory diagram illustrating a screen display of a third modification of the ninth embodiment.

In the example shown in FIG. 41, the selection cursor 76 indicates that the "mild" item in the second result column 72 is selected by the user. In the region of interest column 78, the region of interest index 781 indicating a site that affects the determination that ulcerative colitis is "mild" is displayed.

The user can confirm the portion that influences the determination that the probability of being "mild" is 20% by the attention area index 781. The user can reconfirm the validity of the determination of "mild" by the second model 62, for example, by further observing the location indicated by the region of interest index 781 from a different direction.

[Embodiment 10]
The present embodiment relates to a diagnostic support system 10 that realizes the extraction unit 66 without using back propagation.

FIG. 42 is a flowchart illustrating a processing flow of the subroutine of interest region extraction according to the tenth embodiment. The region of interest extraction subroutine is a subroutine that extracts the region of interest that has influenced the prediction of a predetermined diagnostic criterion from the endoscopic image 49. The subroutine described with reference to FIG. 42 is executed in place of the subroutine described with reference to FIG. 38.

The control unit 21 selects one pixel from the endoscopic image 49 (step S661). The control unit 21 gives a minute change to the pixel selected in step S661 (step S662). The minute change is given by adding or subtracting 1 to any value of RGB (Red Green Blue) of the selected pixel.

The control unit 21 inputs the changed endoscopic image 49 into the first model 61 regarding the item selected by the user, and acquires the diagnostic criterion prediction (step S663). The control unit 21 calculates the amount of change in the diagnostic standard prediction compared with the diagnostic standard prediction acquired based on the endoscopic image 49 before the change is applied (step S664).

The stronger the influence on the diagnostic standard prediction, the larger the amount of change in the diagnostic standard prediction due to a minute change in the pixel. Therefore, the amount of change calculated in step S664 indicates the strength of the influence of the pixel on the diagnostic reference prediction.

The control unit 21 records the amount of change calculated in step S664 in association with the position of the pixel selected in step S661 (step S665). The control unit 21 determines whether or not the processing of all the pixels has been completed (step S666). If it is determined that the process has not been completed (NO in step SS666), the control unit 21 returns to step S661.

When it is determined that the process is completed (YES in step S666), the control unit 21 maps the amount of change based on the position of the pixel and the amount of change (step S667). The mapping is performed, for example, by creating a heat map based on the magnitude of the amount of change or creating contour lines, and the shape and position of the region of interest index 781 indicating the region where the amount of change is large is determined. After that, the control unit 21 ends the process.

In step S661, the control unit 21 may select pixels every few pixels in both the vertical and horizontal directions, for example. By thinning out the pixels and processing, the processing of the subroutine of extracting the region of interest can be speeded up.

The processes from step S651 to step S653 of FIG. 37 are executed instead of step S604 of the program for endoscopy of the third embodiment described with reference to FIG. 24, and the present embodiment is executed in step S652. You may start the subroutine of. A function of displaying the region of interest index 781 can be added to the diagnostic support system 10 of the third embodiment.

The processes from step S651 to step S653 of FIG. 37 are executed instead of step S633 of the program for endoscopy of the fourth embodiment described with reference to FIG. 28, and the present embodiment is executed in step S652. You may start the subroutine of. A function of displaying the region of interest index 781 can be added to the diagnostic support system 10 of the fourth embodiment.

According to this embodiment, the region of interest index 781 is displayed even when the first model 61 does not have the softmax layer 553 and the fully connected layer 552, that is, even when a method other than the neural network model 53 is used. A possible diagnostic support system 10 can be provided.

The program of the present embodiment can also be applied to the extraction of the region of interest of the second model 62. In this case, in step S663 of the subroutine of interest region extraction described with reference to FIG. 42, the control unit 21 inputs the changed endoscopic image 49 into the second model 62 for diagnosis. Get the forecast. In the following step S664, the control unit 21 determines the probability of being "mild" acquired based on the endoscopic image 49 before the change is applied and the probability of being "mild" acquired in step S664. By comparison, the amount of change in the diagnostic prediction is calculated.

[Embodiment 11]
FIG. 43 is a functional block diagram of the information processing device 20 according to the eleventh embodiment. The information processing device 20 includes an image acquisition unit 281, a first acquisition unit 282, an extraction unit 66, and an output unit 283. The image acquisition unit 281 acquires the endoscopic image 49.

The first acquisition unit 282 inputs the endoscopic image 49 acquired by the image acquisition unit 281 into the first model 61 that outputs the diagnostic standard prediction regarding the diagnostic criteria of the disease when the endoscopic image 49 is input. And get the output diagnostic criteria prediction. The extraction unit 66 extracts from the endoscopic image 49 a region that has influenced the diagnostic criterion prediction acquired by the first acquisition unit 282. The output unit 283 associates the diagnostic criterion prediction acquired by the first acquisition unit 282, the index indicating the region extracted by the extraction unit 66, and the diagnostic prediction regarding the disease state acquired based on the endoscopic image 49. Output.

[Embodiment 12]
The present embodiment relates to a mode in which the diagnostic support system 10 of the present embodiment is realized by operating a general-purpose computer 90 and a program 97 in combination. FIG. 44 is an explanatory diagram showing the configuration of the diagnosis support system 10 of the twelfth embodiment. The description of the parts common to the first embodiment will be omitted.

The diagnostic support system 10 of the present embodiment includes a computer 90, an endoscope processor 11, and an endoscope 14. The computer 90 includes a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, a display device I / F26, an input device I / F27, a reading unit 29, and a bus. The computer 90 is an information device such as a general-purpose personal computer, a tablet, or a server computer.

Program 97 is recorded on the portable recording medium 96. The control unit 21 reads the program 97 via the reading unit 29 and stores it in the auxiliary storage device 23. Further, the control unit 21 may read the program 97 stored in the semiconductor memory 98 such as the flash memory mounted in the computer 90. Further, the control unit 21 may download the program 97 from the communication unit 24 and another server computer (not shown) connected via a network (not shown) and store the program 97 in the auxiliary storage device 23.

The program 97 is installed as a control program of the computer 90, loaded into the main storage device 22, and executed. As a result, the computer 90, the endoscope processor 11, and the endoscope 14 function as the above-mentioned diagnostic support system 10.

The technical features (constituent requirements) described in each embodiment can be combined with each other, and by combining them, a new technical feature can be formed.
The embodiments disclosed this time should be considered to be exemplary in all respects and not restrictive. The scope of the present invention is indicated by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

(Appendix 1)
An image acquisition unit that acquires endoscopic images,
When an endoscopic image is input, the endoscopic image acquired by the image acquisition unit is input to the first model that outputs the diagnostic standard prediction regarding the diagnostic criteria of the disease, and the output diagnostic standard prediction is acquired. The first acquisition department to do
An information processing device including an output unit that outputs a diagnostic reference prediction acquired by the first acquisition unit in association with a diagnostic prediction regarding the state of the disease acquired based on the endoscopic image.

(Appendix 2)
The information processing apparatus according to Appendix 1, wherein the first acquisition unit acquires diagnostic standard predictions for each item from a plurality of first models that output diagnostic standard predictions for a plurality of items included in the diagnostic criteria for the disease. ..

(Appendix 3)
The information processing device according to Appendix 1 or Appendix 2, wherein the first model is a learning model generated by machine learning.

(Appendix 4)
The information processing device according to Appendix 1 or Appendix 2, wherein the first model outputs a numerical value calculated based on an endoscopic image acquired by the image acquisition unit.

(Appendix 5)
The information processing apparatus according to any one of Supplementary note 1 to Supplementary note 4, further comprising a first reception unit that receives an operation stop instruction of the first acquisition unit.

(Appendix 6)
The diagnostic prediction is based on the diagnostic prediction output by inputting the endoscopic image acquired by the image acquisition unit into the second model that outputs the diagnostic prediction of the disease when the endoscopic image is input. The information processing apparatus according to any one of Supplementary note 1 to Supplementary note 5.

(Appendix 7)
The information processing apparatus according to Appendix 6, wherein the second model is a learning model generated by machine learning.

(Appendix 8)
The second model is
The input layer where the endoscopic image is input and
An output layer that outputs disease diagnosis predictions,
It is a neural network model including an intermediate layer in which parameters are learned by a plurality of sets of teacher data recorded by associating an endoscopic image with a diagnostic prediction.
The information processing apparatus according to Appendix 6 or Appendix 7, wherein the first model outputs a diagnostic criterion prediction based on a feature amount acquired from a predetermined node in the intermediate layer.

(Appendix 9)
The second model outputs a region prediction for a lesion region containing the disease when an endoscopic image is input.
The first model outputs a diagnostic criterion prediction regarding a diagnostic criterion of a disease when an endoscopic image of a lesion area is input.
The first acquisition unit inputs a portion of the endoscopic image acquired by the image acquisition unit that corresponds to the region prediction output from the second model into the first model, and outputs a diagnostic reference prediction. The information processing apparatus according to Appendix 6 or Appendix 7.

(Appendix 10)
The information processing apparatus according to any one of Supplementary note 6 to Supplementary note 9, further comprising a second reception unit that receives an instruction to stop acquisition of the diagnostic prediction.

(Appendix 11)
The information processing device according to any one of Supplementary note 6 to Supplementary note 10, wherein the output unit also outputs an endoscopic image acquired by the image acquisition unit.

(Appendix 12)
The image acquisition unit acquires an endoscopic image taken during the endoscopy in real time.
The information processing device according to any one of Supplementary note 1 to Supplementary note 11, wherein the output unit outputs in synchronization with the acquisition of an endoscopic image by the image acquisition unit.

(Appendix 13)
An endoscope connection unit to which an endoscope is connected, an image generation unit that generates an endoscope image based on an image signal acquired from an endoscope connected to the endoscope connection unit, and an image generation unit.
When an endoscopic image is input, the endoscopic image generated by the image generator is input to the first model that outputs the diagnostic standard prediction regarding the diagnostic criteria of the disease, and the output diagnostic standard prediction is acquired. The first acquisition department to do
A processor for an endoscope including an output unit that outputs a diagnostic reference prediction acquired by the first acquisition unit in association with a diagnostic prediction regarding the state of the disease acquired based on the endoscopic image.

(Appendix 14)
Get an endoscopic image and
When the endoscopic image is input, the acquired endoscopic image is input to the first model that outputs the diagnostic standard prediction regarding the diagnostic criteria of the disease, and the output diagnostic standard prediction is acquired.
An information processing method in which a computer executes a process of outputting an acquired diagnostic criterion prediction in association with a diagnostic prediction regarding the state of the disease acquired based on the endoscopic image.

(Appendix 15)
Get an endoscopic image and
When the endoscopic image is input, the acquired endoscopic image is input to the first model that outputs the diagnostic standard prediction regarding the diagnostic criteria of the disease, and the output diagnostic standard prediction is acquired.
A program that causes a computer to execute a process of outputting an acquired diagnostic criterion prediction in association with a diagnostic prediction regarding the state of the disease acquired based on the endoscopic image.

(Appendix 16)
Acquire a plurality of sets of teacher data recorded by associating the endoscopic image with the judgment result determined for the diagnostic criteria used for diagnosing the disease.
A method of generating a model that uses the teacher data to generate a first model that outputs a diagnostic criterion prediction that predicts a diagnostic criterion of a disease when an endoscopic image is input.

(Appendix 17)
The teacher data includes a determination result determined for each of a plurality of diagnostic criteria items included in the diagnostic criteria.
The model generation method according to Appendix 16, wherein the first model is generated corresponding to each of the plurality of diagnostic reference items.

(Appendix 18)
The first model is generated by deep learning that adjusts the parameters of the intermediate layer so that the acquired determination result is output from the output layer when the acquired endoscopic image is input to the input layer. Alternatively, the model generation method according to Appendix 17.

(Appendix 19)
The first model is
When the endoscopic image is input, the endoscopic image in the acquired teacher data is input to the neural network model that outputs the diagnosis prediction of the disease.
A plurality of features related to the input endoscopic image are acquired from the nodes constituting the intermediate layer of the neural network model.
From the acquired plurality of features, a feature having a high correlation with the determination result associated with the endoscopic image is selected.
The model according to Appendix 16 or Appendix 17, which is generated by determining a calculation method for calculating the score based on the selected feature amount by regression analysis of the selected feature amount and the score obtained by quantifying the determination result. Generation method.

(Appendix 20)
The first model is
Multiple features are extracted from the acquired endoscopic image and
From the extracted plurality of features, a feature having a high correlation with the determination result associated with the endoscopic image is selected.
The model according to Appendix 16 or Appendix 17, which is generated by determining a calculation method for calculating the score based on the selected feature amount by regression analysis of the selected feature amount and the score obtained by quantifying the determination result. Generation method.

(Appendix 21)
The disease is ulcerative colitis,
The model generation method according to any one of Supplementary note 16 to Supplementary note 20, wherein the diagnostic criterion prediction is a prediction regarding redness of an endoscopic image, blood vessel fluoroscopy, or severity of ulcer.

(Appendix 22)
Acquire a plurality of sets of teacher data recorded by associating the endoscopic image with the judgment result determined for the diagnostic criteria used for diagnosing the disease.
When the endoscopic image is input, the endoscopic image in the acquired teacher data is input to the neural network model that outputs the diagnosis prediction of the disease.
A plurality of features related to the input endoscopic image are acquired from the nodes constituting the intermediate layer of the neural network model.
The acquired multiple features are recorded in association with the numerical score of the judgment result associated with the input endoscopic image.
Based on the correlation between each of the recorded features and the score, a feature with a high correlation with the score is selected.
By determining a calculation method for calculating the score based on the selected feature amount by regression analysis of the selected feature amount and the score, the diagnostic criteria for the disease are predicted when an endoscopic image is input. A program that causes a computer to execute a process that generates a first model that outputs the diagnostic criteria prediction.

10 Diagnostic support system 11 Endoscope processor 12 Endoscope connection part 14 Endoscope 141 Imaging element 142 Insertion part 15 Endoscope connector 16 Display device 161 First display device 162 Second display device 17 Keyboard 19 Model generation system 20 Information processing device 21 Control unit 22 Main storage device 23 Auxiliary storage device 24 Communication unit 26 Display device I / F
27 Input device I / F
281 Image acquisition unit 282 First acquisition unit 283 Output unit 29 Reading unit 30 Server 31 Control unit 32 Main storage device 33 Auxiliary storage device 34 Communication unit 40 Client 41 Control unit 42 Main storage device 43 Auxiliary storage device 44 Communication unit 46 Display unit 47 Input unit 49 Endoscopic image 53 Neural network model 531 Input layer 532 Intermediate layer 533 Output layer 551 Feature quantity extraction unit 552 Fully connected layer 553 Softmax layer 554 Representative value calculation unit 61 1st model 611 1st score learning model 612 2nd score learning model 613 3rd score learning model 62 2nd model 63 Converter 631 1st converter 632 2nd converter 633 3rd converter 64 Teacher data DB
65 Feature 651 1st feature 652 2nd feature 653 3rd feature 66 Extractor 71 1st result column 711 1st stop button 72 2nd result column 722 2nd stop button 73 Endoscopic image column 74 Lesion area 75 Warning column 76 Selection cursor 78 Attention area column 781 Attention area index (index)
81 1st input field 811 1st score input field 812 2nd score input field 813 3rd score input field 82 2nd input field 86 Patient ID field 87 Disease name field 88 Model button 89 Next button 90 Computer 96 Portable recording medium 97 Program 98 semiconductor memory

Claims (8)

An image acquisition unit that acquires endoscopic images,
The diagnostic criteria predictive of the plurality of items relating to the diagnostic criteria for the disease to a plurality of first models to be output respectively when the endoscopic image inputted by an endoscopic image by the image acquisition unit has acquired input respectively, The first acquisition unit that acquires the diagnostic criteria prediction of each item output from each first model, and
A reception unit that accepts selection items from multiple items mentioned above,
From the endoscopic image, an extraction unit that extracts a region that influences the diagnostic criterion prediction regarding the selection item accepted by the reception unit, and an extraction unit.
An output unit that outputs a diagnostic criterion prediction acquired by the first acquisition unit, an index indicating a region extracted by the extraction unit, and a diagnostic prediction regarding the disease state acquired based on the endoscopic image in association with each other. Information processing device equipped with. The information processing device according to claim 1, wherein the output unit outputs the endoscopic image and the index side by side. The information processing device according to claim 1, wherein the output unit outputs the endoscopic image and the index in an overlapping manner. The information processing apparatus according to any one of claims 1 to 3 , further comprising a stop reception unit that receives an operation stop instruction of the extraction unit. A second acquisition in which the endoscopic image acquired by the image acquisition unit is input to the second model that outputs the diagnostic prediction of the disease when the endoscopic image is input, and the output diagnostic prediction is acquired. With a part
The output unit is described in any one of claims 1 to 4 , which outputs a diagnostic standard prediction acquired by the second acquisition unit, a diagnostic prediction acquired by the first acquisition unit, and the index. Information processing equipment. An endoscope connection unit to which an endoscope is connected, an image generation unit that generates an endoscope image based on an image signal acquired from an endoscope connected to the endoscope connection unit, and an image generation unit.
The diagnostic criteria predictive of the plurality of items relating to the diagnostic criteria for the disease to a plurality of first model output to when a video signal acquired from the endoscope is inputted, the video signal acquired from the endoscope type, respectively And the first acquisition unit that acquires the diagnostic criteria prediction of each item output from each first model,
A reception unit that accepts selection items from multiple items mentioned above,
From the endoscopic image, an extraction unit that extracts a region that influences the diagnostic criterion prediction regarding the selection item accepted by the reception unit, and an extraction unit.
An output unit that outputs a diagnostic criterion prediction acquired by the first acquisition unit, an index indicating a region extracted by the extraction unit, and a diagnostic prediction regarding the disease state acquired based on the endoscopic image in association with each other. Endoscopic processor with and. Get an endoscopic image and
When the endoscopic image is input, the acquired endoscopic image is input to each of the plurality of first models that output the diagnostic criteria prediction of a plurality of items related to the diagnostic criteria of the disease, and each first model. Get the diagnostic criteria prediction for each item output from
Accepts selection items from multiple items mentioned above,
From the endoscopic image, the region that affected the diagnostic criterion prediction regarding the accepted selection item was extracted.
The acquired diagnostic criteria predictive, and indicator of the extracted area, the information processing method of a computer diagnostic prediction and outputting in association with about the state of the disease acquired is performed based on the endoscopic image. Get an endoscopic image and
The diagnostic criteria predictive of the plurality of items relating to the diagnostic criteria for the disease to a plurality of first models to be output respectively when the endoscopic image inputted, type endoscope images obtained respectively, each of the first model Get the diagnostic criteria prediction for each item output from
Accepts selection items from multiple items mentioned above,
From the endoscopic image, the region that affected the diagnostic criterion prediction regarding the accepted selection item was extracted.
A program that causes a computer to execute a process of associating and outputting an acquired diagnostic criterion prediction, an index indicating an extracted area, and a diagnostic prediction regarding the state of the disease acquired based on the endoscopic image.

Images