WO2023026538A1

WO2023026538A1 - Medical assistance system, medical assistance method, and evaluation assistance device

Info

Publication number: WO2023026538A1
Application number: PCT/JP2022/010844
Authority: WO
Inventors: 真司渡辺; 健人竹中
Original assignee: ソニーグループ株式会社
Priority date: 2021-08-27
Filing date: 2022-03-11
Publication date: 2023-03-02

Abstract

Provided is a medical assistance system comprising: an image acquisition unit (202) that sequentially acquires a plurality of affected part images obtained by imaging an affected part; a determination unit (206) that determines whether the acquired affected part images satisfy a quality condition for evaluating the affected part; an evaluation execution determination unit (208) that determines whether evaluation of the affected part can be executed on the basis of the affected part images determined to satisfy the image quality being continuously acquired; an evaluation unit (210) that, when it is determined that the affected part is to be evaluated, derives information pertaining to an evaluation value for the affected part on the basis of the plurality of affected part images determined to satisfy the image quality condition; and an output unit (214) that outputs information pertaining to the evaluation value.

Description

Medical support system, medical support method and evaluation support device

The present disclosure relates to a medical support system, a medical support method, and an evaluation support device.

Doctors treat patients using various methods. For example, ulcerative colitis is treated using an endoscope. Specifically, the doctor evaluates the disease condition of the affected area from the captured image obtained by the endoscope, and performs treatment while controlling the dosage and the like based on the evaluation. In addition, when it is difficult to evaluate the condition of a patient by follow-up observation using such an endoscope, a part of the affected tissue may be excised for pathological diagnosis.

WO2020/121906

In recent years, technology that uses artificial intelligence (AI) systems (medical support systems) has been proposed for endoscopic disease assessment of ulcerative colitis. Specifically, an image of the affected area (affected area image) is taken with an endoscope, and an AI system that functions as a determiner built in advance by machine learning is used to analyze the affected area image and evaluate the disease status of the affected area. In this way, by using the AI system, it is possible to avoid variations in evaluation among doctors, and treat patients based on evaluation results that are more objective and highly reproducible. becomes possible.

In addition, for the technology for disease status evaluation using AI system, in order for doctors to perform stress-free and appropriate disease status evaluation, along with the display of movies etc. of the affected area, highly accurate real-time AI system Evaluation results are required to be presented.

Therefore, the present disclosure proposes a medical support system, a medical support method, and an evaluation support device capable of presenting highly accurate evaluation results in real time.

According to the present disclosure, there is provided a medical support system including an information processing device and a program for causing the information processing device to execute information processing related to medical support, wherein the information processing device treats an affected area according to the program. an image acquisition unit that sequentially acquires a plurality of images of the affected area obtained by imaging; a determination unit that determines whether the acquired images of the affected area satisfy image quality conditions for evaluating the affected area; and the image quality conditions. and an evaluation execution determination unit that determines whether or not to execute the evaluation of the affected area based on the continuous acquisition of the affected area images determined to satisfy an evaluation unit that derives information about an evaluation value for the affected area based on the plurality of affected area images determined to satisfy the image quality condition; and an output unit that outputs information about the evaluation value. A functioning medical support system is provided.

Further, according to the present disclosure, the processor sequentially acquires a plurality of affected area images obtained by imaging the affected area, and whether the acquired affected area images satisfy image quality conditions for evaluating the affected area Determining whether to execute the evaluation of the affected area based on the fact that the affected area images determined to satisfy the image quality condition are continuously acquired; deriving information about an evaluation value for the affected area based on the plurality of images of the affected area determined to satisfy the image quality condition when execution of evaluation is determined; and outputting information about the evaluation value. A method of medical assistance is provided, comprising:

Furthermore, according to the present disclosure, an image acquisition unit that sequentially acquires a plurality of evaluation target images obtained by imaging an evaluation target, and an image quality condition for the acquired evaluation target images to evaluate the evaluation target. a determination unit that determines whether the image quality condition is satisfied; and a decision as to whether or not to perform the evaluation of the evaluation target based on the continuous acquisition of the evaluation target images that are determined to satisfy the image quality condition. and an evaluation execution determination unit that, when execution of evaluation of the evaluation target is determined, based on the plurality of evaluation target images determined to satisfy the image quality condition, information regarding an evaluation value for the evaluation target. An evaluation support device is provided, comprising: an evaluation unit that derives the evaluation value; and an output unit that outputs information about the evaluation value together with the evaluation target image.

1 is a schematic diagram showing a technical overview of a medical support system 10 according to an embodiment of the present disclosure; FIG. 1 is a block diagram showing an example configuration of a medical support system 10 according to an embodiment of the present disclosure; FIG. 4 is a flow chart showing the flow of work by a user in the medical support method according to the embodiment of the present disclosure; 4 is a flowchart (part 1) showing a score derivation method performed by the medical support device 200 in the medical support method according to the embodiment of the present disclosure; FIG. 4 is a schematic diagram illustrating an overview of skip frame determination and parallel processing in the derivation method according to the embodiment of the present disclosure; FIG. 4 is a schematic diagram illustrating an example of step determination in skip frame determination according to an embodiment of the present disclosure; FIG. 4 is a schematic diagram illustrating details of skip frame determination in the derivation method according to the embodiment of the present disclosure; 2 is a flowchart (part 2) showing a score derivation method performed by the medical support device 200 in the medical support method according to the embodiment of the present disclosure; FIG. 3 is a schematic diagram illustrating an example of a derivation method according to an embodiment of the present disclosure; FIG. FIG. 3 is a schematic diagram (part 1) illustrating an example of display in the embodiment of the present disclosure; 10 is a flowchart showing ranking display performed by the medical support device 200 in the medical support method according to the embodiment of the present disclosure; FIG. 2 is a schematic diagram (part 2) illustrating an example of display in the embodiment of the present disclosure; FIG. 3 is a schematic diagram illustrating an example of an ultrasound image; 2 is a hardware configuration diagram showing an example of a computer that implements the functions of the medical support device 200. FIG.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description. In addition, in this specification and drawings, a plurality of components having substantially the same or similar functional configuration may be distinguished by attaching different alphabets after the same reference numerals. However, when there is no particular need to distinguish between a plurality of components having substantially the same or similar functional configurations, only the same reference numerals are used.

Note that the description will be given in the following order.
1. Technical overview 2. 3. Background leading to the creation of the embodiments of the present disclosure. Embodiment 3.1 Medical support system 10
3.2 Medical support device 200
3.3 Medical support method 3.4 Example 3.5 Application 4. Summary 5. 6. Hardware configuration. supplement

<<1. Technical Overview＞＞
First, a technical overview of a medical support system 10 according to an embodiment of the present disclosure will be described with reference to FIG. FIG. 1 is a schematic diagram illustrating a technical overview of a medical support system 10 according to the present disclosure. The medical support system 10 according to the embodiment of the present disclosure is a medical evaluation support system or a research evaluation system that can be used to determine the condition of an evaluation target in medical care or research. Examples of evaluation targets for the medical support system 10 include humans (patients) and animals (patients).

As shown in FIG. 1, in diagnosis of a digestive system disease, the doctor 1 examines the inside of the patient 2 using an endoscope 100 included in the medical support system 10, for example. At that time, an affected part image 400 inside the body of the patient 2 captured by the endoscope 100 is displayed by the display device 300 included in the medical support system 10 . The doctor 1 can examine the inside of the patient 2 by viewing the affected part image 400 displayed on the display device 300 . Furthermore, in the technology according to the present disclosure, the display device 300 displays the diseased part image 400 and the evaluation result of the disease state of the affected part derived by an algorithm (determiner) constructed by machine learning. Therefore, the doctor 1 can improve the diagnosis accuracy by making a diagnosis considering the evaluation result of the disease condition by the above algorithm.

Here, the evaluation result is a score (evaluation value) consisting of a specific numerical value indicating the evaluation of the disease state of the affected area determined by the doctor (endoscopist) 1 by visually recognizing the surface of the affected area, or based on the score. It is a graded score and also includes a value indicating the pathological examination evaluation obtained by pathological examination of the affected area by a doctor (pathologist).

Specifically, the score (evaluation value) for the disease state of the affected area is the value scored by Doctor 1 for the bleeding, tumor, or fluoroscopic image of the affected area. In the embodiment of the present disclosure, an evaluation value obtained by any known scoring method can be used as the score for the disease state of the affected area.

For example, a score based on the Ulcerative Colitis Endoscopic Index of Severity (UCEIS) can be used as the score for the disease condition of the affected area. The UCEIS score is an index that has recently come into use as an evaluation value of the severity of ulcerative colitis. UCEIS scores are defined for assessment of at least vascular permeability, bleeding, and ulceration of ulcerative colitis. Since the UCEIS score can be finely classified, it is possible to formulate a fine diagnostic policy and to reduce evaluation variations among endoscopists.

In addition, the Mayo score can be used as the score for the disease condition of the affected area. The Mayo score can classify the severity of ulcerative colitis on several scales from 0 to 3. Specifically, the Mayo score consists of assessment values for each of vascular permeability, bleeding, and ulcerative severity of ulcerative colitis by endoscopic findings, rectal bleeding, and comprehensive assessment by Physician 1.

In the Mayo score, "Mayo score remission" is defined as a case where the sum of the scores for each of the above items is 2 or less and none of the subscores exceeds 1. In the Mayo score, if the Mayo score has decreased by 3 or more or 30% or more from the baseline, and the rectal bleeding subscore has decreased by 1 or more, or the rectal bleeding subscore is 0 or 1 is defined as “Mayo score improvement”. Furthermore, the Mayo score defines "mucosal healing" when the endoscopic findings have a subscore of 0 or 1.

The evaluation value of the pathological examination is the evaluation value determined by the first grade doctor for the diagnosis result of the pathological examination by biopsy of the affected area. For example, the Geboes biopsy tissue finding score can be used as the evaluation value of the pathological examination. The Geboes score is an index generally used as an index for scoring pathological findings of ulcerative colitis.

As described above, the medical support system 10 according to the embodiment of the present disclosure can be used for diagnosis of digestive system diseases using the endoscope 100, for example. Specifically, the medical support system 10 can be suitably used for diagnosis of inflammatory bowel disease such as ulcerative colitis or Crohn's disease. In the following description, a case where the medical support system 10 according to the embodiment of the present disclosure is used for diagnosing ulcerative colitis will be described as an example. Application is not limited.

<<2. Background leading to the creation of the embodiments of the present disclosure >>
Next, before describing the embodiments of the present disclosure, the background leading to the creation of the embodiments of the present disclosure by the present inventors will be described.

Ulcerative colitis, one of the inflammatory bowel diseases, is a chronic disease in which erosions and ulcers occur due to inflammation in the mucous membrane of the large intestine. It affects more than 200,000 people, mainly young people. It is said that there are Major symptoms of ulcerative colitis include diarrhea, bloody stool, abdominal pain, fever, anemia, etc. In addition, various complications may occur. However, the cause of ulcerative colitis has not been identified, nor has any therapy been established, and it is one of the specific diseases designated by the Ministry of Health, Labor and Welfare.

In recent years, various treatment methods have been established for ulcerative colitis, and if the symptoms can be suppressed by appropriate treatment, patient 2 can continue a daily life that is almost the same as that of a healthy person. is becoming Under such circumstances, it is required to appropriately select a treatment method according to the disease state of ulcerative colitis, and disease state evaluation by lower gastrointestinal endoscopy, for example, has become very important.

In addition, ulcerative colitis may progress to a chronic state by repeating remission (symptoms are calm) and relapses (symptoms are worsening). known to increase the risk of developing In light of this, it is considered very important to appropriately estimate the prognosis of ulcerative colitis based on disease activity evaluation by lower gastrointestinal endoscopy.

However, the evaluation of ulcerative colitis and the like by visual recognition of the affected area depends on subjective judgment, and the degree of difficulty is high. It is difficult to avoid the occurrence of variations between 1.

In addition, instead of visualizing the affected area as described above, or in order to supplement the evaluation based on visual recognition of the affected area, it is effective to cut out a part of the tissue of the affected area and perform pathological examination and pathological diagnosis. However, tissue collection may damage the patient 2 and cause complications. Furthermore, since the number of pathologists who perform pathological examinations is small, pathological examinations cannot be performed anywhere, and diagnosis takes time. In addition, additional costs are incurred for the patient 2 by performing the pathological examination.

Under these circumstances, the present inventors have repeatedly studied methods for evaluating the disease state of ulcerative colitis using an endoscope, and it is effective to use an artificial intelligence (AI) system in the disease state evaluation. I came to the idea that it is. Specifically, in the disease state evaluation by the present inventors, an image of the affected area (affected area image 400) is captured by the endoscope 100, and an AI system (medical support system) that functions as a determiner built in advance by machine learning 10) is used to analyze the affected area image 400 and evaluate the disease state of the affected area. In this way, by using the AI system, it is possible to avoid variations in evaluation between doctors 1, and to treat patient 2 based on the evaluation results evaluated on a more objective and highly reproducible basis. can be done. Furthermore, by using the AI system, it is possible to thoroughly analyze the affected area image 400 compared to the case where the doctor 1 who is a human being performs the analysis, so that it is possible to prevent omissions in diagnosis.

Furthermore, when the present inventors diligently studied the technology for evaluating the disease state using an AI system, in order for the doctor 1 to perform stress-free and appropriate disease state evaluation, together with the display of a moving image of the affected area, etc. It turned out that it is important to present the evaluation results by the AI system in real time. In other words, at the same timing as the clinical findings by visual recognition (camera work) of the state of the affected area by the doctor 1, the highly accurate evaluation results by the AI system are presented, so the disease state evaluation technology using the AI system. , high usability is ensured, and the system effectively functions as a system for supporting diagnosis by the doctor 1 .

However, there are, for example, 30 (60 if interlaced) frames (still images) per second in a moving image of an affected area. Analyzing and evaluating all of these enormous amounts of frames by an AI system results in a very high processing load. Since the processing load is high, for example, the display of the evaluation results is likely to be delayed, or problems may occur during the display. Therefore, from the viewpoint of hardware and software resources, it is difficult to present highly accurate evaluation results by an AI system in real time.

In light of this situation, the present inventors have made intensive studies from the viewpoint of image analysis. It turned out that there is a feature that the images of the images) are very similar to each other. Furthermore, during the study, some of the multiple frames include frames that are out of focus, frames that include motion blur (motion blur) due to movement of the subject and the endoscope, and shadow changes due to those movements, It was found that there are frames in which the image of the affected area is not clear due to evaporation or reflection of the liquid substance present in the affected area. It has been found that it is difficult to obtain highly accurate evaluation results when analyzing such frames by an AI system.

Therefore, the present inventors came to create the embodiments of the present disclosure described below based on the unique knowledge as described above. In the embodiment of the present disclosure, the frames (still images) to be analyzed by the AI system are limited to, for example, frames with appropriate focus and frames without motion blur, rather than all obtained frames. In this way, excluding frames in which the diseased part is unclear from the analysis targets is effective in improving the accuracy of evaluation by the AI system, and is also effective in reducing the amount of data to be processed. Therefore, according to the present embodiment, it is possible to reduce the processing load while improving the accuracy of the evaluation, so that it is possible to display the image of the affected area and present the evaluation result in real time.

That is, in the embodiment of the present disclosure created by the present inventors, the analysis by the AI system includes, for example, the analysis and evaluation of the affected area, such as a frame with appropriate focus (still image) and a frame without motion blur. Use only frames that are appropriate for Furthermore, in this embodiment, the above analysis uses consecutive frames with similar images. By doing so, according to the present embodiment, it is possible to reduce the processing load while improving the accuracy of the evaluation, so that the image of the affected area can be displayed and the evaluation results can be presented in real time. becomes. The details of such embodiments of the present disclosure will be sequentially described below.

<<3. Embodiment>>
<3.1 Medical support system 10>
A configuration of a medical support system 10 according to an embodiment of the present disclosure will be described with reference to FIG. FIG. 2 is a block diagram showing an example of the configuration of the medical support system 10 according to this embodiment.

As shown in FIG. 2, the medical support system 10 according to the embodiment of the present disclosure mainly includes an endoscope 100, a medical support device 200, and a display device 300. The medical support system 10 receives an affected part image 400 captured by the endoscope 100 as an input, derives a score (evaluation value) of the affected part in the medical support apparatus 200, and displays the score (evaluation value) of the affected part to a user (for example, doctor 1) via the display device 300. The score of the affected area can be presented. In this embodiment, the configuration of the medical support system 10 shown in FIG. 2 is an example, and the configuration is not limited to that shown in FIG. Each device included in the medical support system 10 according to the present embodiment will be sequentially described below.

(Endoscope 100)
The endoscope 100 can acquire an affected part image 400 by imaging the inside of the body of the patient 2 . The affected part image 400 is an image including the affected part examined by the doctor 1 . Specifically, the affected area image 400 is an image in which the blood vessel image, unevenness, and ulcer area of the affected area can be visually recognized. The affected area image 400 may be an image of the affected area captured by the endoscope 100 or may be a still image obtained by pausing a moving image continuously captured by the endoscope 100 . In this embodiment, an example of using the endoscope 100 as an imaging device for imaging an evaluation target is shown, but this embodiment is not limited to the endoscope 100, and can be used with an endoscope, a microscope, Any imaging device capable of imaging an object to be evaluated, such as an ultrasonic inspection device, may be used.

Specifically, in the present embodiment, a visible light image such as a white light image obtained by the endoscope 100 may be used as the affected area image 400 of the affected area. White-light images are images that can be captured most easily with the basic functions of endoscopes, so if disease conditions can be evaluated using white-light images, the time, items, and costs of endoscopic examinations can be reduced. be able to.

In addition, when it is difficult to evaluate the disease state with a white light image, the affected area image 400 is captured by irradiating special light, for example, so that the blood vessel region included in the image is emphasized. A special image such as a Band Imaging image or a stained image obtained by indigo carmine staining to emphasize the unevenness of the evaluation target may be used. In this embodiment, depending on the situation, the evaluation accuracy of the affected area can be improved by additionally using these special images in addition to the white light image.

(Medical support device 200)
The medical support apparatus 200 processes an affected area image 400 captured by the endoscope 100 and displays it on a display device 300, which will be described later, and analyzes the affected area image 400 to derive an evaluation result of the disease state of the affected area. and can be performed in parallel. For example, the medical support apparatus 200 subdivides the affected area image 400, derives a score (more specifically, a tile evaluation value) for the subdivided image (more specifically, the tile image 402) (see FIG. 9), and further By deriving the score for the entire image from each of these scores, the evaluation result of the disease state of the affected area may be derived. Specifically, the medical support apparatus 200 includes, for example, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) that perform various types of arithmetic processing, a ROM (Read Only Memory) in which data is stored in advance, and data that is temporarily stored in It is composed of a computer having a RAM (Random Access Memory) for storage. Details of the medical support device 200 will be described later.

(Display device 300)
The display device 300 can present the affected part image 400 and evaluation results (scores, etc.) to the user (for example, the doctor 1). The display device 300 may be, for example, a touch panel display, a three-dimensional display, a spatial display, a projection display, or the like. Specifically, the display device 300 may be any of a CRT (Cathode Ray Tube) display, a liquid crystal display, a plasma display, an EL (Electro Luminescence) display, a laser projector, an LED (Light Emitting Diode) projector, and the like.

By using such a medical support system 10, the doctor 1 can diagnose the state of disease with more certainty by considering the evaluation result of the affected area derived by the medical support system 10 and his own diagnosis result. .

<3.2 Medical support device 200>
Next, an example of the functional configuration of the medical support device 200 according to this embodiment will be described with reference to FIG. Specifically, the medical support apparatus 200 includes an image acquisition unit 202, an image processing unit 204, a determination unit 206, an evaluation execution determination unit 208, an evaluation unit 210, an evaluation value determination unit 212, an output unit 214, and a storage unit 216. Mainly have Each functional unit included in the medical support device 200 according to this embodiment will be described below.

(Image acquisition unit 202)
The image acquisition unit 202 can sequentially acquire a plurality of affected area images 400 (frames) (evaluation target images) obtained by imaging the affected area with the endoscope 100 and output them to the image processing unit 204 described later.

(Image processing unit 204)
The image processing unit 204 performs processing such as image display processing, grayscale conversion processing, reduction processing, binarization processing, and segmentation processing on the affected area image 400 output from the image acquisition unit 202. It can be output to the determination unit 206, the output unit 214, and the storage unit 216, which will be described later. Furthermore, the image processing unit 204 may correct optical distortion in the image or adjust brightness. Details of various processes performed by the image processing unit 204 will be described later.

(Determination unit 206)
The determination unit 206 can determine whether the affected area image 400 output from the image processing unit 204 satisfies image quality conditions for evaluating the affected area (evaluation target) by the evaluation unit 210 described later. Furthermore, the determination unit 206 can link the determination result to the identification information of the affected area image 400, for example, and output it to the evaluation execution determination unit 208, which will be described later. Specifically, the determination unit 206 performs the determination by, for example, comparing an index indicating image quality such as a blur value, motion amount, luminance, and contrast of the affected area image 400 with a predetermined threshold value (first threshold value). be able to.

More specifically, for example, the determination unit 206 classifies the blur value of the diseased part image 400 into three stages, and determines the motion amount of the diseased part image 400 and a predetermined threshold value (first threshold value) according to the classification of the blur value. ), the evaluation unit 210 can determine whether the image quality conditions for evaluating the affected area are satisfied. In this way, in the present embodiment, the affected area image 400 suitable for evaluating the affected area can be evaluated. Therefore, the accuracy of the evaluation can be improved and the processing load on the medical support apparatus 200 can be reduced. can be reduced. Therefore, according to the present embodiment, since evaluation can be performed quickly and accurately, evaluation results can be presented in real time to the doctor 1 or the like who is observing the affected area. Details of the determination performed by the determination unit 206 will be described later.

Here, the blur value means a value obtained by indexing the degree of motion blur occurring in the diseased part image 400 . Also, the amount of motion means the amount of motion (relative position change) of the same subject between successive affected area images (frames) 400 .

(Evaluation execution determination unit 208)
The evaluation execution determination unit 208 determines that the affected area images 400 determined by the determination unit 206 to satisfy the image quality conditions for evaluating the affected area are continuously acquired, and the evaluation unit 210 described later. It is possible to decide whether to perform evaluation of the affected area by Specifically, the evaluation execution determination unit 208 determines the number of consecutive acquisitions of the affected area images 400 determined to satisfy the image quality conditions for evaluating the affected area and a predetermined threshold (second threshold). It is possible to decide whether to perform the evaluation of the affected area based on the result of comparison with. Details of the determination performed by the evaluation execution determination unit 208 will be described later.

(Evaluation unit 210)
The evaluation unit 210 evaluates the affected area based on a plurality of continuous affected area images 400 (frames) determined to satisfy the image quality conditions when the evaluation execution determining unit 208 determines to perform the evaluation of the affected area. , a score (evaluation value) for the affected area and information based on the score can be derived in real time. As described above, this embodiment has the feature that the images of adjacent frames are very similar to each other for a site requiring histological evaluation, such as an affected area of ulcerative colitis. Therefore, the evaluation unit 210 performs evaluation using consecutive frames having similar images (in other words, a plurality of frames acquired before and after). By doing so, in the present embodiment, highly reproducible evaluation can be performed.

In addition, the evaluation unit 210 can derive a score (evaluation value) for the affected area in real time using an algorithm (determiner) obtained by machine learning. In this embodiment, as described above, there are various evaluations and various items as the score for the disease state of the affected area, and the score derived and presented in real time may be a part of them. Furthermore, in the present embodiment, the evaluation unit 210 uses the derived score to calculate a histological (G) score such as a three-level index (for example, “Active” / “Healing” / “Unknown”). can be derived. Then, the evaluation unit 210 can output the derived score or the like to the evaluation value determination unit 212 and the output unit 214, which will be described later.

More specifically, the evaluation unit 210, for example, cuts out the affected area image 400 as a plurality of tile-shaped tile images 402 (see FIG. 9), and calculates a tile evaluation value indicating the evaluation of the affected area in each of the plurality of tile images 402. derive Then, the evaluation unit 210 can select one tile evaluation value from among the plurality of derived tile evaluation values and set it as the score (evaluation value) of the diseased part image 400 . Specifically, for example, the evaluation unit 210 may statistically process a plurality of derived tile evaluation values, and acquire the average value, median value, etc. obtained by the statistical processing as the score of the affected area image 400. can. Furthermore, as described above, the evaluation unit 210 determines that adjacent affected area images 400 (frames) are very similar to each other for areas that require histological evaluation, such as affected areas such as ulcerative colitis. Therefore, the score obtained by statistically processing the scores of a plurality of consecutively obtained frames with similar images is used as a representative score of the group of similar affected area images 400. can be derived.

Also, in the present embodiment, the algorithm (determiner) is generated by machine learning. Specifically, the determiner includes at least one of a large number of affected area images 400, an evaluation result of a doctor 1 (for example, a specialist) based on visual recognition of the affected area image 400, and an evaluation result of a pathological examination by a pathologist. One or more can be generated by machine learning. In machine learning, by learning at least 10,000 or more affected area images 400 and the above-mentioned evaluation results, evaluation (judgment) is performed with an accuracy equivalent to the difference in score between a plurality of doctors 1. It is possible to generate an algorithm (determiner) that can For example, the algorithm (determiner) can be generated by machine learning using a deep neural network or the like.

As explained above, evaluation of ulcerative colitis, etc. by visual recognition of the affected area is highly difficult, so the evaluation by Doctor 1 based on visual confirmation of the affected area varies among doctors 1, even among specialists. According to this embodiment, by using an algorithm (determiner) generated by machine learning, diagnosis and evaluation of diseases such as ulcerative colitis can be performed without a specialist. In addition, by using an algorithm (determiner) generated by machine learning, variations in evaluation between doctors 1 were avoided, and patient 2 was evaluated on a more objective and highly reproducible basis. It is possible to get a score.

In addition, pathological examination may damage patient 2 during tissue collection and may cause complications. Furthermore, since the number of pathologists who perform pathological examinations is small, pathological examinations cannot be performed anywhere, and diagnosis takes time. In addition, additional costs are incurred for the patient 2 by performing the pathological examination. According to this embodiment, by using an algorithm (determiner) generated by machine learning, evaluation results can be derived without tissue sampling, so physical, time, and financial It is possible to reduce the burden of

The details of the evaluation performed by the evaluation unit 210 will be described later.

(Evaluation value determination unit 212)
The evaluation value determination unit 212 compares (statistically processes) a plurality of scores (evaluation values) corresponding to the consecutive affected area images 400 (frames) derived by the evaluation unit 210 described above, and, for example, quartile, standard Detect outliers among multiple scores using deviation, Smirnov-Grubbs test, Dixon test, or the like. Furthermore, the evaluation value determination unit 212 can exclude outliers from the plurality of scores and output the results to the storage unit 216, which will be described later. In this embodiment, by excluding outliers, the quality of the score to be referred to when making a final diagnosis can be improved, and the accuracy of the final diagnosis by the doctor 1 can be further improved. Details of the determination performed by the evaluation value determination unit 212 will be described later.

(Output unit 214)
The output unit 214 can output to the display device 300 in real time the affected area image 400 and information regarding evaluation including a score (evaluation value) and an index based on the score. For example, the output unit 214 can superimpose the score on the affected area image 400 and display the score or the like on the display device 300 in real time. Furthermore, the output unit 214 outputs a predetermined number of top scores set by the user (for example, doctor 1) in a ranking format from a plurality of scores stored in the storage unit 216, which will be described later. The score selected from the ranking and the diseased part image 400 corresponding to the score can be displayed. At this time, the output unit 214 can also display the tile image 402 corresponding to the affected area image 400 with the score selected by the user in color or brightness according to the tile evaluation value of the tile image 402 .

(storage unit 216)
The storage unit 216 stores the score (evaluation value) output from the evaluation value determination unit 212 described above, that is, the score (evaluation value) that is not excluded as an outlier, as the diseased part image 400 corresponding to the score or the identification information of the diseased part image 400. can be stored in association with

Note that in the present embodiment, the functional configuration of the medical support device 200 is not limited to the functional configuration shown in FIG. 2, and may include other functional units, for example.

<3.3 Medical support method>
(work flow)
Next, a medical support method according to an embodiment of the present disclosure will be described with reference to FIG. FIG. 3 is a flow chart showing the flow of operations by a user (for example, the doctor 1, etc.) in the medical support method according to this embodiment. Specifically, as shown in FIG. 3, the user's work in this embodiment can mainly include a plurality of steps from step S1 to step S5. The details of each of these steps according to the present embodiment will be sequentially described below.

First, the doctor 1 inserts the endoscope 100 into the body of the patient 2 (for example, the large intestine, etc.) (step S1).

Next, the doctor 1 advances the distal end of the endoscope 100 while observing the affected area based on the affected area image 400 inside the body of the patient 2 captured by the endoscope 100 displayed on the display device 300 (step S2).

Since the permission value of the affected area is derived in real time based on the affected area image 400 by the medical support apparatus 200 according to the present embodiment, the doctor 1 can see the score (evaluation value) displayed in real time on the display device 300. is confirmed (step S3). At this time, the display device 300 displays, for example, a score derived in real time, or a three-stage index (eg, “Active”/“Healing”/“Unknown”) determined based on the score. 400 is superimposed and displayed.

Further, the doctor 1 confirms a plurality of scores derived from a series of observations, and depending on the situation, confirms all scores such as the stored individual affected area images 400 and their scores (evaluation values), tile evaluation values, etc. (Step S4). At this time, the doctor 1 can specify the affected area image 400, the score, etc. of interest, and confirm the tile image 402 (see FIG. 9), the tile evaluation value, and the like. Furthermore, in this embodiment, each score and tile evaluation value derived in real time can be displayed in ranking after being stored in the storage unit 216 . For example, it is also possible for the doctor 1 to refer to the affected area image 400 and its score and tile evaluation value after the fact by selecting a score from the ranking table.

Then, the doctor 1 makes a diagnosis based on the affected area image 400 and the score confirmed so far (step S5).

(Derivation of score)
Next, a method of deriving a score (evaluation value) performed by the medical support apparatus 200 in step S3 will be described with reference to FIG. FIG. 4 is a flowchart showing a score derivation method performed by the medical support device 200 in the medical support method according to this embodiment. Specifically, as shown in FIG. 4, the score derivation method performed by the medical support device 200 can mainly include a plurality of steps from step S31 to step S39. Details of each of these steps according to the present embodiment will be sequentially described below.

First, the medical support device 200 sequentially acquires a plurality of affected area images 400 (frames) obtained by imaging the affected area with the endoscope 100 (step S31).

Next, the medical support apparatus 200 performs image display processing, grayscale conversion processing, reduction processing, and other processing on the affected area image 400 output from the image acquisition unit 202 (step S32).

Then, the medical support device 200 outputs the affected area image 400 processed in step S32 described above to the display device 300 (step S33).

Next, the medical support apparatus 200 determines whether or not the affected area image 400 processed in step S32 described above satisfies the image quality conditions for evaluating the affected area (step S34). Specifically, the medical support apparatus 200 can make the determination by comparing the blur value and motion amount of the affected area image 400 with a predetermined threshold, for example. Then, the medical support apparatus 200 temporarily stores (buffers) the affected area image 400 that satisfies the image quality conditions for evaluating the affected area, and uses the stored affected area image 400 in subsequent processing. Further, the medical support apparatus 200 counts (accumulates) the number of consecutively acquired affected area images 400 determined to satisfy the image quality conditions for evaluating the affected area.

Then, the medical support apparatus 200 determines whether or not the diseased part images 400 determined to satisfy the image quality conditions for evaluating the diseased part have been obtained n times (predetermined threshold) (step S35). . When the medical support apparatus 200 determines that the affected area images 400 determined to satisfy the image quality conditions for evaluating the affected area have been obtained continuously n times (predetermined threshold value) (step S35: Yes) decides to perform an evaluation of the affected area and proceeds to step S36. On the other hand, when the medical support apparatus 200 determines that the affected area images 400 determined to satisfy the image quality conditions for evaluating the affected area have not been obtained n consecutive times (predetermined threshold) (step S35). : No), the process returns to step S31.

In the present embodiment, for example, the affected area image 400 determined to satisfy the image quality conditions for evaluating the affected area is obtained continuously 8 to 10 times (n times) (predetermined threshold value). In that case, the medical support device 200 decides to perform the evaluation of the affected area. Note that in the present embodiment, the processing of steps S34 and S35 is also processing for determining whether or not to execute the evaluation of the affected area based on the image quality of the frame (affected area image 400), so it is called "skip frame determination". Details of the skip frame determination will be described later.

Next, the medical support apparatus 200 calculates a score (evaluation value ) is derived in real time (step S36). Furthermore, in the present embodiment, the medical support apparatus 200 uses the derived score to calculate a histological (G) score such as a three-level index (for example, “Active”/“Healing”/“Unknown”). can be derived. Details of the evaluation performed in step S36 will be described later.

Then, the medical support apparatus 200 renders the score or the like so as to superimpose the score or the like derived in step S36 on the affected area image 400, and outputs the rendered score or the like to the display device 300 in real time (step S37). Details of the superimposed display executed in step S37 will be described later.

Next, the medical support apparatus 200 compares the score (evaluation value) derived in step S36 described above with the scores of the diseased part images 400 acquired consecutively to the diseased part image 400 corresponding to the score (statistical processing). Then, it is determined whether or not it is an outlier (step S38). If the score is not an outlier (step S38: Yes), the medical support apparatus 200 proceeds to step S39 to store the score in the storage unit 216. FIG. On the other hand, if the score is an outlier (step S38: No), the medical support apparatus 200 excludes the score from the values to be stored, and returns to the process of step S31. In this embodiment, by excluding outliers from being stored, the quality of the score to be referred to when making a final diagnosis can be improved, and the accuracy of the final diagnosis by the doctor 1 can be further improved. The details of the outlier detection performed in step S38 will be described later.

Then, the medical support apparatus 200 uses the score (evaluation value) derived in step S36 described above and not excluded in step S38 as the diseased part image 400 corresponding to the score or the identification information (ID) of the diseased part image 400. ) (step S39). Then, the stored score is used in the final diagnosis by the doctor 1 or the like.

In this embodiment, the steps described with reference to FIG. 4 are not necessarily limited to being processed in the illustrated order. Up to S37 are preferably processed in parallel.

(skip frame determination)
Next, the "skip frame determination" process, which is the process of steps S34 and S35 described above, will be described with reference to FIGS. FIG. 5 is a schematic diagram for explaining an overview of skip frame determination and parallel processing in the derivation method according to this embodiment, and FIG. 6 is a schematic diagram for explaining an example of step determination in skip frame determination according to this embodiment. It is a diagram. Furthermore, FIG. 7 is a schematic diagram for explaining the details of skip frame determination in the derivation method according to this embodiment.

First, as shown in FIG. 5, the medical support apparatus 200 determines whether the affected area image 400 satisfies the image quality conditions for evaluating the affected area (image quality determination). Here, the medical support apparatus 200 uses the blur value and the amount of motion to determine whether image quality conditions for evaluating the affected area are satisfied. Details of the blur value and the amount of motion will be described below.

(bokeh value)
The blur value means a value obtained by indexing the degree of motion blur occurring in the diseased part image 400, as described above. Note that, in the present embodiment, the blur value is also referred to as the F value. Specifically, for example, the affected area image 400 is reduced (resized), the Canny function is applied to the reduced affected area image 400, and a portion with a large luminance difference on the image is detected as a contour, Generate an edge image. Furthermore, the blur value can be obtained by calculating the standard deviation of the edge image. Also, the lower the standard deviation value, the greater the image blur. Therefore, in the present embodiment, in order to reduce the processing load while increasing the evaluation accuracy, the diseased part image 400 with large image blur is excluded from the evaluation target.

(movement amount)
The amount of motion means the amount of motion (relative position change) of the same subject between successive affected area images (frames) 400, as described above. Note that, in the present embodiment, the amount of motion is also referred to as the M value. Specifically, for example, the diseased part image 400 is reduced (resized), and the pixel difference value ( The amount of motion can be obtained by calculating the pixel difference absolute value). Also, the higher the pixel difference value, the larger the motion amount. Therefore, in the present embodiment, in order to reduce the processing load while increasing the evaluation accuracy, the diseased part image 400 with large motion blur (movement amount) is excluded from the evaluation target.

Furthermore, in the present embodiment, the indices used for determining whether the image quality conditions for evaluating the affected area are satisfied are not limited to the above-described blur value and motion amount. For example, the index may be the brightness of the affected area image 400 . More specifically, the medical support apparatus 200 determines the degree of blown-out highlights and blocked-up shadows in the affected area image 400, and based on the determination result, determines whether image quality conditions for evaluating the affected area are satisfied. may be performed. In addition, the medical support apparatus 200 calculates the variance value of the brightness in the time axis direction in the continuous affected area images 400, and based on the calculated variance value, the affected area images 400 with stable brightness changes are used to evaluate the affected area. may be extracted as an image that satisfies the image quality condition of .

Also, in the present embodiment, the contrast ratio (maximum/minimum brightness) of the affected area image 400 may be used as an index used to determine whether the image quality conditions for evaluating the affected area are satisfied.

Furthermore, in the present embodiment, it is preferable to perform stepwise determination in order to improve the accuracy of determining whether an image is appropriate for deriving the score (evaluation value) of ulcerative colitis. For example, as shown in FIG. 6, the blur value (F value) of the diseased part image 400 is divided into three stages, and the motion amount (M value) of the diseased part image 400 and a predetermined threshold value are determined according to the classification of the blur value. By comparing , it is possible to appropriately determine whether or not the image quality conditions for evaluating the affected area are satisfied.

In the present embodiment, for example, as shown in FIG. 6, it is determined whether the blur value (F value) of the diseased part image 400 is classified into one of three levels (upper range, intermediate range, lower range). Which range the blur value falls into can be determined by comparing the blur value with a preset threshold value.

Specifically, it is assumed that the affected area image 400 without blur has a sufficiently high blur value (F value). Therefore, as shown in FIG. 6, when the blur value is classified into the upper range, it is determined that the image quality conditions for evaluating the affected area are satisfied.

In addition, it is assumed that the affected area image 400 located in the boundary area with little blur and small movement is included in the intermediate range in both the blur value (F value) and the amount of motion (M value). Therefore, as shown in FIG. 6, when the blur value is classified into the intermediate range, if the motion amount is classified into the intermediate range, it is determined that the image quality conditions for evaluating the affected area are satisfied. It should be noted that it is possible to determine which range the motion amount falls into by comparing the motion amount with a preset threshold value.

In addition, it is assumed that the affected area image 400 with no texture (no change in texture of the surface of the subject) has a blur value (F value) and a motion amount (M value) included in the lower range. Therefore, as shown in FIG. 6, when the blur value is classified into the lower range, if the motion amount is classified into the lower range, it is determined that the image quality condition for evaluating the affected part is satisfied. It should be noted that it is possible to determine which range the motion amount falls into by comparing the motion amount with a preset threshold value.

It should be noted that, in the present embodiment, by appropriately setting each threshold value described above, it is possible to appropriately extract the part that the doctor 1 wants to determine. However, in the present embodiment, the image quality is affected by the endoscope 100 , so it is preferable that each threshold is set individually for each endoscope 100 .

Specifically, the above threshold can be obtained experimentally. For example, using an actual case image for which evaluation by the doctor 1 is known, characteristics such as an imaging unit (not shown) and a light source (not shown) of the endoscope 100, and image processing by the medical support apparatus 200 are experimentally evaluated. and pre-set thresholds to reflect the characteristics.

Also, the above threshold may be set using a Look Up Table (LUT). For example, the LUT is created as a two-dimensional table that normalizes all patterns consisting of axes of blur value and motion amount. Then, in the LUT, thresholds are set for classification according to the two values of the blur value and the motion amount. Although this method can also deal with local values, errors may occur depending on the fineness of division in the LUT.

Also, the threshold may be set by calibration. For example, using a color chart/resolution chart, various adjustments (calibrations) may be performed with thresholds predetermined by the medical support system 10 so as to obtain evaluation results above a certain level. With this method, since the white level and black level can also be obtained, it is possible to adjust the contrast in addition to the color and resolution.

Then, in the present embodiment, as shown in FIG. 5, whether or not the affected area image 400 determined to satisfy the image quality conditions for evaluating the affected area is obtained n times (predetermined threshold value) consecutively. is determined (evaluation execution determination). After that, in the present embodiment, as shown in FIG. 5, in order to display the score in real time, the evaluation and display processing are performed in parallel. At this time, the derived scores are stored, for example, for about 15 frames in consideration of delay, and are sequentially output in the order in which they are stored (First IN First Out; FIFO) (in FIG. 5, n+1 scores are sequentially stored and output sequentially).

Furthermore, details of the "skip frame determination" process will be described with reference to FIG. First, the medical support apparatus 200 performs grayscale conversion processing on the affected area image 400, and further performs image reduction processing to reduce the affected area image 400 to 1/2. Then, in order to obtain a blur value (F value), the medical support apparatus 200 applies the Canny function to the reduced image 400 of the diseased area, and detects a portion with a large luminance difference on the image as a contour. Edge detection processing is performed. Furthermore, the medical support apparatus 200 calculates the standard deviation of the edge image (standard deviation calculation), and performs normalization processing using the number of pixels.

On the other hand, in order to acquire the amount of motion (M value), the medical support apparatus 200 uses the n-th acquired affected area image 400 that has undergone reduction processing and the (n−1)-th acquired and temporarily stored affected area image 400. A pixel difference value (pixel difference absolute value) between 400 and 400 is calculated (difference value calculation, absolute value calculation), and normalization processing is performed by the number of pixels.

Furthermore, the medical support apparatus 200 performs stepwise threshold determination as described above on the blur value (F value) and the amount of motion (M value) acquired as described above, and the affected area image 400 is used to evaluate the affected area. It is determined whether the image quality condition for performing is satisfied (threshold processing). Subsequently, the medical support apparatus 200 continuously stores the affected area images 400 determined to satisfy the image quality conditions for evaluating the affected area, and accumulates the number (accumulation processing). Then, the medical support apparatus 200 evaluates the affected area when the affected area images 400 determined to satisfy the image quality conditions for evaluating the affected area are obtained n times (predetermined threshold value) consecutively. Judgment to execute (judgement).

(Evaluation by AI system)
Next, with reference to FIGS. 8 and 9, an example of an affected area evaluation method by the AI system in this embodiment will be described. FIG. 8 is a flowchart showing a score (evaluation value) derivation method performed by the medical support device 200 in the medical support method according to this embodiment, and FIG. 9 illustrates an example of the derivation method according to this embodiment. It is a schematic diagram to do. As described above, in this embodiment, the evaluation unit 210 can derive the score for the affected area in real time based on a plurality of successive affected area images 400 . Specifically, as shown in FIG. 8, an example of the score derivation method performed by the medical support device 200 can mainly include a plurality of steps from step S361 to step S364. The details of each of these steps according to the present embodiment will be sequentially described below.

The evaluation unit 210 sequentially acquires a plurality of continuous affected area images 400 determined to satisfy the image quality condition when the evaluation execution determining unit 208 determines to perform the evaluation of the affected area (step S361). ).

Next, the evaluation unit 210 cuts out the affected area image 400 as a tile-shaped tile image 402 having a polygonal shape, as shown in FIG. 9 (step S362). The tile image 402 is preferably sized such that the disease state of the affected area can be determined, which may include, for example, an image of a blood vessel or an object to be examined such as an ulcer. Specifically, the evaluation unit 210 may cut out the affected area image 400 into a rectangular shape such as a square. Alternatively, the evaluation unit 210 may cut out the affected part image 400 so that the circumscribed circle of the quadrangular shape is 5 mm or more and 15 mm or less. Note that in the present embodiment, the shape of the tile image 402 is not limited to a polygon, and may be any shape that can subdivide the diseased part image 400 . Alternatively, the tile image 402 may be cut out in a random shape that differs depending on the location.

Then, the evaluation unit 210 derives a tile evaluation value based on the clipped tile image 402 using an algorithm (determiner) obtained by machine learning (step S363). Here, the tile evaluation value is the score (evaluation value) of the affected area derived for each tile image 402 .

For example, the evaluation unit 210 derives a tile evaluation value for each evaluation item as the tile evaluation value. For example, the evaluation item may be an item of bleeding in an affected area, a tumor, or a fluoroscopic image of a blood vessel, or may be an item of a pathological examination. Also, the evaluation unit 210 may derive a tile evaluation value that is a mixture of evaluation items. For example, a tile evaluation value may be derived using evaluation values of at least two or more of bleeding, ulceration, and fluoroscopic images of the affected area. For example, in this case, the tile evaluation value may be obtained by summing the evaluation values of at least two or more of the bleeding, ulcer, and fluoroscopic images of the affected area.

The evaluation unit 210 derives the overall score (evaluation value) of the affected area image 400 based on the tile evaluation values (step S364). This allows the user (for example, the doctor 1, etc.) to evaluate the entire affected area image 400 rather than a portion of it, thereby enabling more accurate diagnosis of ulcerative disease. Specifically, the score may be derived as an average value obtained by averaging the tile evaluation values of the tile images 402 included in the affected area image 400 . However, when evaluating an evaluation item characterized by the presence or absence of occurrence rather than the degree of a medical condition, such as bleeding, the evaluation unit 210 may derive a score by weighting the bleeding site. In such a case, the evaluation unit 210 may derive a score using the maximum value or the median value of the tile evaluation values instead of or in addition to the average value of the tile evaluation values. In addition, the evaluation unit 210 may further use the probability distribution (variance, standard deviation, etc.) of the tile evaluation values to derive the score.

(Display example)
Next, an example of superimposed display of scores (evaluation values) and the like according to the present embodiment will be described with reference to FIG. 10 . FIG. 10 is a schematic diagram illustrating an example of display in this embodiment. Specifically, as shown in FIG. 10, in the present embodiment, the display device 300 superimposes the score 500 of the affected area image 400 on the affected area image 400 and displays the score 500 in real time. By doing so, in the present embodiment, at the same timing as the clinical findings by visual recognition (camera work) of the state of the affected area by the doctor 1, the AI system presents highly accurate evaluation results. Usability is ensured, and diagnosis support by the doctor 1 can be effectively performed. Furthermore, in the present embodiment, the diseased part image 400 and the score 500 are superimposed and displayed, so that the doctor 1 can easily visually recognize both the diseased part image 400 and the score 500 without moving the line of sight. .

In addition, in the present embodiment, the derived score (evaluation value) 500 is used to derive a three-level index (for example, "Active" / "Healing" / "Unknown"), etc., instead of the score 500, It may be superimposed on the affected part image 400 . Furthermore, in this embodiment, the whole or part of the affected area image 400 may be displayed with color or brightness according to the derived score 500 . Alternatively, in the present embodiment, the score 500 superimposed on the affected area image 400 may have color, brightness, character type, size, or thickness corresponding to the score 500 .

Note that this embodiment is not limited to the form of superimposed display as shown in FIG.

(outlier detection)
Next, the outlier detection process, which is the process of step S36 described above, will be described. In the present embodiment, by excluding outlier scores (evaluation values) from the storage targets of the storage unit 216, the quality of the scores referred to when making a final diagnosis is improved, and the final diagnosis by the doctor 1 is improved. Accuracy can be further improved.

For example, in the present embodiment, outliers are detected using quartiles from a plurality of scores (evaluation values) 500 of the affected area images 400 obtained in succession, and the detected outliers are counted as a plurality of scores 500 , and the remaining top scores 500 are stored as scores 500 to be used in the final diagnosis. Specifically, in this embodiment, the multiple scores 500 are rearranged in descending order, and the rearranged multiple scores 500 are divided into four. Then, the second delimiter position (50th percentile (median value)) is extracted from the large number side, and the other scores of 500 are detected as outliers.

In addition, as a result of experimental verification by the present inventors, the score (evaluation value) 500 larger than the position of the first delimiter from the large number side (75th percentile (third quartile)) is regarded as an outlier Detecting a score of 500 that is greater than the position of the second delimiter (50th percentile (median)) as an outlier than detecting it is a score that is closer to the evaluation result based on clinical findings by Doctor 1. I know I can get 500. Further, in the present embodiment, it is preferable to appropriately select which range is to be detected as an outlier according to the number of obtained scores 500 and the statistical processing method thereof. In addition, outlier detection using quartiles can obtain relatively good accuracy even with a small amount of data compared to outlier detection using variance and standard deviation. It is said that there is However, in this embodiment, in order to avoid a situation in which the number of data is extremely small, the diseased part image 400 corresponding to the target score 500 is replaced with about 150 frames of the diseased part image 400 acquired before and after the affected part image 400. Outlier detection is preferably performed with the corresponding score 500 .

Further, in the present embodiment, it is not limited to the detection of outliers using quartiles. For example, Smirnov-Grubbs test, Dixon test, deviation and standard deviation using the test, etc. can be used. can. For example, the Smirnov-Grubbs test and Dixon test are tests that detect outliers on the premise that the data follow a normal distribution, and the test using the deviation and standard deviation is the value calculated using the deviation and standard deviation. A test that detects outliers by comparison with significant points.

(ranking display)
In this embodiment, in steps S4 and S5 previously described with reference to FIG. 3, the stored scores (evaluation values) 500 and the like can be displayed in ranking order for final diagnosis by the doctor 1. FIG. Therefore, the details of the ranking display of the score 500 will be described with reference to FIGS. 11 and 12. FIG. FIG. 11 is a flowchart showing ranking display performed by the medical support device 200 in the medical support method according to this embodiment, and FIG. 12 is a schematic diagram illustrating an example of display in this embodiment.

In the evaluation based on clinical findings by Doctor 1, it is common to extract and consider the top five scores (evaluation values) 500. However, in this embodiment, it is preferable to extract and display the top 5 to 20 scores 500 in consideration of over-detection, erroneous detection due to forceps or NBI images, and the like. For example, in cases where diagnosis is difficult, a large number of scores 500 are extracted and displayed to effectively assist the diagnosis of the doctor 1 .

In the present embodiment, outlier scores (evaluation values) 500 are excluded by outlier detection as described above. The number of top scores 500 are displayed in a ranking format. Note that, in the present embodiment, based on the above-described amount of motion (M value), the affected area images 400 (frames) in which the amount of motion suddenly increases are separated, and the partition is defined as a change point (scene change) in observation of the affected area. may be regarded as In this case, the highest score in a plurality of affected area images 400 included in one observation section (scene) may be regarded as a representative evaluation value, and the top representative evaluation values may be displayed in a ranking format. By doing so, it is possible to easily grasp the overall case.

Next, the ranking display processing according to the embodiment of the present disclosure will be described with reference to FIGS. 11 and 12. FIG. As shown in FIG. 11, the ranking display according to this embodiment can mainly include a plurality of steps from step S41 to step S44. The details of each of these steps according to the present embodiment will be sequentially described below. Here, the top scores 500 from a plurality of scores (evaluation values) 500 generated in real time are displayed in a ranking format. Furthermore, when the doctor 1 selects one score 500 from the ranking display, all scores such as the affected part image 400 corresponding to the score 500, the tile image 402, and the tile evaluation value are displayed. Therefore, the doctor 1 can confirm the diagnosis based on various data regarding the score 500 selected by him/herself.

First, the medical support apparatus 200 selects the top scores 500 by a predetermined number from among the plurality of stored scores 500 from which outlier scores (evaluation values) 500 are excluded, in descending order. Displayed in a ranking format (step S41). For example, the display device 300 can display a ranking table 502, as shown in FIG. Note that the ranking table 502 may have, for example, a cursor 600 or the like for moving the displayed score 500 up and down.

Next, the medical support device 200 receives a selection input of the score (evaluation value) 500 by the doctor 1 (step S42). For example, the doctor 1 can perform a selection input by designating a selection area 504 on the ranking table 502 shown in FIG.

Then, the medical support apparatus 200 acquires data such as the affected area image 400, the tile image 402, and the tile evaluation value corresponding to the selected score (evaluation value) 500 from the storage unit 216 (step S43).

Further, the medical support apparatus 200 displays the affected area image 400, the tile image 402, the score (evaluation value), and all scores including the tile evaluation value obtained in step S43 (step S44). For example, as shown in FIG. 12, the display device 300 displays a diseased part image 400 corresponding to the selected score 500 and a collective image 404 in which tile images 402 obtained by cutting out the diseased part image 400 into tiles are aggregated. You may At this time, the collective image 404 may be an image in which the tile images 402 having different colors according to the tile evaluation values are collected. Further, the collective image 404 may be displayed together with the collective images 404 of the lesion images 400 acquired before and after it. In addition, for example, the display device 300 may also display a cursor 600 to chronologically change the collective image 404 to be displayed. In this case, the doctor 1 can change the collective image 404 to be displayed in time series by moving the cursor 600 left and right.

In the present embodiment, as described above, adjacent affected area images 400 are very similar to each other. may be regarded as a change point (scene change) in observation of the affected area. In this case, one affected area image 400 may be selected from among the plurality of affected area images 400 included in one observation section (scene) and displayed as the affected area image 400 representing the observation section (scene). . By doing so, it is possible to easily grasp the overall case.

It should be noted that the present embodiment is not limited to the form of display as shown in FIG.

<3.4 Examples>
In order to verify the effect of the embodiment of the present disclosure described above, from April 2020 to March 2021, ulcerative colitis patients (a total of 770 people) were subjected to endoscope 100 using this embodiment. We automatically selected the inflamed area from the entire movie and derived the endoscopic score. An endoscopist collected mucosal biopsies from five regions of the large intestine (ascending colon, transverse colon, descending colon, sigmoid colon, and rectum), performed pathological evaluation, and compared the scores according to the present embodiment. Additionally, a UCEIS score was determined for each patient by an expert physician and compared to the score according to the present embodiment.

In addition, in order to verify the benefit of the embodiment of the present disclosure, from April 2018 to April 2019, patients with ulcerative colitis (875 people in total) were calculated using this embodiment Scores were compared to patient prognosis (1 year later) and expert predictions.

As a result, according to this embodiment, it was found that the presence or absence of histological inflammation could be evaluated in real time in 81.0% of all patients compared with the results of pathological evaluation. Furthermore, the precision for histological remission was 97.9% sensitive and 94.6% specific. The intraclass correlation coefficient between the scores according to the present embodiment and the expert scores for the UCEIS score is 0.927, indicating that the scores according to the present embodiment have similar accuracy to the expert scores. rice field. It is said that if the intraclass correlation coefficient is 0.85 or more, it can be regarded as clinically consistent.

In addition, regarding prognosis prediction after one year, the prediction ability (evaluated by hazard ratio) according to the present embodiment for the above patients was hospitalization 48.4, surgery 46.4, steroid use 10.2, relapse 8.8. rice field. In contrast, specialist predictions were 44.7 hospitalizations, 43.0 surgeries, 13.4 steroid use, and 9.5 relapses. Summarizing these results, it was found that there was no significant difference in prognosis prediction ability between the score according to this embodiment and the score by the specialist, and that the score was as accurate as the patient prognosis prediction by the specialist.

From the above, according to the present embodiment, it is possible to consistently derive an accurate endoscopic score, and to evaluate the presence or absence of histological inflammation in real time. It can be said that it is possible to reduce the number of tests.

<3.5 Application>
(Application to other diseases)
The present embodiment is not limited to application to ulcerative colitis, and can be applied to examinations and diagnoses that require real-time endoscopic examination (oral, transnasal, large intestine). . Specifically, in this embodiment, scirrhous gastric cancer (cancer that progresses as it permeates the stomach wall and tissues, and the stomach becomes hard and thick), early esophageal cancer, early colon cancer, Crohn's disease (a type of inflammatory bowel disease, a chronic disease of unknown cause in which erosion and ulcers occur due to inflammation occurring mainly in the digestive tract, such as the small intestine and large intestine).

(Application to other medical images)
Further, the present embodiment is not limited to application to endoscopic images and the like, and can also be applied to ultrasound images, for example. An ultrasound image is an image acquired by the doctor 1 or the like while moving the head portion of the probe, and is required to be an appropriate contrast image with high visibility for highly accurate diagnosis.

For example, an ultrasound image 410 is shown in FIG. 13, which is a schematic diagram illustrating an example of an ultrasound image. Since such an ultrasound image 410 is acquired while the doctor 1 or the like moves the head portion of the probe, it is considered effective to have a real-time property. For example, when the head portion is swayed sideways or when the patient 2 is taking a deep breath, large blurring (motion blur) is likely to occur in the ultrasonic image 410, and an image suitable for diagnosis cannot be obtained. difficult to obtain.

Therefore, by applying the present embodiment to the ultrasound image 410, it is possible to extract only images suitable for diagnosis and analyze only the extracted images, thereby enabling accurate evaluation in real time. becomes.

<<4. Summary>>
As described above, in the embodiment of the present disclosure, for analysis by the AI system, for example, an affected area image 400 with appropriate focus, an affected area image 400 without motion blur, and other affected area images suitable for analysis and evaluation of the affected area Try to use only 400. Furthermore, in the present embodiment, the analysis uses consecutive affected area images 400 having similar images. By doing so, according to the present embodiment, it is possible to reduce the processing load while improving the accuracy of the evaluation, so that the evaluation result can be presented in real time together with the display of the affected area image 400. .

As a result, according to the present embodiment, at the same timing as clinical findings based on the visual recognition (camera work) of the state of the affected area by the doctor 1, highly accurate evaluation results are presented by the AI system, resulting in high usability. This ensures that the doctor 1 can effectively assist diagnosis.

The medical support system 10 according to the present embodiment aims to assist the diagnosis of the doctor 1, but in the future, various conditions (legal development, etc.) will be cleared, and the accuracy of evaluation, etc. will be improved. Provided that it can also function as a diagnostic system that automatically performs diagnostics.

In addition, the above-described embodiments of the present disclosure are not limited to application to medical applications, and can also be applied to applications such as academic research. is not particularly limited as long as the application requires performing analysis and displaying the analysis results in real time.

<<5. Hardware configuration >>
Also, the medical support apparatus 200 according to the above-described embodiments is implemented by, for example, a computer 1000 configured as shown in FIG. FIG. 14 is a hardware configuration diagram showing an example of a computer that realizes the functions of the medical support device 200. As shown in FIG. The computer 1000 includes a CPU (Central Processing Unit) 1100, a RAM (Random Access Memory) 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface (I/F) 1500, an input/output interface (I /F) 1600 and a media interface (I/F) 1700 .

The CPU 1100 (including the Graphics Processing Unit (GPU)) operates based on programs stored in the ROM 1300 or HDD 1400 and controls each part. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, a program depending on the hardware of the computer 1000, and the like.

The HDD 1400 stores programs executed by the CPU 1100 and data used by these programs. Communication interface 1500 receives data from another device via a predetermined communication network, sends the data to CPU 1100, and transmits data generated by CPU 1100 to another device via a predetermined communication network.

The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice, via the input/output interface 1600 . CPU 1100 acquires data from an input device via input/output interface 1600 . CPU 1100 also outputs the generated data to an output device via input/output interface 1600 .

The media interface 1700 reads programs or data stored in the recording medium 1800 and provides them to the CPU 1100 via the RAM 1200. CPU 1100 loads such a program from recording medium 1800 onto RAM 1200 via media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase-change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor. memory and the like.

For example, when the computer 1000 functions as the medical support device 200 according to the embodiment of the present disclosure, the CPU 1100 of the computer 1000 executes a program loaded on the RAM 1200 to obtain the image acquisition unit 202, the image processing unit 204, It realizes the functions of the determination unit 206, the evaluation execution determination unit 208, the evaluation unit 210, the evaluation value determination unit 212, the output unit 214, and the like. CPU 1100 of computer 1000 reads these programs from recording medium 1800 and executes them, but as another example, these programs may be obtained from another device via a predetermined communication network. Also, the HDD 1400 stores programs, data, and the like according to the embodiment of the present disclosure.

<<6. Supplement >>
The above-described embodiment of the present disclosure includes, for example, a medical support method executed by the medical support system 10 as described above, a program for operating the medical support system 10, and a program in which the program is recorded. May include non-transitory tangible media. Also, the program may be distributed via a communication line (including wireless communication) such as the Internet.

Also, each step in the processing of the embodiment of the present disclosure described above does not necessarily have to be processed in the described order. For example, each step may be processed in an appropriately changed order. Also, each step may be partially processed in parallel or individually instead of being processed in chronological order. Furthermore, the processing of each step does not necessarily have to be processed in accordance with the described method, and may be processed by another method by another functional unit, for example.

Of the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all of the processes described as being performed manually Alternatively, some can be done automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

Also, the effects described in this specification are merely descriptive or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification, in addition to or instead of the above effects.

Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modifications or modifications within the scope of the technical idea described in the claims. are naturally within the technical scope of the present disclosure.

Note that the present technology can also take the following configuration.
(1)
an information processing device;
a program for causing the information processing device to execute information processing related to medical support;
including,
A medical support system,
The information processing device, according to the program,
an image acquisition unit that sequentially acquires a plurality of images of the affected area obtained by imaging the affected area;
a determination unit that determines whether the acquired affected area image satisfies image quality conditions for evaluating the affected area;
an evaluation execution determination unit that determines whether or not to perform evaluation of the affected area based on successive acquisition of the affected area images determined to satisfy the image quality condition;
an evaluation unit that derives information regarding an evaluation value for the affected area based on the plurality of affected area images determined to satisfy the image quality condition when execution of evaluation of the affected area is determined;
an output unit that outputs information about the evaluation value;
functioning as
Medical support system.
(2)
The medical support system according to (1) above, wherein the output unit outputs information about the evaluation value in real time together with the affected area image.
(3)
The medical support system according to (1) or (2) above, wherein the determination unit performs the determination based on a comparison result between an index indicating image quality of the affected area image and a first threshold value.
(4)
The determination unit according to (3) above, wherein the determination is performed using at least one index of blur value, motion amount, brightness, and contrast of the affected area image as an index indicating image quality of the affected area image. Medical support system.
(5)
The determination unit is
classifying the blur value of the affected area image into three stages according to the value;
comparing the motion amount of the affected area image with the first threshold according to the classification of the blur value;
The medical support system according to (4) above.
(6)
The evaluation execution determination unit determines whether or not the evaluation of the affected area can be performed based on a comparison result between the number of consecutively acquired affected area images determined to satisfy the image quality condition and a second threshold. The medical support system according to any one of (1) to (5) above, which determines the
(7)
Any of the above (1) to (6), wherein the evaluation unit analyzes a plurality of consecutively acquired affected area images using an algorithm obtained by machine learning to derive an evaluation value for the affected area. or the medical support system according to one.
(8)
The evaluation unit
cutting out the affected area image as a plurality of tiled tile images;
deriving a tile evaluation value indicating an evaluation of the affected area in each of the plurality of tile images;
The medical support system according to (7) above.
(9)
The medical support system according to (8) above, wherein the evaluation unit selects one of the derived tile evaluation values as the evaluation value.
(10)
The medical support system according to (8) above, wherein the evaluation unit obtains the evaluation value by statistically processing the plurality of derived tile evaluation values.
(11)
The medical support system according to any one of (8) to (10) above, wherein the output unit displays the evaluation value or an index based on the evaluation value superimposed on the affected area image in real time.
(12)
an evaluation value determination unit that detects outliers based on statistical processing from among the plurality of derived evaluation values and excludes the detected outliers from the plurality of evaluation values;
a storage unit that stores the plurality of non-excluded evaluation values in association with identification information of the affected area image corresponding to each of the evaluation values;
further comprising
The medical support system according to any one of (8) to (11) above.
(13)
The medical support system according to (12) above, wherein the evaluation value determination unit detects the outlier using one of quartiles, standard deviation, Smirnov-Grubbs test, and Dixon test.
(14)
The medical support system according to (13) above, wherein the output unit outputs a predetermined number of high-ranking evaluation values from the plurality of evaluation values stored in the storage unit in a ranking format.
(15)
The medical support system according to (14) above, wherein the output unit displays the evaluation value selected by the user from the ranking and the affected part image corresponding to the evaluation value.
(16)
The output unit further displays the tile image corresponding to the affected area image together with the affected area image of the evaluation value selected by the user from the ranking.
The medical support system according to (15) above.
(17)
The medical support system according to (16) above, wherein the tile image has a color or brightness corresponding to the tile evaluation value.
(18)
The medical support system according to any one of (1) to (17) above, wherein the image of the affected area is obtained from any one of an endoscope, an endoscope, a microscope, and an ultrasonic examination apparatus. .
(19)
the processor
Sequentially acquiring a plurality of images of the affected area obtained by imaging the affected area;
Determining whether the acquired affected area image satisfies image quality conditions for evaluating the affected area;
Determining whether or not to perform the evaluation of the affected area based on the continuous acquisition of the affected area images determined to satisfy the image quality condition;
deriving information about an evaluation value for the affected area based on the plurality of affected area images determined to satisfy the image quality condition when execution of evaluation of the affected area is determined;
outputting information about the evaluation value;
medical assistance methods, including;
(20)
the computer,
an image acquisition unit that sequentially acquires a plurality of images of the affected area obtained by imaging the affected area;
a determination unit that determines whether the acquired affected area image satisfies image quality conditions for evaluating the affected area;
an evaluation execution determination unit that determines whether or not to perform evaluation of the affected area based on successive acquisition of the affected area images determined to satisfy the image quality condition;
an evaluation unit that derives information regarding an evaluation value for the affected area based on the plurality of affected area images determined to satisfy the image quality condition when execution of evaluation of the affected area is determined;
an output unit that outputs information about the evaluation value;
function as
program.
(21)
an image acquisition unit that sequentially acquires a plurality of evaluation target images obtained by imaging an evaluation target;
a determination unit that determines whether the acquired evaluation target image satisfies an image quality condition for evaluating the evaluation target;
an evaluation execution determination unit that determines whether or not to execute the evaluation of the evaluation target based on the continuous acquisition of the evaluation target images determined to satisfy the image quality condition;
an evaluation unit that derives information about an evaluation value for the evaluation target based on the plurality of evaluation target images determined to satisfy the image quality condition when execution of evaluation of the evaluation target is determined;
an output unit that outputs information about the evaluation value together with the evaluation target image;
comprising
Evaluation support device.
(22)
an information processing device;
a program for causing the information processing device to execute information processing related to evaluation support;
including,
An evaluation support system,
The information processing device, according to the program,
an image acquisition unit that sequentially acquires a plurality of evaluation target images obtained by imaging an evaluation target;
a determination unit that determines whether the acquired evaluation target image satisfies an image quality condition for evaluating the evaluation target;
an evaluation execution determination unit that determines whether or not to execute the evaluation of the evaluation target based on the continuous acquisition of the evaluation target images determined to satisfy the image quality condition;
an evaluation unit that derives information about an evaluation value for the evaluation target based on the plurality of evaluation target images determined to satisfy the image quality condition when execution of evaluation of the evaluation target is determined;
an output unit that outputs information about the evaluation value together with the evaluation target image;
functioning as
Evaluation support system.
(23)
the processor
Sequentially acquiring a plurality of evaluation target images obtained by imaging the evaluation target;
Determining whether the acquired evaluation target image satisfies an image quality condition for evaluating the evaluation target;
Determining whether or not to execute the evaluation of the evaluation target based on the continuous acquisition of the evaluation target images determined to satisfy the image quality condition;
deriving information about an evaluation value for the evaluation target based on the plurality of evaluation target images determined to satisfy the image quality condition when execution of evaluation of the evaluation target is determined;
outputting information about the evaluation value together with the evaluation target image;
evaluation support methods, including;
(24)
the computer,
an image acquisition unit that sequentially acquires a plurality of evaluation target images obtained by imaging an evaluation target;
a determination unit that determines whether the acquired evaluation target image satisfies an image quality condition for evaluating the evaluation target;
an evaluation execution determination unit that determines whether or not to execute the evaluation of the evaluation target based on the continuous acquisition of the evaluation target images determined to satisfy the image quality condition;
an evaluation unit that derives information about an evaluation value for the evaluation target based on the plurality of evaluation target images determined to satisfy the image quality condition when execution of evaluation of the evaluation target is determined;
an output unit that outputs information about the evaluation value together with the evaluation target image;
function as
program.

1 doctor 2 patient 10 medical support system 100 endoscope 200 medical support device 202 image acquisition unit 204 image processing unit 206 determination unit 208 evaluation execution determination unit 210 evaluation unit 212 evaluation value determination unit 214 output unit 216 storage unit 300 display device 400 Affected area image 402 Tile image 404 Group image 410 Ultrasound image 500 Score (evaluation value)
502 Ranking table 504 Selection area 600 Cursor

Claims

an information processing device;
a program for causing the information processing device to execute information processing related to medical support;
including,
A medical support system,
The information processing device, according to the program,
an image acquisition unit that sequentially acquires a plurality of images of the affected area obtained by imaging the affected area;
a determination unit that determines whether the acquired affected area image satisfies image quality conditions for evaluating the affected area;
an evaluation execution determination unit that determines whether or not to perform evaluation of the affected area based on successive acquisition of the affected area images determined to satisfy the image quality condition;
an evaluation unit that derives information regarding an evaluation value for the affected area based on the plurality of affected area images determined to satisfy the image quality condition when execution of evaluation of the affected area is determined;
an output unit that outputs information about the evaluation value;
functioning as
Medical support system.
The medical support system according to claim 1, wherein the output unit outputs information about the evaluation value in real time together with the affected area image.
The medical support system according to claim 1, wherein the determination unit makes a determination based on a comparison result between an index indicating image quality of the affected area image and a first threshold value.
4. The medical treatment according to claim 3, wherein the determination unit performs determination using at least one index of blur value, motion amount, brightness, and contrast of the affected area image as an index indicating image quality of the affected area image. support system.
The determination unit is
classifying the blur value of the affected area image into three stages according to the value;
comparing the motion amount of the affected area image with the first threshold according to the classification of the blur value;
The medical support system according to claim 4.
The evaluation execution determination unit determines whether or not the evaluation of the affected area can be performed based on a comparison result between the number of consecutively acquired affected area images determined to satisfy the image quality condition and a second threshold. 2. The medical support system according to claim 1, wherein the system determines
The medical support system according to claim 1, wherein the evaluation unit uses an algorithm obtained by machine learning to analyze a plurality of consecutively acquired images of the affected area to derive an evaluation value for the affected area.
The evaluation unit
cutting out the affected area image as a plurality of tiled tile images;
deriving a tile evaluation value indicating an evaluation of the affected area in each of the plurality of tile images;
The medical support system according to claim 7.
The medical support system according to claim 8, wherein the evaluation unit selects one of the derived tile evaluation values as the evaluation value.
The medical support system according to claim 8, wherein the evaluation unit obtains the evaluation values by statistically processing the derived tile evaluation values.
The medical support system according to claim 8, wherein the output unit superimposes the evaluation value or an index based on the evaluation value on the affected area image in real time.
an evaluation value determination unit that detects outliers based on statistical processing from among the plurality of derived evaluation values and excludes the detected outliers from the plurality of evaluation values;
a storage unit that stores the plurality of non-excluded evaluation values in association with identification information of the affected area image corresponding to each of the evaluation values;
further comprising
The medical support system according to claim 8.
13. The medical support system according to claim 12, wherein said evaluation value determination unit detects said outlier using one of quartiles, standard deviation, Smirnov-Grubbs test, and Dixon test.
14. The medical support system according to claim 13, wherein the output unit outputs a predetermined number of high ranking evaluation values from the plurality of evaluation values stored in the storage unit in a ranking format.
The medical support system according to claim 14, wherein the output unit displays the evaluation value selected by the user from the ranking and the affected area image corresponding to the evaluation value.
The output unit further displays the tile image corresponding to the affected area image together with the affected area image of the evaluation value selected by the user from the ranking.
The medical support system according to claim 15.
The medical support system according to claim 16, wherein said tile image has color or brightness according to said tile evaluation value.
　The medical support system according to claim 1, wherein the affected part image is obtained from any one of an endoscope, an endoscope, a microscope, and an ultrasonic examination apparatus.
the processor
Sequentially acquiring a plurality of images of the affected area obtained by imaging the affected area;
Determining whether the acquired affected area image satisfies image quality conditions for evaluating the affected area;
Determining whether or not to perform the evaluation of the affected area based on the continuous acquisition of the affected area images determined to satisfy the image quality condition;
deriving information about an evaluation value for the affected area based on the plurality of affected area images determined to satisfy the image quality condition when execution of evaluation of the affected area is determined;
outputting information about the evaluation value;
medical assistance methods, including;
an image acquisition unit that sequentially acquires a plurality of evaluation target images obtained by imaging an evaluation target;
a determination unit that determines whether the acquired evaluation target image satisfies an image quality condition for evaluating the evaluation target;
an evaluation execution determination unit that determines whether or not to execute the evaluation of the evaluation target based on the continuous acquisition of the evaluation target images determined to satisfy the image quality condition;
an evaluation unit that derives information about an evaluation value for the evaluation target based on the plurality of evaluation target images determined to satisfy the image quality condition when execution of evaluation of the evaluation target is determined;
an output unit that outputs information about the evaluation value together with the evaluation target image;
comprising
Evaluation support device.