JP2023106327A - Inspection support device, inspection support method, and inspection support program - Google Patents

Abstract

To provide an inspection support device which even when connected to an endoscope system of a different model, performs accurate diagnosis by using an image output by each endoscope system, and supports a decision made by a medical doctor.SOLUTION: The inspection support device, which is used in connection with an endoscope system provided with a camera unit inserted into the body of a subject, comprises: an information acquisition unit which acquires model information of the endoscope system; an image acquisition unit which acquires a captured image captured by the camera unit; a diagnosis unit which generates input data on the basis of the model information acquired by the information acquisition unit and the captured image acquired by the image acquisition unit and inputs the data to a learned model, thereby diagnosing the inside of the body; and an output unit which outputs a result of the diagnosis by the diagnosis unit.SELECTED DRAWING: Figure 3

Description

The present invention relates to an examination support device, an examination support method, and an examination support program.

2. Description of the Related Art An endoscope system is known that takes an image of, for example, the inside of a subject's stomach using an endoscope and displays the image on a monitor. Recently, an examination support device that analyzes an image captured by an endoscope system and informs a doctor of the result is becoming popular (see, for example, Patent Document 1). In addition, a mode of use in which an examination support device is connected to an endoscope system and an in-vivo image captured by the endoscope system is analyzed in real time by the examination support device is also reaching the stage of practical use.

JP 2019-42156 A

It is preferable that the examination support device used by being connected to the endoscope system is designed so that it can be connected to a plurality of types of endoscope systems. In this case, not only can it be connected to multiple models of endoscope systems manufactured by the same manufacturer, but it may also be designed to be connectable to different models of endoscope systems manufactured by different manufacturers. obtain.

An endoscope system takes an image of an inspection site inside a subject's body, processes the imaged signal, and generates image data. Therefore, if the model is different, image data corresponding to each characteristic is generated. For example, when the same inspection site of the same subject is imaged by different models of endoscope systems, the images of the image data generated by each may differ from each other in color tone, sharpness, distortion, noise, and the like. If the examination support device receives such an image and inputs it directly into the previous learned model to perform in-vivo diagnosis, the diagnosis results may differ even for the same examination site of the same subject. There was a problem.

The present invention has been made to solve such problems, and even when connecting to endoscope systems of different models, the images output by each endoscope system can be used to improve accuracy. The purpose of the present invention is to provide an examination support device or the like that makes a highly accurate diagnosis and supports a doctor's judgment.

An examination support apparatus according to a first aspect of the present invention is an examination support apparatus that is used by being connected to an endoscope system having a camera unit that is inserted into the body of a subject. An information acquisition unit that acquires information, an image acquisition unit that acquires a captured image captured by the camera unit, and input data is generated based on the model information acquired by the information acquisition unit and the captured image acquired by the image acquisition unit. and a diagnostic unit for diagnosing the inside of the body by inputting it to the trained model, and an output unit for outputting the diagnostic result of the diagnostic unit.

An examination support method according to a second aspect of the present invention is an examination support method using an examination support apparatus that is used in connection with an endoscope system having a camera unit that is inserted into the body of a subject, An information acquisition step of acquiring model information of an endoscope system, an image acquisition step of acquiring a captured image captured by a camera unit, and the model information acquired in the information acquisition step and the captured image acquired in the image acquisition step. A diagnosis step for diagnosing the inside of the body by generating input data based on the model and inputting it to the trained model, and an output step for outputting the diagnosis result of the diagnosis step.

An examination support program according to a third aspect of the present invention is an examination support program for controlling an examination support apparatus used in connection with an endoscope system having a camera unit inserted into the body of a subject, An information acquisition step of acquiring model information of an endoscope system, an image acquisition step of acquiring a captured image captured by a camera unit, and the model information acquired in the information acquisition step and the captured image acquired in the image acquisition step. The computer is caused to execute a diagnosis step of diagnosing the inside of the body by generating input data based on the generated data and inputting the data to the learned model, and an output step of outputting the diagnosis result of the diagnosis step.

According to the present invention, even when connected to endoscope systems of different models, the examination support apparatus performs more accurate diagnosis using the images output by each endoscope system and assists the doctor's decision. etc. can be provided.

It is a figure which shows the state of the endoscopy using an endoscope system and the examination assistance apparatus which concerns on this embodiment. 2 is a hardware configuration diagram of an inspection support device; FIG. It is a figure explaining the process until it displays a diagnosis result. FIG. 4 is a diagram illustrating learning work for generating a neural network for analysis; It is a figure which shows the example of the setting screen regarding model information. FIG. 5 is a flow diagram for explaining the processing procedure of the arithmetic processing unit; FIG. 10 is a diagram illustrating learning work for generating an analysis neural network in the first modified example; It is a figure explaining the process until the diagnosis result is displayed in a 1st modification. FIG. 11 is a flow diagram illustrating a processing procedure of an arithmetic processing unit in the first modified example; FIG. 10 is a diagram illustrating learning work for generating a neural network for analysis in the reference example; FIG. 10 is a diagram showing an example of a setting screen regarding past history information in the reference example; FIG. 11 is a diagram illustrating learning work for generating an analysis neural network in the second modified example; It is a figure explaining the process until the diagnosis result is displayed in a 2nd modification. FIG. 11 is a flow diagram illustrating a processing procedure of an arithmetic processing unit in a second modified example;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, the present invention will be described through embodiments of the invention, but the invention according to the scope of claims is not limited to the following embodiments. Moreover, not all the configurations described in the embodiments are essential as means for solving the problems.

FIG. 1 is a diagram showing an endoscopy using an endoscope system 200 and an examination support device 100 according to this embodiment. Both the endoscope system 200 and the examination support apparatus 100 are installed in the examination space. The endoscope system 200 has a camera unit 210 , a system body 211 and a system monitor 220 . As illustrated, the camera unit 210 is inserted through the oral cavity of a lying subject into an internal organ such as the stomach, and transmits an imaging signal obtained by imaging the inside of the organ to the system main body 211 . The insertion of the camera unit 210 into the organ and the imaging operation are performed by a doctor. In this embodiment, the camera unit 210 is detachable from the system body 211 .

The system body 211 processes the imaging signal sent from the camera unit 210 to generate image data. Then, the image data is adjusted to a visible display signal, and displayed as a captured image 221 on a system monitor 220 configured by, for example, a liquid crystal panel. Further, the system main body 211 displays examination information 222 including subject information, camera information of the camera unit 210 and the like on the system monitor 220 .

The examination support device 100 is connected to the endoscope system 200 by a connection cable 250 . The endoscope system 200 also transmits the display signal to be transmitted to the system monitor 220 to the examination support apparatus 100 via the connection cable 250 . That is, the display signal in this embodiment is an example of an image signal that the endoscope system 200 provides to an external device. The examination support apparatus 100 has an apparatus main body 101 and a display monitor 120 . The device main body 101 extracts an image signal corresponding to the captured image 221 from the display signal sent from the endoscope system 200, and displays it as the captured image 121 on the display monitor 120 configured by, for example, a liquid crystal panel.

A doctor who is a user of the examination support apparatus 100 proceeds with the examination while visually recognizing the captured image 121 displayed almost in real time according to the operation of the camera unit 210 . The doctor operates the camera unit 210 so that the target part to be diagnosed by the examination support apparatus 100 is displayed as the captured image 121 .

For example, when detecting a diagnosis trigger by a doctor, the examination support apparatus 100 generates input data based on the captured image 121 at that time and the model information of the endoscope system 200, and inputs the input data to an analysis neural network, which will be described later. Then, the diagnostic result generated from the output of the neural network for analysis is converted into computer graphics and displayed on the display monitor 120 as the CG index 122 .

Diagnostic triggers can be set in various ways. For example, when detecting that the extracted captured image 121 is a series of still images (freeze) over a certain period of time (for example, two seconds), the examination support apparatus 100 detects the freeze as a diagnostic trigger. Note that FIG. 1 shows a state in which a substantially real-time captured image 121 sequentially generated from a display signal is displayed in a situation where no diagnostic trigger is detected, and the CG index 122 is in a diagnostic standby state. It indicates the purpose.

FIG. 2 is a hardware configuration diagram of the inspection support apparatus 100. As shown in FIG. The examination support apparatus 100 is mainly composed of an arithmetic processing section 110 , a display monitor 120 , an input/output interface 130 , an input device 140 and a storage section 150 . The arithmetic processing unit 110 is a processor (CPU: Central Processing Unit) that performs control of the examination support apparatus 100 and program execution processing. The processor may be configured to cooperate with an arithmetic processing chip such as an ASIC (Application Specific Integrated Circuit) or a GPU (Graphics Processing Unit). The arithmetic processing unit 110 reads out the examination assistance program stored in the storage unit 150 and executes various processes related to examination assistance.

The display monitor 120 is, for example, a monitor having a liquid crystal panel as described above, and displays the captured image 121, the CG index 122, and the like. Input/output interface 130 is a connection interface for exchanging information with an external device, including a connector for connecting connection cable 250 . The input/output interface 130 includes, for example, a LAN unit, takes in an examination support program and update data of an analysis neural network 151 (to be described later) from an external device, and delivers them to the arithmetic processing unit 110 .

The input device 140 is, for example, a keyboard, a mouse, or a touch panel superimposed on the display monitor 120. A doctor or an assistant operates these devices to change the settings of the examination support apparatus 100 or input information necessary for examination. or

The storage unit 150 is a non-volatile storage medium, and is configured by, for example, an HDD (Hard Disk Drive). The storage unit 150 can store various parameter values, functions, display element data, lookup tables, etc. used for control and calculation, in addition to programs for executing control and processing of the examination support apparatus 100 . The storage unit 150 particularly stores an analysis neural network 151 (hereinafter referred to as "analysis NN 151"). The analysis NN 151 is a trained model that outputs regions and probabilities in which lesions exist in input diagnostic images, although the details will be described later. Note that the storage unit 150 may be composed of a plurality of pieces of hardware. For example, the storage medium storing the program and the storage medium storing the analysis NN 151 may be composed of separate pieces of hardware.

The computation processing unit 110 also serves as a functional computation unit that executes various computations according to the processing instructed by the examination support program. The arithmetic processing unit 110 can function as an information acquisition unit 111 , an image acquisition unit 112 , a diagnosis unit 113 and a display control unit 114 . The information acquisition unit 111 acquires model information of the endoscope system 200 . The image acquisition unit 112 acquires an image captured by the camera unit 210 via a display signal sent from the endoscope system 200 . The diagnosis unit 113 generates input data based on the model information acquired by the information acquisition unit 111 and the captured image acquired by the image acquisition unit 112, and inputs the input data to the analysis NN 151, which is a trained model. Make an in-vivo diagnosis. The display control unit 114 controls the display of the display monitor 120 by generating a display signal for a display screen to be displayed on the display monitor 120 and transmitting the signal to the display monitor 120 . The display control unit 114 functions as an output unit that outputs the diagnosis result of the diagnosis unit 113 . The output destination of the diagnosis result is not limited to the display monitor 120. For example, when outputting to an external device, the arithmetic processing unit 110 and the input/output interface 130 work together to function as an output unit. Specific processing of each functional calculation unit will be described later.

Now, it is preferable that an examination support device that is connected to an endoscope system and assists a doctor's diagnosis using captured images output by the endoscope system can be used so as to be connectable to various types of endoscope systems. . Therefore, such an examination support apparatus is sometimes designed so that it can be connected to a plurality of types of endoscope systems. In this case, not only can it be connected to multiple models of endoscope systems manufactured by the same manufacturer, but it may also be designed to be connectable to different models of endoscope systems manufactured by different manufacturers. obtain. For example, it is designed to be connectable to endoscope system models X1, X2, and X3 manufactured by company A and to endoscope system models Y1 and Y2 manufactured by company B, respectively.

Each endoscope system includes at least a camera unit and a system body that processes imaging signals output from the camera unit and generates image data. The camera unit includes an imaging sensor that captures an image of a site to be inspected inside the subject's body, and an imaging signal output therefrom corresponds to the characteristics of the imaging sensor and the control circuit that controls it. That is, if the camera units are different, even if the same inspection site is imaged, the imaging signals output by each are different. In addition, the system body that receives the imaging signal from the camera unit and processes it into image data may have different image processing parameters or different image generation programs. Even if received, the same image data is not necessarily generated. That is, it can be said that the endoscope system generates image data according to the characteristics of the model. Specifically, images of image data generated by the respective endoscope systems may differ from each other in color, sharpness, distortion, noise, and the like.

When a conventional examination support apparatus receives such image data or an image signal adjusted from the image data, generates a diagnostic image, and performs in-vivo diagnosis using a trained model, the same Even if the examination site is targeted, the diagnosis result may differ depending on the connected endoscope system. It is not preferable that the diagnosis results differ depending on the model of the connected endoscope system even though the same test site of the same subject is targeted, because it may lead to misdiagnosis. Therefore, the examination support apparatus 100 according to the present embodiment incorporates the model information of the connected endoscope system 200, thereby minimizing discrepancies in diagnosis results due to model differences.

First, an explanation will be given of the overall flow from when the examination support apparatus 100 according to the present embodiment acquires a display signal from the endoscope system 200 and displays the diagnosis result. FIG. 3 is a diagram for explaining the process until the examination support apparatus 100 displays the diagnosis result.

A signal reproduction image 225 acquired by the image acquisition unit 112 from the display signal sent from the endoscope system 200 and developed is the same as the display image displayed on the system monitor 220 of the endoscope system 200 . The signal reproduced image 225 includes the captured image 221 and examination information 222 . The examination information 222 is, for example, text information. The image acquisition unit 112 determines which of the image areas of the signal-reproduced image 225 is the image area representing the captured image, using the signal-reproduced image 225 that is successively developed at the beginning of connection. Since it is assumed that the image area changes in a short period of time compared to other areas, for example, a changing area in which the amount of difference between successive signal-reproduced images is equal to or greater than a reference amount is determined as the image area. After determining which of the image areas of the signal reproduction image 225 is the image area representing the captured image, the image acquisition unit 112 cuts out the determined image area to obtain the captured image 121 . It can be said that the captured image 121 is substantially an image obtained by reproducing the captured image captured by the camera unit 210 and image-processed by the system main body 211 by the inspection support apparatus 100 .

If the image acquisition unit 112 does not detect a diagnostic trigger at that time and the extracted captured image 121 is not an image to be diagnosed, the image acquisition unit 112 passes the captured image 121 to the display control unit 114 . The display control unit 114 sequentially updates the captured image 121 and displays it on the display monitor 120 in order to realize the display screen in the diagnosis standby state shown in FIG. The image acquisition unit 112 detects a diagnosis trigger at that time, and when the cut-out captured image 121 is an image to be diagnosed, the image acquisition unit 112 transfers the captured image 121 to the display control unit 114 and to the diagnosis unit 113 . . Note that FIG. 3 shows a case where a diagnostic trigger is detected.

The diagnosis unit 113 adjusts the captured image 121 according to the specifications of the analysis NN 151 and generates a diagnostic image. The adjustment from the captured image 121 to the diagnostic image here targets, for example, the aspect ratio and image size of the image, and prepares the analysis NN 151 into a data format that accepts input. not for adjustment. Further, the diagnosis unit 113 receives model information of the endoscope system 200 acquired by the information acquisition unit 111 from the information acquisition unit 111 as described later. Then, the model information and the diagnostic image are unified to generate input data, which is input to the analysis NN 151 read from the storage unit 150 . The analysis NN 151 outputs suspected lesion areas in the image together with their estimated probabilities as diagnostic results. In the example of FIG. 3, an area near the center showing an estimated probability of 90% is output. The diagnosis unit 113 hands over the diagnosis result to the display control unit 114 .

The display control unit 114 expands and arranges the captured image data of the captured image 121 received from the image acquisition unit 112, the diagnosis result received from the diagnosis unit 113, and the model information applied to the diagnosis according to preset display standards. is displayed on the display monitor 120. Specifically, a captured image 121 used as a diagnostic image is arranged on the left side, and a CG index 122 representing the diagnosis result is arranged on the right side. The CG index 122 is computer graphics including a numerical value ("90%" in the example shown) of the estimated probability of being a lesion. Also, a diagnostic area frame 121 a indicating an area estimated to be a lesion is superimposed on the captured image 121 . In addition, model information applied to the diagnosis is expressed in text and placed below the CG index 122 as application information 123 . The result display state continues for a certain period of time (for example, 2 seconds), and then the display returns to the diagnosis standby state.

In this embodiment, the image acquisition unit 112 extracts and acquires the captured image 121 from the display signal received from the endoscope system 200, but the endoscope system 200 receives image data as digital data from an external device. When provided to the device, the captured image may be acquired by receiving and developing this. Regardless of the format, the image acquisition unit 112 may acquire a captured image captured by the camera unit 210 and processed by the system main body 211 .

Next, the analysis NN 151 will be explained. The analytical NN 151 read out from the storage unit 150 and used is a trained model generated by repeating learning using teacher data until it is suitable for operation. Such learning work may be performed using another device, in which case the inspection support device 100 acquires the analyzed NN 151 that has completed learning from the other device via the input/output interface 130. .

FIG. 4 is a diagram for explaining the learning work for generating the analytical NN 151. As shown in FIG. The input data of the analysis NN 151 is, as described above, data in which the model information and diagnostic images are unified. In this embodiment, the model information is composed of body information about the system body 211 that executes image processing in the endoscope system 200 and camera information about the camera unit 210 attached to the system body 211 . That is, the model information represents the combination of which system main body 211 and which camera unit 210 the diagnostic image of interest was generated. For example, if the target diagnostic image is generated by the endoscope system 200 configured by combining the system main body 211 of the model M3 and the camera unit 210 of the model N7, the model information is the model information which is the main body information. This is a combination of M3 and model N7, which is camera information. The data obtained by encoding the model information and adding it to the image data of the diagnostic image becomes the input data of the NN 151 for analysis.

Note that in the present embodiment, as model information, a combination of main body information and camera information, both of which affect the quality of diagnostic images, is employed. This is because it is assumed that the camera unit 210 can be attached to and detached from the system main body 211 and that the combination can be changed among a plurality of models. This is because the quality of the captured image and the diagnostic image generated from the captured image change. However, model information is not limited to a combination of main body information and camera information. Any element of the endoscope system 200 that affects the quality of diagnostic images can be a target of model information. For example, if a lighting unit that illuminates the inspection site can be arbitrarily combined with a camera unit, lighting information indicating which model of lighting unit is combined may also be employed as the model information. Conversely, if the examination support device is to be connected only to an endoscope system in which the combination of the camera unit and the system body cannot be changed, the model information does not need to include the camera information and the body information, respectively. It suffices to indicate the model of the mirror system. The camera information is not limited to information about the model of the camera unit 210, and may be information about specifications (for example, the angle of view, focal length, or F value of the lens of the camera unit 210).

The teacher data is obtained by adding annotation data to the above input data. The annotation data is correct information as to whether or not a cancer lesion area exists in the diagnostic image, and area information of the lesion area if it exists. Judgment on each diagnostic image is made by, for example, a skilled doctor by observing the diagnostic image individually, and the annotation data is created by the doctor or an assistant inputting the judgment result to the learning work device. It is a thing. The area information of the lesion area is represented by, for example, the upper left coordinate A and the lower right coordinate B of a rectangle that includes the site determined to be cancerous.

In this way, the teacher data with annotation data attached to the input data includes the model information (here, the main unit information and the camera information) that influenced the quality of the diagnostic image, and the judgment result by the doctor's judgment. . As for the teacher data, a sufficient number of data is prepared to generate a practical analytical NN 151 through learning. Whether or not it can withstand practical use is determined, for example, by whether or not the percentage of correct answers in judgment exceeds a reference value. When the input data is input, the analysis NN 151 learned by the teacher data prepared in this way has the probability that cancer exists in the diagnostic image that is part of the input data, and the region (for example, The coordinate value of the upper left coordinate A and the coordinate value of the lower right coordinate B) are output. At this time, the analysis NN 151 as a trained model can output a diagnosis result based on differences in the quality of diagnostic images depending on the model. That is, the analysis NN 151 that has been learned including the difference in model information, for example, when the same inspection site of the same subject is targeted, even if the model of the connected endoscope system is different. , can be expected to output the same diagnostic results. In the present embodiment, it is assumed that supervised learning is performed using the above-described teacher data as learning data. Other machine learning methods may be adopted as long as learning is performed based on learning data associated with the output captured image.

The model information can be obtained, for example, by accepting input from a doctor or an assistant before starting an examination. FIG. 5 is a diagram showing an example of a setting screen regarding model information displayed on the display monitor 120. As shown in FIG.

The information acquisition unit 111 displays a setting screen regarding model setting on the display monitor 120 via the display control unit 114 . Specifically, along with a title 124 indicating "model setting", selection items 125 indicating selectable models for each of the system main body 211 and the camera unit 210 are displayed.

For example, when the display monitor 120 has a touch panel as the input device 140, when the user who is a doctor or an assistant taps the pull-down button 127, the information acquisition unit 111 detects the operation and selects a selectable model name. expand. FIG. 5 shows that the pull-down button 127 for model selection of the camera unit 210 is tapped and selectable model names are listed. A scroll indicator 128 indicates that a part of selectable model names is displayed, and the user can move it up and down to display other model names. When the user taps a specific model name (“X Company X-0B” in the figure), the model name is highlighted to inform the user that the selection has been accepted. That is, the information acquisition unit 111 acquires model information from the user through such a user interface.

It should be noted that the model information is not limited to direct input by the user, and can be realized by other methods. For example, as described above, the signal reproduced image 225 includes the inspection information 222. If the inspection information 222 includes the model information of the system main body 211 and the camera unit 210, the information acquisition unit 111 acquires the inspection information 222. The model information may be obtained by OCR analysis. Also, for example, if the endoscope system 200 provides model information to an external device together with image data as digital data, the information acquisition unit 111 can acquire the model information more directly. Even if the model information is not provided to the external device together with the image data, the feature amount of the image data, the format of the image data, and the character string ( As an example, the model information may be determined based on at least one character image representing the model information.

Next, an example of a processing procedure for main processing of examination support executed by the arithmetic processing unit 110 will be described using a flow chart. FIG. 6 is a flowchart for explaining the processing procedure of the arithmetic processing unit 110. As shown in FIG. The flow starts when the examination support apparatus 100 is activated and the examination support program is executed.

The information acquisition unit 111 acquires model information through input to the input device 140 by the user in step S101. The information acquisition unit 111 passes the acquired model information to the diagnosis unit 113 . When the acquisition of the model information is completed, the process proceeds to step S102.

When proceeding to step S102, the image acquisition unit 112 acquires a display signal from the endoscope system 200 to generate the signal reproduction image 225, and in step S103, cuts out the captured image 121 from the signal reproduction image 225 and controls display. Hand over to department 114 . Moreover, if a diagnostic trigger is detected at that time, it is also handed over to the diagnostic unit 113 . Upon receiving the captured image 121 from the image acquisition unit 112, the diagnostic unit 113 generates a diagnostic image from the captured image 121 and unifies the diagnostic image and model information to generate input data in step S104.

The diagnosis unit 113 advances to step S105, inputs the generated input data to the analysis NN 151 read from the storage unit 150, and causes the region suspected of having a lesion in the diagnostic image and its estimated probability to be calculated and output. The diagnosis unit 113 passes the diagnosis result output by the analysis NN 151 to the display control unit 114 together with the model information applied to the diagnosis.

In step S106, the display control unit 114 displays the captured image data of the captured image 121 received from the image acquisition unit 112, the diagnosis result received from the diagnosis unit 113, and the model information applied to the diagnosis as shown in FIG. Displayed on the display monitor 120 . When the result display state continues for a certain period of time, the process proceeds to step S107.

Arithmetic processing unit 110 detects whether or not the in-vivo examination by camera unit 210 is completed in step S107. If the end is not detected, the process returns to step S102 to continue the inspection. When the end is detected, the defined end processing is executed to end the series of processing. Note that the arithmetic processing unit 110 detects the end of the in-vivo examination, for example, by catching a sudden change in the brightness of the captured image 121 due to the camera unit 210 being discharged from the oral cavity.

Next, some modifications will be described. In the above-described embodiment, as described with reference to FIG. 4, annotation data is added to input data in which model information and diagnostic images are unified to prepare teacher data, and analysis is performed by performing learning using this teacher data. NN151 for That is, when the generated analysis NN 151 receives the input data in which the model information and the diagnostic image are unified, it outputs the diagnostic result based on the difference in the quality of the diagnostic image according to the model of the endoscope system. do. However, if differences in the quality of diagnostic images depending on the model are offset by standardization processing in advance, the neural network for analysis only needs to accept standardized diagnostic images, and the neural network for analysis itself can be adjusted according to the model. There is no need to absorb differences in diagnostic image quality. A first modified example creates and operates an analysis neural network that assumes such standardization processing.

FIG. 7 is a diagram for explaining the learning work for generating the analysis neural network 151' (hereinafter referred to as "analysis NN 151'") in the first modified example. In the first modified example, before starting learning of the analysis NN 151', first, diagnostic images that serve as teacher data are prepared in advance.

A diagnostic image is an image captured by a certain camera unit 210 and image-processed by a certain system main body 211. Standardization processing is performed to remove characteristics that depend on the model of the camera unit 210 and the model of the system main body 211. It is converted into a standardized image by applying The standardization process is, for example, a filtering process that applies an image filter to a captured image to correct model-specific quirks caused by differences in the quality of captured images due to differences in models. For example, an image filter for M3-N7 prepared for correcting the peculiarities of this combination is applied to a captured image generated by combining the camera unit 210 of model N7 with the system main body 211 of model M3. to convert it into a standardized image. A standardized image that has undergone standardization processing in this way has differences in quality due to differences in endoscope system models removed. Note that the quality (color, sharpness, distortion, noise, etc.) of the standardized image converted by the standardization process is set in advance within a range that can be realized by the filter process.

Therefore, the input data of the analysis NN 151' becomes a diagnostic image as a standardized image thus standardized. The teacher data is obtained by adding annotation data to such a diagnostic image. Similar to the annotation data described with reference to FIG. 4, the annotation data is correct information as to whether or not a cancer lesion area exists in the diagnostic image, and if it exists, area information of the lesion area. .

When a standardized diagnostic image is input, the analysis NN 151′ learned by the teacher data prepared in this way has the probability that cancer exists in the diagnostic image, and the region (for example, upper left The coordinate value of the coordinate A and the coordinate value of the lower right coordinate B) are output. That is, the diagnosis unit 113 generates a diagnostic image that has undergone standardization processing using the model information, inputs this to the analysis NN 151′ as a learned model, and causes it to perform calculations, so that the model of the endoscope system It is possible to obtain a diagnostic result that eliminates the influence caused by it as much as possible.

FIG. 8 is a diagram illustrating processing up to display of diagnosis results in the first modification. Here, differences from the processing described with reference to FIG. 3 will be described.

Upon receiving the model information from the information acquisition unit 111 , the diagnosis unit 113 reads an image filter corresponding to the model information from the storage unit 150 . Then, the captured image 121 received from the image acquisition unit 112 is subjected to standardization processing in which the image filter is applied to generate a diagnostic image. At this time, an adjustment process for adjusting the aspect ratio, image size, etc. of the image to prepare the data format for the analysis NN 151' to accept the input is also executed.

The diagnostic unit 113 inputs the diagnostic image generated as input data in this way to the analytical NN 151 ′ read from the storage unit 150 . The analysis NN 151' outputs suspected lesion regions present in the image together with their estimated probabilities as diagnostic results. The diagnosis unit 113 hands over the diagnosis result to the display control unit 114 .

FIG. 9 is a flowchart for explaining the processing procedure of the arithmetic processing unit 110 in the first modified example. Processing steps similar to those in the flow chart shown in FIG. 6 are given the same step numbers, and descriptions thereof are omitted.

Upon receiving the captured image 121 from the image acquisition unit 112, the diagnosis unit 113 performs standardization processing on the received captured image 121 in step S204. Specifically, the image filter corresponding to the model information acquired in step S101 is read from the storage unit 150 and applied to the received captured image 121 . In step S205, the diagnosis unit 113 inputs the diagnostic image as input data obtained by applying the image filter to the analysis NN 151', thereby obtaining a diagnostic result for the diagnostic image. Then, the diagnosis result is handed over to the display control unit 114 together with the model information applied to the diagnosis.

Next, in describing the second modified example, a reference example as a premise thereof will be described. In the embodiments described so far, it has been described that the image quality of the image data generated by the endoscope system can vary from model to model of the endoscope system. However, even if the lesion is the same, differences in color and texture may occur due to other factors, even if the image is captured using the same type of endoscope system. . One of the factors is the medical history of the subject.

Depending on the medical history of the subject, its effects may appear at the examination site. For example, it appears as discoloration or atrophy of the examination site. When such an inspection site is used as an inspection target, the diagnosis results may differ depending on the presence or absence of a specific medical history of the subject. It is not preferable that the same lesion has the same condition but the diagnosis result differs due to the difference in the medical history of the subject, which may lead to misdiagnosis. Therefore, in the examination support apparatus according to the reference example, by taking in the medical history information regarding the medical history of the subject, discrepancies in the diagnosis results due to differences in the medical history of the subjects are suppressed as much as possible.

FIG. 10 is a diagram for explaining the learning work for generating the analysis neural network 152 (hereinafter referred to as "analysis NN 152") in the reference example. The input data of the analysis NN 152 is data in which medical history information and diagnostic images are unified. Here, the past history information is a target of the past history information, which may cause differences in image quality between the captured images of the inspection site captured by the endoscope system.

As shown in the figure, past medical history information includes eradication history of Helicobacter pylori (yes/no), serum H. pylori antibody (negative/positive), breath test (negative/positive), pepsinogen test (negative/positive). , prostate cancer (postoperative/no), lung cancer (postoperative/no), tumor marker (high/mildly high/low), gastric SMT (suspected/no), history of gastric ulcer (yes/no), gastric malignant lymphoma Past history (yes/no), history of early gastric cancer (yes/no), ESD (yes/no), history of tongue cancer (yes/no), history of floor of the mouth cancer (yes/no), hypopharynx A history of cancer (yes/no), a history of colorectal cancer (yes/no), etc. can be targets. If the endoscope system targets a specific examination site, the history may be limited to the past history that affects the examination site. Further, for example, when there is a history of eradication of Helicobacter pylori, the number of years after eradication may be included in the medical history information. The data obtained by encoding the past history information and adding it to the image data of the diagnostic image becomes the input data of the analysis NN 152 .

The teacher data is obtained by adding annotation data to the above input data. Similar to the annotation data described with reference to FIG. 4, the annotation data is correct information as to whether or not a cancer lesion area exists in the diagnostic image, and if it exists, area information of the lesion area. . When the input data is input, the analysis NN 152 learned by the teacher data prepared in this way has the probability that cancer exists in the diagnostic image that is a part of the input data, and the region (for example, The coordinate value of the upper left coordinate A and the coordinate value of the lower right coordinate B) are output. At this time, the analysis NN 152 as a learned model can output diagnostic results based on differences in the quality of diagnostic images according to the medical history of the subject. That is, the analysis NN 152 that has been learned including differences in past history information outputs the same diagnostic result regardless of the difference in the past history of the subject if, for example, the same lesion is in the same situation. can be expected.

Medical history information can be obtained, for example, by receiving input from a doctor or an assistant before starting an examination. FIG. 11 is a diagram showing an example of a setting screen regarding past history information in the reference example.

The information acquisition unit 111 displays a setting screen for setting the subject's past history on the display monitor 120 via the display control unit 114 . Specifically, along with a title 124 indicating "setting of past history", selection items 125 indicating selectable past histories are displayed. A radio button 129 is arranged at the head of each selection item 125, and the user can select the selection item 125 by tapping this. The information acquisition unit 111 acquires the past history information of the subject from the user through such a user interface. Note that the information acquisition unit 111 may acquire past medical history information via a network, for example, from an electronic medical record system terminal in which subject information is recorded.

In the second modification, past history information is added to the input data in addition to the above-described embodiment in which input data is generated from model information and diagnostic images. A neural network for analysis corresponding to such input data is created by learning including anamnesis information. FIG. 12 is a diagram for explaining the learning work for generating the analysis neural network 151″ (hereinafter referred to as “analysis NN 151″”) in the second modification.

The input data of the analysis NN 151″ is data in which model information, medical history information, and diagnostic images are unified. and camera information about the camera unit 210 attached to the system main body 211. As described above, the past history information is an image of the inspection site captured by the endoscope system. The model information and the past history information are coded and added to the image data of the diagnostic image to be the input data of the analysis NN 151''.

The teacher data is obtained by adding annotation data to the above input data. Similar to the annotation data described with reference to FIG. 4, the annotation data is correct information as to whether or not a cancer lesion area exists in the diagnostic image, and if it exists, area information of the lesion area. . When the input data is input, the analysis NN 151'' learned by the teacher data prepared in this way obtains the probability that cancer exists in the diagnostic image that is a part of the input data, and the region ( For example, the coordinate value of the upper left coordinate A and the coordinate value of the lower right coordinate B) are output.
At this time, the analysis NN 151 ″ as a learned model outputs diagnostic results based on the difference in diagnostic image quality depending on the model and the difference in diagnostic image quality depending on the patient's medical history. That is, the analysis NN 151 ″ that has been learned including differences in model information and past history information can be the same regardless of the model of the endoscope system or the patient's past history. It can be expected to output diagnostic results.

FIG. 13 is a diagram illustrating processing up to display of diagnosis results in the second modification. Here, differences from the processing described with reference to FIG. 3 will be described.

The diagnostic unit 113 adjusts the captured image 121 according to the specifications of the analysis NN 151″ and generates a diagnostic image. Information and the patient's past history information are received from the information acquisition unit 111. Then, the model information, the past history information, and the diagnostic image are unified to generate input data, which is input to the analysis NN 151'' read out from the storage unit 150. do. The analysis NN 151 ″ outputs regions in the image that are suspected to be lesions, together with their estimated probabilities, as diagnostic results.

The display control unit 114 displays the captured image data of the captured image 121 received from the image acquisition unit 112, the diagnosis result received from the diagnosis unit 113, and the model information and medical history information applied to the diagnosis according to preset display standards. It is developed, arranged, and displayed on the display monitor 120 . The anamnesis information applied to the diagnosis is similarly displayed as the application information 123 together with the model information arranged below the CG index 122 .

FIG. 14 is a flowchart for explaining the processing procedure of the arithmetic processing unit in the second modification. Processing steps similar to those in the flow chart shown in FIG. 6 are given the same step numbers, and descriptions thereof are omitted.

After acquiring the model information in step S101, the information acquiring unit 111 acquires past history information in step S301. Specifically, the medical history information of the subject is obtained from the user through the user interface described with reference to FIG. When acquisition of past history information is completed, the process proceeds to step S102. The order of obtaining the model information and the obtaining of the medical history information may be reversed, or they may be obtained in parallel.

Upon receiving the captured image 121 from the image acquiring unit 112 in step S103, the diagnostic unit 113 generates a diagnostic image from the captured image 121 in step S304, and unifies the diagnostic image, model information, and past history information. Generate input data.

The diagnosis unit 113 advances to step S305, inputs the generated input data to the analysis NN 151″ read out from the storage unit 150, and causes the area suspected of being a lesion in the diagnostic image and its estimated probability to be calculated and output. The diagnosis unit 113 transfers the diagnosis result output by the analysis NN 151″ to the display control unit 114 together with the model information and the medical history information applied to the diagnosis.

In the reference example and the second modified example described above, as in the first modified example, a diagnostic image is generated as a standardized image standardized by performing standardization processing according to the medical history information, and the first It can also be configured to input to the analysis NN 151' used in the modified example. In this case, for example, an image filter for correcting the difference in the quality of the captured image caused by the presence or absence of a medical history is prepared in advance for each medical history. Then, in the examination stage, when information such as “gastric ulcer present” is acquired as past history information, the diagnostic unit 113 applies an image filter for gastric ulcer to the captured image 121 for canceling the characteristics of the image caused by the experience of gastric ulcer. and convert it to a normalized image.

In the present embodiment described above, it is assumed that the endoscope system 200 and the examination support apparatus 100 are connected via the connection cable 250, but wireless connection instead of wired connection may be used. In the above-described embodiment, the camera unit 210 included in the endoscope system 200 is assumed to be a flexible endoscope, but even if the camera unit 210 is a rigid endoscope, There is no difference in the configuration or processing procedure of the examination support apparatus 100 . It should be noted that the diagnosis by the diagnosis unit in the present embodiment only assists the doctor's diagnosis, and the final decision is made by the doctor.

In the present embodiment described above, an example in which the examination support apparatus 100 acquires model information from the endoscope system 200 has been shown. At least one of the zoom magnification, light source intensity, and sharpness of the camera unit 210 at the time may be obtained from the endoscope system 200 . In this case, the analysis NN 151 is a trained model that has been trained using a learning data set containing imaging information. The diagnosis unit 113 generates input data based on the model information and imaging information acquired by the information acquisition unit 111 and the captured image acquired by the image acquisition unit 112, and inputs the input data to the analysis NN 151. make a diagnosis.

Further, in the present embodiment described above, the CG index 122 indicating the diagnosis result displays the numerical value of the estimated probability of being a lesion, but the display mode of the diagnosis result is not limited to this. For example, the diagnostic region frame 121a may be highlighted when an estimated probability equal to or greater than a threshold is output. In this case, the display of the CG index 122 may be omitted, or simply a warning display indicating that a result exceeding the threshold has been output.

DESCRIPTION OF SYMBOLS 100... Examination support apparatus, 101... Apparatus main body, 110... Arithmetic processing part, 111... Information acquisition part, 112... Image acquisition part, 113... Diagnosis part, 114... Display control part, 120... Display monitor, 121... Captured image, 121a... Diagnosis area frame, 122... CG index, 123... Application information, 124... Title, 125... Selection item, 126... Selection index, 127... Pull-down button, 128... Scroll index, 129... Radio button, 130... Input/output interface , 140... Input device 150... Storage unit 151, 151', 151", 152... Analysis neural network 200... Endoscope system 210... Camera unit 211... System body 220... System monitor 221... Captured image 222 Inspection information 225 Signal reproduced image 250 Connection cable

Claims (12)

An examination support device that is used in connection with an endoscope system that includes a camera unit that is inserted into the body of a subject,
an information acquisition unit that acquires model information of the endoscope system;
an image acquisition unit that acquires an image captured by the camera unit;
a diagnosis unit that generates input data based on the model information acquired by the information acquisition unit and the captured image acquired by the image acquisition unit, and inputs the input data to a trained model to diagnose the interior of the body;
and an output unit for outputting a diagnosis result by the diagnosis unit. The learned model is learned based on learning data in which model information indicating one of a plurality of endoscope systems and captured images output from the endoscope system are associated with each other. can be,
The diagnosis unit generates the input data by associating the model information acquired by the information acquisition unit with the captured image acquired by the image acquisition unit, and inputs the input data to the trained model, so that the inside of the body The examination support device according to claim 1, which performs diagnosis. The trained model is learned by using standardized images obtained by standardizing captured images output from each of a plurality of endoscope systems and removing characteristics dependent on the model of the endoscope system as learning data. and
The diagnosis unit generates the input data by applying the standardization processing based on the model information acquired by the information acquisition unit to the captured image acquired by the image acquisition unit, and inputs the input data to the trained model. 2. The examination support apparatus according to claim 1, wherein the inside of the body is diagnosed by performing the examination. 2. The apparatus according to claim 1, wherein the model information acquired by the information acquisition unit includes main body information about a system main body that executes image processing in the endoscope system, and camera information about the camera unit attached to the system main body. inspection support device. 5. The inspection support apparatus according to claim 4, wherein the camera information includes at least one of information regarding a model of the camera unit and information regarding specifications of the camera unit. 2. The examination support apparatus according to claim 1, wherein the information acquisition unit acquires the model information by receiving a designation by a user. The information acquisition unit also acquires past history information related to the past history of the subject,
The examination support according to claim 1, wherein the diagnosis unit diagnoses the inside of the body based on the model information and the medical history information acquired by the information acquisition unit and the captured image acquired by the image acquisition unit. Device. The image acquisition unit determines an image region corresponding to the captured image among the reproduced images based on a difference amount between images of a plurality of reproduced images generated from image signals continuously acquired from the endoscope system. 2. The examination support apparatus according to claim 1, wherein the captured image is generated and obtained based on the determined image area. The examination support apparatus according to claim 1, wherein the information acquisition unit acquires the model information from the captured image. The information acquisition unit further acquires imaging information of the endoscope system,
The examination according to claim 1, wherein the diagnosis unit diagnoses the interior of the body based on a combination of the model information and the imaging information acquired by the information acquisition unit and the captured image acquired by the image acquisition unit. support equipment. An examination support method using an examination support apparatus that is used in connection with an endoscope system having a camera unit that is inserted into the body of a subject,
an information acquisition step of acquiring model information of the endoscope system;
an image acquisition step of acquiring a captured image captured by the camera unit;
a diagnosis step of generating input data based on the model information acquired in the information acquisition step and the captured image acquired in the image acquisition step, and inputting the input data to a trained model to diagnose the interior of the body;
and an output step of outputting a diagnosis result obtained by the diagnosis step. An examination support program for controlling an examination support apparatus used in connection with an endoscope system having a camera unit inserted into the body of a subject,
an information acquisition step of acquiring model information of the endoscope system;
an image acquisition step of acquiring a captured image captured by the camera unit;
a diagnosis step of generating input data based on the model information acquired in the information acquisition step and the captured image acquired in the image acquisition step, and inputting the input data to a trained model to diagnose the interior of the body;
and an output step of outputting a diagnosis result obtained by the diagnosis step.

Images