JP2023049952A - Computer program, information processing device, information processing method, and learning model generation method - Google Patents

Abstract

To provide a computer program, an information processing device, an information processing method, and a learning model generation method capable of improving lesion detection accuracy.SOLUTION: A computer program causes a computer to execute processing for acquiring medical image data indicating a cross-sectional image of a hollow organ, acquiring position data on a predetermined part by inputting the acquired medical image data to a first learning model for outputting position data on the predetermined part of the hollow organ when the medical image data indicating the cross-sectional image of the hollow organ is input, and outputting presence or absence of an object by inputting the acquired medical image data and the acquired position data on the predetermined part to a second learning model for outputting presence or absence of the object in the hollow organ when the medical image data indicating the cross-sectional image of the hollow organ and the position data on the predetermined part of the hollow organ are input.SELECTED DRAWING: Figure 2

Description

The present invention relates to a computer program, an information processing apparatus, an information processing method, and a learning model generation method.

In PCI (percutaneous coronary intervention) procedures, IVUS (Intra Vascular Ultra Sound), OCT (Optical Coherence Tomography), OFDI (Optical Frequency Domain Imaging) Interpretation of imaging modalities such as MRI is highly difficult, and automation for standardization of interpretation is being considered.

In Patent Document 1, a catheter is inserted into a blood vessel, and a blood vessel cross-sectional image is generated based on signals (ultrasonic waves emitted toward the blood vessel tissue and reflected waves) obtained by an imaging core housed in the catheter. A diagnostic imaging apparatus is disclosed.

WO2017/164071

However, lesions that lead to complications such as dissection and thrombosis are difficult to identify and may be overlooked.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a computer program, an information processing apparatus, an information processing method, and a learning model generation method that can improve the detection accuracy of lesions.

The present application includes a plurality of means for solving the above problems. To give one example, a computer program acquires medical image data representing a cross-sectional image of a hollow organ in a computer, and generates a cross-sectional image of the hollow organ. is input to a first learning model for outputting position data of a predetermined portion of the hollow organ to obtain the position data of the predetermined portion of the lumen; When medical image data representing a cross-sectional image of a hollow organ and position data of a predetermined portion of the hollow organ are input, the obtained medical image is input to a second learning model that outputs the presence or absence of an object in the hollow organ. The data and the acquired position data of the predetermined portion are input, and the processing of outputting the presence or absence of the object is executed.

According to the present invention, lesion detection accuracy can be improved.

1 is a diagram showing an example of a configuration of a diagnostic imaging system according to an embodiment; FIG. It is a figure which shows an example of a structure of an information processing apparatus. It is a figure which shows an example of a structure of a 1st learning model. It is a figure which shows an example of a structure of a 1st learning model. FIG. 10 is a diagram showing an example of a configuration of a second learning model; FIG. It is a figure which shows the 1st example of the detection of the presence or absence of a target object by an information processing apparatus. FIG. 10 is a diagram showing a first example of input processing of weight position information; FIG. 10 is a diagram showing a first example of a method for generating a second learning model; FIG. 10 is a diagram showing a second example of input processing of weight position information; FIG. 10 is a diagram illustrating a second example of a method for generating a second learning model; It is a figure which shows the 2nd example of the detection of the presence or absence of a target object by an information processing apparatus. FIG. 10 is a diagram illustrating a third example of detection of presence/absence of an object by an information processing device; FIG. 10 is a diagram showing a fourth example of detection of presence/absence of an object by an information processing device; It is a figure which shows the 1st example of the detection process of the target object by an information processing apparatus. It is a figure which shows the 2nd example of the detection process of the target object by an information processing apparatus. It is a figure which shows an example of the production|generation process of the 1st learning model by an information processing apparatus. It is a figure which shows the 1st example of the production|generation process of the 2nd learning model by an information processing apparatus. FIG. 10 is a diagram illustrating a second example of processing for generating a second learning model by the information processing device;

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below. FIG. 1 is a diagram showing an example of the configuration of a diagnostic imaging system 100 according to this embodiment. The diagnostic imaging system 100 is an apparatus for performing intravascular imaging (diagnostic imaging) used for cardiac catheterization (PCI). Cardiac catheterization is a method of treating a narrowed portion of a coronary artery by inserting a catheter from a blood vessel such as the groin, arm, or wrist. There are two methods of intravascular imaging: intravascular ultrasound (IVUS) and optical coherence tomography (OFDI). IVUS utilizes the reflection of ultrasound to interpret the inside of a blood vessel as a tomographic image. Specifically, a thin catheter equipped with an ultra-small sensor at the tip is inserted into the coronary artery, passed through the affected area, and then ultrasonic waves emitted from the sensor can be used to generate medical images of the inside of the blood vessel. OFDI uses near-infrared rays to interpret the state of blood vessels with high-resolution images. Specifically, similar to IVUS, a catheter is inserted into a blood vessel, near-infrared rays are emitted from the distal end, a cross section of the blood vessel is measured by an interferometry, and a medical image is generated. OCT is an intravascular imaging diagnosis that applies near-infrared rays and optical fiber technology. In the present specification, medical images (medical image data) include those generated by IVUS, OFDI, or OCT, but the case where the IVUS method is mainly used will be described below.

The diagnostic imaging system 100 includes a catheter 10 , an MDU (Motor Drive Unit) 20 , a display device 30 , an input device 40 and an information processing device 50 . A server 200 is connected to the information processing device 50 via the communication network 1 .

A catheter 10 is a diagnostic imaging catheter for obtaining an ultrasonic tomographic image of a blood vessel by the IVUS method. The catheter 10 has an ultrasonic probe at its distal end for obtaining ultrasonic tomographic images of blood vessels. The ultrasonic probe has an ultrasonic transducer that emits ultrasonic waves in a blood vessel and an ultrasonic sensor that receives reflected waves (ultrasonic echoes) reflected by structures such as biological tissue of the blood vessel or medical equipment. The ultrasonic probe is configured to advance and retreat in the longitudinal direction of the blood vessel while rotating in the circumferential direction of the blood vessel.

The MDU 20 is a driving device to which the catheter 10 can be detachably attached, and controls the operation of the catheter 10 inserted into the blood vessel by driving the built-in motor according to the operation of the medical staff. The MDU 20 can rotate in the circumferential direction while moving the ultrasonic probe of the catheter 10 from the tip (distal) side to the base end (proximal) side (pullback operation). The ultrasonic probe continuously scans the inside of the blood vessel at predetermined time intervals, and outputs reflected wave data of detected ultrasonic waves to the information processing device 50 .

The information processing device 50 generates (acquires) a plurality of time-series medical images including tomograms of blood vessels based on the reflected wave data output from the ultrasonic probe of the catheter 10 . Since the ultrasound probe scans the inside of the blood vessel while moving from the tip (distal) side to the base end (proximal) side, multiple medical images in chronological order are multiple images from the distal to the proximal side. It becomes a tomographic image of the blood vessel observed at the point.

The display device 30 includes a liquid crystal display panel, an organic EL display panel, or the like, and can display the results of processing by the information processing device 50 . The display device 30 can also display medical images generated (acquired) by the information processing device 50 .

The input device 40 is an input interface such as a keyboard, a mouse, etc., for receiving input of various setting values, operation of the information processing device 50, and the like when performing an examination. The input device 40 may be a touch panel, soft keys, hard keys, or the like provided on the display device 30 .

The server 200 is, for example, a data server and may include an image DB storing medical image data.

FIG. 2 is a diagram showing an example of the configuration of the information processing device 50. As shown in FIG. The information processing apparatus 50 can be configured by a computer, and includes a control section 51 that controls the entire information processing apparatus 50 , a communication section 52 , an interface section 53 , a recording medium reading section 54 , a memory 55 and a storage section 56 .

The control unit 51 incorporates a required number of CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-purpose computing on graphics processing units), TPU (Tensor Processing Unit), etc. configured as follows. Further, the control unit 51 may be configured by combining DSPs (Digital Signal Processors), FPGAs (Field-Programmable Gate Arrays) quantum processors, and the like. The control unit 51 has functions of a first acquisition unit, a second acquisition unit, and an output unit, and also has functions implemented by a computer program 57 described later.

The memory 55 can be composed of a semiconductor memory such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), or a flash memory.

The communication unit 52 includes, for example, a communication module and has a function of communicating with the server 200 via the communication network 1 . Also, the communication unit 52 may have a communication function with an external device (not shown) connected to the communication network 1 .

The interface section 53 provides interface functions among the catheter 10 , the display device 30 and the input device 40 . The information processing device 50 (control unit 51 ) can transmit and receive data and information to and from the catheter 10 , the display device 30 and the input device 40 through the interface unit 53 .

The recording medium reading unit 54 can be configured by, for example, an optical disk drive, and reads a computer program (program product) recorded on a recording medium 541 (for example, an optically readable disk storage medium such as a CD-ROM). It can be read by the unit 54 and stored in the storage unit 56 . The computer program 57 is developed in the memory 55 and executed by the control unit 51 . Note that the computer program 57 may be downloaded from an external device via the communication unit 52 and stored in the storage unit 56 .

The storage unit 56 can be configured by, for example, a hard disk or a semiconductor memory, and can store required information. The storage unit 56 can store a first learning model 58 and a second learning model 59 in addition to the computer program 57 . The first learning model 58 and the second learning model 59 include a pre-learning model, an in-learning model, or a trained model. Details of the first learning model 58 and the second learning model 59 will be described later.

Below, the first learning model 58 and the second learning model 59 are described, and then, by using both the first learning model 58 and the second learning model 59, an object in a hollow organ (e.g., blood vessel) A method for detecting the presence or absence of an object will be described.

The first learning model 58 outputs position data of a predetermined portion of the hollow organ when medical image data representing a cross-sectional image of the hollow organ is input. The predetermined site can be classified into, for example, a case that includes a lumen area and a case that includes both lumen and blood vessel areas. The first learning model 58 is a model for recognizing a predetermined object (predetermined site) included in medical images. The first learning model 58 can classify objects in units of pixels by, for example, using an image recognition technique using semantic segmentation. area can be recognized. Hereinafter, the first learning model 58 that outputs the region of the lumen will be referred to as the first learning model 581, and the first learning model 58 that outputs the regions of the lumen and the blood vessel will be referred to as the first learning model 582. Note that the first learning models 581 and 582 are collectively represented by a first learning model 58 .

FIG. 3 is a diagram showing an example of the configuration of the first learning model 581. As shown in FIG. The first learning model 581 includes an input layer 581a, an intermediate layer 581b, and an output layer 581c, and can be configured by U-Net, for example. The middle layer 581b comprises multiple encoders and multiple decoders. A plurality of encoders repeat convolution processing on the medical image data input to the input layer 581a. A plurality of decoders repeat upsampling (deconvolution) processing for the image convolved by the encoder. When decoding the convolved image, the feature map generated by the encoder is added to the image to be deconvolved. This makes it possible to retain the position information that is lost due to the convolution process, and to output a more accurate segmentation (which pixel belongs to which class). The first learning model 581 can output position data indicating the region of the lumen when medical image data is input. The position data is coordinate data of picture elements (pixels) indicating the region of the lumen. In other words, the learning model 581 can classify each pixel of the input medical image data into two classes, classes 1 and 2, for example. Class 1 indicates Background and indicates the area outside the lumen. Class 2 indicates Lumen and indicates the lumen of a blood vessel. Therefore, the boundary between pixels classified as class 1 and pixels classified as class 2 indicates the boundary of the lumen. Note that the first learning model 581 is not limited to U-Net, and may be, for example, GAN (Generative Adversarial Network), SegNet, or the like.

A method of generating the first learning model 581 can be as follows. First, medical image data representing a cross-sectional image of a blood vessel and first training data including position data of a region of the lumen of the blood vessel are acquired. The first training data may be collected and stored in the server 200 and then acquired from the server 200, for example. Next, based on the first training data, the first learning model 581 is configured to output the position data of the lumen area when the medical image data representing the cross-sectional image of the blood vessel is input to the first learning model 581. should be generated. Note that the first learning model 581 may be generated so as to include position data of the boundary of the lumen in the training data instead of the position data of the region of the lumen and output the position data of the boundary of the lumen. good.

FIG. 4 is a diagram showing an example of the configuration of the first learning model 582. As shown in FIG. The first learning model 582 comprises an input layer 582a, an intermediate layer 582b, and an output layer 582c, and can be configured by U-Net, for example. The middle layer 582b, like the first learning model 581, comprises multiple encoders and multiple decoders. When medical image data is input, the first learning model 582 can output position data indicating the lumen area and the blood vessel area. The position data is coordinate data of pixels indicating the lumen area and the blood vessel area. In other words, the learning model 582 can classify each pixel of the input medical image data into three classes 1, 2, and 3, for example. Class 1 indicates Background, which indicates the area outside the blood vessel. Class 3 stands for (Plaque + Media) and indicates areas of blood vessels containing plaque. Class 2 indicates Lumen and indicates the lumen of a blood vessel. Therefore, the boundary between the pixels classified into class 2 and the pixels classified into class 3 indicates the boundary of the lumen, and the boundary between the pixels classified into class 1 and the pixels classified into class 3 indicates the boundary of the blood vessel. indicates The first learning model 582 is also not limited to U-Net, and may be, for example, GAN (Generative Adversarial Network), SegNet, or the like.

A method of generating the first learning model 582 can be as follows. First, medical image data representing a cross-sectional image of a blood vessel and first training data including position data of the lumen region and the blood vessel region of the blood vessel are acquired. The first training data may be collected and stored in the server 200 and then acquired from the server 200, for example. Next, based on the first training data, when medical image data representing a cross-sectional image of a blood vessel is input to the first learning model 582, the first learning model 582 is configured to output position data of each of the lumen region and the blood vessel region. 1 learning model 582 may be generated. In addition, the position data of the lumen boundary and the blood vessel boundary are included in the training data instead of the position data of the lumen and the blood vessel region, and the position data of the lumen boundary and the blood vessel boundary are output. A first learning model 582 may be generated.

Next, the second learning model 59 will be explained. The second learning model 59 outputs the presence or absence of an object in the hollow organ when medical image data representing a cross-sectional image of the hollow organ is input. The second learning model 59 is a model for recognizing objects included in medical images. Objects include lesions and structures. Lesions include, for example, dissections, protrusions, or thrombi that occur only on the superficial layers of the lumen. Lesions also include calcified or attenuating plaques that occur from the lining of the lumen to the blood vessels. Structures include stents or guidewires.

FIG. 5 is a diagram showing an example of the configuration of the second learning model 591. As shown in FIG. The second learning model 591 includes an input layer 591a, an intermediate layer 591b, and an output layer 591c, and can be configured by, for example, a convolutional neural network (CNN). The intermediate layer 591b comprises multiple convolutional layers, multiple pooling layers, and a fully connected layer. The second learning model 591 can output the presence/absence of an object when medical image data is input. The medical image data input to the input layer 591a undergoes a convolution operation using a convolution filter (also referred to as a filter) in the convolution layer to output a feature map. The pooling layer performs processing to reduce the size of the feature map output from the convolutional layer. With the pooling layer, for example, even if the characteristic portion is slightly deformed or displaced in the medical image, the characteristic portion can be extracted by absorbing the difference due to the deformation or displacement.

The output layer 591c is composed of 360 nodes, and values corresponding to the presence or absence of an object on a scanning line extending radially from a predetermined position of the medical image (for example, object present: 1, object absent : 0, etc.). A scanning line is composed of 360 line segments obtained by dividing the entire circumference into 360 equal parts. The structure is not limited to dividing the entire circumference into 360 equal parts. For example, the entire circumference may be equally divided into halves, 3 equal parts, 36 equal parts, or the like. Note that the second learning model 591 may be configured to detect the presence or absence of a target within a medical image without using a scan line, instead of detecting the presence or absence of the target on the scanning line.

In this embodiment, when the presence or absence of an object is detected using the second learning model 591, the first learning model 58 described above is used. Moreover, in the present embodiment, when generating the second learning model 591, the first learning model 58 is used. First, the method of detecting the presence or absence of an object using the second learning model 591 will be described below for each of the following first to fourth examples. A first example is when the predetermined site is a lumen region and the target lesion is dissection, protrusion, or thrombus. A second example is a case where the predetermined site is a region of a lumen and a blood vessel, and the target lesion is calcified or attenuating plaque. A third example is a case where the predetermined site is a region of a lumen and a blood vessel, and the structure as the object is a stent. A fourth example is a case where the predetermined site is the region of the lumen and the structure as the object is the guidewire. A method of generating the second learning model 591 will also be described.

FIG. 6 is a diagram showing a first example of detection of the presence or absence of an object by the information processing device 50. As shown in FIG. The computer program 57 acquires medical image data representing a cross-sectional image of a blood vessel (lumen organ) and inputs the acquired medical image data to the first learning model 581 . The first learning model 581 outputs the position data of the region of the lumen as illustrated in FIG. The computer program 57 identifies area data of the candidate object area based on the position data of the area of the lumen. Here, the candidate object region can be the region near the lumen boundary. The region data can be, for example, data obtained by extracting only the portion corresponding to the region near the lumen boundary from the medical image data. The computer program 57 generates weighted position information (weighted information) by giving different weights to the area data inside and outside the area near the lumen boundary. In the weighting process, for example, the data inside (including the boundary) the area near the lumen boundary (object candidate area) is set to 1, and the data outside the area near the lumen boundary is set to 0. It should be noted that a numerical value in the range of 0 to 1 may be used so that the weighting may gradually approach 0 from 1 as the distance from the lumen boundary increases outside the region near the lumen boundary.

The computer program 57 inputs the acquired medical image data and weight position information (weighting information) to the second learning model 591 . The second learning model 591 outputs the presence or absence of dissection, protrusion or thrombus (object).

There are, for example, two methods for inputting the weight position information to the second learning model 591 . Input processing of weight position information for the second learning model 591 will be described below.

FIG. 7 is a diagram showing a first example of input processing of weight position information. In the first example, mask processing is performed on the medical image data input to the second learning model 591 , and the masked medical image data is input to the second learning model 591 . Mask processing is performed using weight position information. Specifically, when the weighting within the candidate region (including the boundary) is 1 and the weighting outside the candidate region is 0, the medical image data after mask processing is such that the values of the pixels other than the corresponding pixels within the candidate region are becomes 0. In the example of FIG. 6, the candidate region is the region near the lumen boundary. The presence or absence of a rupture or thrombus can be detected. Dissection, protrusion, and thrombus are lesions that occur only in the surface layer of the lumen. Therefore, detection of the presence or absence of objects limited to the area near the lumen boundary can improve the accuracy of lesion detection. can.

As described above, in the first example of the weight position information input process, the computer program 57 acquires medical image data representing a cross-sectional image of a blood vessel, inputs the acquired medical image data to the first learning model, , the area data of the target candidate area is specified based on the acquired position data, and the specified area data is weighted differently inside and outside the target object candidate area, and the weighted position information is obtained. Generate. Then, the computer program 57 weights the acquired medical image data based on the generated weight position information, inputs the weighted medical image data (medical image data after mask processing) to the second learning model, The presence or absence of can be output.

Next, a method for generating the second learning model 591 corresponding to FIGS. 6 and 7 will be described.

FIG. 8 is a diagram showing a first example of a method for generating the second learning model 591. As shown in FIG. A method for generating the second learning model 591 can be, for example, as follows. First, the computer program 57 acquires second training data including medical image data representing cross-sectional images of blood vessels and data representing the presence or absence of objects. The second training data may be collected and stored in the server 200, and acquired from the server 200, for example. The computer program 57 inputs the medical image data included in the second training data to the first learning model 58, acquires the position data of the predetermined part output by the first learning model 58, and based on the acquired position data, Generate weight position information. In the case of FIGS. 6 and 7, the predetermined site is the area of the lumen. Using the second training data, the computer program 57 weights the acquired medical image data based on the weight position information, and uses the weighted medical image data (medical image data after mask processing) as the second training input data. The second learning model 591 is generated so as to output the presence/absence of the object when inputting to the learning model. Specifically, the first 2 The parameters of the learning model 591 may be adjusted. In the case of Figures 6 and 7, the object is a dissection, protrusion or thrombus.

In addition, in FIG. 6, the weighting process may be omitted. In this case, the computer program 57 converts the obtained medical image data and region data of the region near the lumen boundary (data obtained by extracting only the portion corresponding to the region near the lumen boundary out of the medical image data) to the second Different channels of the input layer 591a of the learning model 591 may be input to output the presence or absence of dissection, protrusion, or thrombus (object).

In this case, the method of generating the second learning model 591 can be, for example, as follows. Obtaining second training data including medical image data representing a cross-sectional image of a blood vessel and data representing the presence or absence of an object. The computer program 57 inputs the medical image data included in the second training data to the first learning model 58 and acquires the position data of the predetermined part output by the first learning model 58 . In the case of FIGS. 6 and 7, the predetermined site is the area of the lumen. The computer program 57 identifies region data of the candidate region identified based on the position data of the predetermined part output by the first learning model 58, and uses the obtained medical image data and region data as training input data for second learning. A second learning model 591 is generated to output the presence or absence of an object when input to the model. Specifically, the first 2 The parameters of the learning model 591 may be adjusted. In the case of Figures 6 and 7, the object is a dissection, protrusion or thrombus.

FIG. 9 is a diagram showing a second example of the weight position information input process. In the second example, medical image data is input to the second learning model 591, mask processing is performed on the feature map obtained as a result of convolution processing by the second learning model 591, and the feature map after mask processing is used. , the subsequent stage of the convolution processing is performed. Specifically, if the size of the weight position information is converted to the same size as the feature map, and the weighting within the candidate region (including the boundary) is 1 and the weighting outside the candidate region is 0, the feature map after masking is , the value of the pixels other than the corresponding pixels in the candidate area becomes zero. In the example of FIG. 6, since the candidate region is the region near the lumen boundary, the second learning model 591 calculates the value of the feature map corresponding to the region near the lumen boundary among the feature maps obtained by the convolution process. can be emphasized. Since a dissection, protrusion, or thrombus is a lesion that occurs only in the superficial layer of the lumen, enhancing the region near the lumen boundary can improve the detection accuracy of the lesion. It should be noted that the feature map of which convolutional layer is to be used may be appropriately selected from among the plurality of convolutional layers.

Next, a method for generating the second learning model 591 corresponding to FIGS. 6 and 9 will be described.

FIG. 10 is a diagram showing a second example of the method of generating the second learning model 591. As shown in FIG. A method for generating the second learning model 591 can be, for example, as follows. First, the computer program 57 acquires second training data including medical image data representing cross-sectional images of blood vessels and data representing the presence or absence of objects. The second training data may be collected and stored in the server 200, and acquired from the server 200, for example. The computer program 57 inputs the medical image data included in the second training data to the first learning model 58, acquires the position data of the predetermined part output by the first learning model 58, and based on the acquired position data, Generate weight position information. In the case of FIGS. 6 and 7, the predetermined site is the area of the lumen. The computer program 57 weights the convolution-processed feature map based on the weight position information. When the acquired medical image data is input to the second learning model 591 as training input data, the computer program 57 generates the second learning model 591 so as to output the presence or absence of the object. Specifically, the first 2 The parameters of the learning model 591 may be adjusted. Here, the object is dissection, protrusion or thrombus.

FIG. 11 is a diagram showing a second example of detection of the presence or absence of an object by the information processing device 50. As shown in FIG. The computer program 57 acquires medical image data representing a cross-sectional image of a blood vessel (lumen organ) and inputs the acquired medical image data to the first learning model 582 . The first learning model 582 outputs position data for each of the lumen region and the blood vessel region, as illustrated in FIG. The computer program 57 identifies the area data of the object candidate area based on the position data of the lumen area and the blood vessel area. Here, the object candidate area can be an area from the vicinity of the lumen boundary to the vicinity of the blood vessel boundary. The region data can be, for example, data obtained by extracting only the portion corresponding to the region from the vicinity of the lumen boundary to the vicinity of the blood vessel boundary from the medical image data. The computer program 57 generates weight position information (weighting information) by giving different weights to the area data inside and outside the area from the vicinity of the lumen boundary to the vicinity of the blood vessel boundary. In the weighting process, for example, the data within the region (object candidate region) from the vicinity of the lumen boundary to the vicinity of the blood vessel boundary (including the boundary) is set to 1, and the data outside the region from the vicinity of the lumen boundary to the vicinity of the blood vessel boundary is set to 1. 0. It should be noted that a numerical value in the range of 0 to 1 may be used so as to gradually approach 0 from 1 as the distance from the vascular boundary increases outside the region from the vicinity of the lumen boundary to the vicinity of the vascular boundary.

The computer program 57 inputs the acquired medical image data and weight position information (weighting information) to the second learning model 591 . A second learning model 591 can output the presence or absence of calcified or attenuating plaque (object). The process of inputting the weight position information to the second learning model 591 is the same as in the case of FIG. 7 or 9, so the description is omitted.

Since the example in FIG. 11 is the region from the vicinity of the lumen boundary to the vicinity of the blood vessel boundary, the second learning model 591 limits the entire medical image to the region from the vicinity of the lumen boundary to the vicinity of the blood vessel boundary, The presence or absence of calcified or attenuating plaque of interest can be detected. Calcified or attenuated plaque is a lesion that occurs from the surface layer of the lumen to the blood vessel. Accuracy can be improved.

Also, the second learning model 591 can emphasize the value of the feature map corresponding to the region from the vicinity of the lumen boundary to the vicinity of the blood vessel boundary among the feature maps obtained by the convolution processing. Calcified or attenuated plaque is a lesion that develops from the superficial layer of the lumen to the blood vessel, so by emphasizing the region from the vicinity of the lumen boundary to the vicinity of the blood vessel boundary, the lesion detection accuracy can be improved.

FIG. 12 is a diagram showing a third example of detection of the presence or absence of an object by the information processing device 50. In FIG. The first learning model 581 is the same as in FIG. The computer program 57 identifies the area data of the object candidate area based on the position data of the lumen area and the blood vessel area. Here, the object candidate area can be an area from the inside of the lumen to the vicinity of the blood vessel boundary. The region data can be, for example, data obtained by extracting only the portion corresponding to the region from the inside of the lumen to the vicinity of the blood vessel boundary out of the medical image data. The weighting process and weighting position information are the same as in the case of FIG. 11, so description thereof will be omitted. The computer program 57 inputs the acquired medical image data and weight position information (weighting information) to the second learning model 591 . The second learning model 591 can output the presence or absence of a stent (object). The process of inputting the weight position information to the second learning model 591 is the same as in the case of FIG. 7 or 9, so the description is omitted.

Since the stent is a structure that extends from the inside of the lumen to the boundary of the blood vessel, the presence or absence of the target object can be detected by limiting the region from the inside of the lumen to the vicinity of the boundary of the blood vessel, or emphasizing the region. The structure detection accuracy can be improved.

FIG. 13 is a diagram showing a fourth example of detection of the presence or absence of an object by the information processing device 50. As shown in FIG. The first learning model 581 is the same as in FIG. The computer program 57 identifies area data of the candidate object area based on the position data of the area of the lumen. Here, the candidate object area can be an area from the inside of the lumen to the vicinity of the lumen boundary. The region data can be, for example, data obtained by extracting only a portion corresponding to the region from the inside of the lumen to the vicinity of the boundary of the lumen out of the medical image data. The weighting process and weighting position information are the same as in the case of FIG. 11, so description thereof will be omitted. The computer program 57 inputs the acquired medical image data and weight position information (weighting information) to the second learning model 591 . The second learning model 591 can output the presence or absence of the guidewire (object). The process of inputting the weight position information to the second learning model 591 is the same as in the case of FIG. 7 or 9, so the description is omitted.

Since the guidewire is a structure that extends from the inside of the lumen to the boundary of the lumen, it is possible to detect the presence or absence of an object by limiting the region from the inside of the lumen to the boundary of the lumen, or emphasizing that region. Therefore, the detection accuracy of the structure can be improved.

As described above, the computer program 57 acquires medical image data representing a cross-sectional image of a blood vessel (a hollow organ), and when inputting medical image data representing a cross-sectional image of the blood vessel, the computer program 57 obtains position data of a predetermined portion of the blood vessel. The obtained medical image data is input to the first learning model that outputs the position data of the predetermined part, and the medical image data showing the cross-sectional image of the blood vessel and the position data of the predetermined part are input. The acquired medical image data and the position data of the predetermined part can be input to the second learning model that outputs the presence or absence of the object, and the computer can be caused to execute processing for outputting the presence or absence of the object.

More specifically, the computer program 57 identifies the area data of the object candidate area based on the acquired position data of the predetermined part, and generates the medical image data showing the cross-sectional image of the blood vessel and the area data of the object candidate area. is input, the acquired medical image data and the region data of the specified target candidate region are input to the second learning model that outputs the presence or absence of the target object in the blood vessel, and the processing of outputting the presence or absence of the target object. can be run on a computer.

In this case, the region data is weighted differently inside and outside the foreground object candidate region to generate weighted position information, the acquired medical image data is weighted based on the generated weighting information, and the weighted medical image is obtained. The data may be input to the second learning model, and the presence or absence of the object may be output.

In addition, the region data is weighted differently inside and outside the foreground object candidate region to generate weighted position information, and based on the generated weighting information, the feature map after convolution processing for the acquired medical image data is weighted. Then, the presence or absence of the object may be output using the weighted feature map.

Next, a processing procedure of the information processing device 50 will be described.

FIG. 14 is a diagram showing a first example of object detection processing by the information processing device 50 . The processing of FIG. 14 corresponds to the case of FIG. In the following description, for the sake of convenience, the main body of processing is assumed to be the control unit 51 . The control unit 51 acquires a plurality of pieces of medical image data (S11). Here, the plurality of medical images can be images obtained by one pullback operation. The control unit 51 inputs the acquired medical image data to the first learning model and acquires the position data of the predetermined part (S12).

The control unit 51 identifies an object candidate area (candidate area) based on the acquired position data (S13), and generates weighted position information based on the area data of the object candidate area (S14). The control unit 51 performs mask processing on the obtained medical image data using the weighted position information (S15), inputs the medical image data after the mask processing to the second learning model, and outputs the presence or absence of the target object (S16 ) and terminate the process.

FIG. 15 is a diagram showing a second example of object detection processing by the information processing device 50 . The processing of FIG. 15 corresponds to the case of FIG. The control unit 51 acquires a plurality of pieces of medical image data (S21). Here, the plurality of medical images can be images obtained by one pullback operation. The control unit 51 inputs the acquired medical image data to the first learning model and acquires the position data of the predetermined part (S22).

The control unit 51 identifies an object candidate area (candidate area) based on the acquired position data (S23), and generates weighted position information based on the area data of the object candidate area (S24). The control unit 51 inputs the acquired medical image data to the second learning model, extracts a feature map (S25), and uses the weight position information to mask the extracted feature map (S26). The control unit 51 inputs the masked feature map to the second learning model, outputs the presence or absence of the object (S27), and ends the process.

FIG. 16 is a diagram showing an example of processing for generating the first learning model 58 by the information processing device 50. As shown in FIG. The control unit 51 acquires first training data including medical image data and position data of a predetermined part (S31). The first training data is data relating to a plurality of medical images and the positions of predetermined parts in the medical images, and can include the amount of data necessary for machine learning. The control unit 51 sets the initial values of the parameters of the first learning model 58 (S32).

The control unit 51 inputs medical image data to the first learning model 58 based on the first training data (S33). The control unit 51 adjusts the parameters so that the value of the loss function based on the position data of the predetermined part output by the first learning model 58 and the position data of the predetermined part included in the first training data is minimized ( S34).

The control unit 51 determines whether the value of the loss function is within the allowable range (S35), and if the value of the loss function is not within the allowable range (NO in S35), continues the processing from step S33 onward. If the value of the loss function is within the allowable range (YES in S35), the control unit 51 stores the generated first learning model 58 in the storage unit 56 (S36), and terminates the process.

FIG. 17 is a diagram showing a first example of processing for generating the second learning model 59 by the information processing device 50. As shown in FIG. The processing of FIG. 17 corresponds to the case of FIG. The control unit 51 acquires second training data including medical image data and data indicating the presence or absence of an object (S41). The second training data is a plurality of medical images and data regarding the presence or absence of objects in the medical images, and can include the amount of data necessary for machine learning. The control unit 51 sets the initial values of the parameters of the second learning model 59 (S42).

The control unit 51 inputs the medical image data to the first learning model 58, acquires the position data of the predetermined part (S43), and specifies the object candidate region (candidate region) based on the acquired position data (S44). ), the weighted position information is generated based on the area data of the object candidate area (S45). The control unit 51 uses the weight position information to mask the obtained medical image data (S46), and inputs the masked medical image data to the second learning model (S47).

The control unit 51 sets parameters so that the value of the loss function based on the data indicating the presence or absence of the object output by the second learning model 59 and the data indicating the presence or absence of the object included in the second training data is minimized. is adjusted (S48). The control unit 51 determines whether or not the value of the loss function is within the allowable range (S49), and if the value of the loss function is not within the allowable range (NO in S49), continues the processing from step S47. If the value of the loss function is within the allowable range (YES in S49), the control unit 51 stores the generated second learning model 59 in the storage unit 56 (S50), and terminates the process.

FIG. 18 is a diagram showing a second example of processing for generating the second learning model 59 by the information processing device 50. As shown in FIG. The processing of FIG. 18 corresponds to the case of FIG. The control unit 51 acquires second training data including medical image data and data indicating the presence or absence of an object (S61). The second training data is a plurality of medical images and data regarding the presence or absence of objects in the medical images, and can include the amount of data necessary for machine learning. The control unit 51 sets the initial values of the parameters of the second learning model 59 (S62).

The control unit 51 inputs the medical image data to the first learning model 58, acquires the position data of the predetermined part (S63), and specifies the object candidate region (candidate region) based on the acquired position data (S64). ), the weighted position information is generated based on the area data of the object candidate area (S65). The control unit 51 inputs the medical image data to the second learning model 59 and extracts the feature map after convolution processing (S66).

Using the weight position information, the control unit 51 performs mask processing on the extracted feature map (S67). The parameters are adjusted so that the value of the loss function based on the data indicating the presence or absence of is minimized (S68).

The control unit 51 determines whether the value of the loss function is within the allowable range (S69), and if the value of the loss function is not within the allowable range (NO in S69), continues the processing from step S67. If the value of the loss function is within the allowable range (YES in S69), the control unit 51 stores the generated second learning model 59 in the storage unit 56 (S70), and ends the process.

As described above, the computer program 57 acquires first training data including medical image data representing a cross-sectional image of a blood vessel (lumen organ) and position data of a predetermined portion of the blood vessel, and acquires the acquired first training data. medical image data representing a cross-sectional image of a blood vessel, generating a first learning model so as to output position data of a predetermined portion of the blood vessel when medical image data representing a cross-sectional image of the blood vessel is input based on; and obtaining second training data including data indicating the presence or absence of the target object of the blood vessel, and based on the obtained second training data, inputting medical image data to the first learning model to obtain position data of the predetermined part. Then, when medical image data representing a cross-sectional image of a blood vessel and acquired position data of a predetermined part are input based on the second training data, second learning is performed to output the presence or absence of an object in the blood vessel. A computer can be caused to perform the process of generating the model.

In the present embodiment, as the second learning model 59, a learning model that outputs the presence or absence of dissection, protrusion, or thrombus as a lesion, a learning model that outputs the presence or absence of calcification or attenuating plaque as a lesion, a structure A learning model that outputs the presence or absence of a stent as a structure and a learning model that outputs the presence or absence of a guidewire as a structure may be provided separately, and several learning models are integrated into one second learning model 59. Alternatively, all learning models may be configured as one second learning model 59 .

In the present embodiment, the first learning models 581 and 582 may be separately provided as the first learning model 58, and the first learning model 582 may be substituted for the first learning model 581 as well.

In the above-described embodiment, the information processing device 50 is configured to detect the presence or absence of a target object and generate a learning model, but the present invention is not limited to this. For example, the information processing apparatus 50 may be a client apparatus, the learning model generation process may be performed by an external server, and the learning model may be acquired from the server. Alternatively, the detection of the presence or absence of the target object and the generation of the learning model may be performed by an external server, and the information processing apparatus may acquire the detection result of the presence or absence of the target object from the server.

1 communication network 10 catheter 20 MDU
30 display device 40 input device 50 information processing device 51 control unit 52 communication unit 53 interface unit 54 recording medium reading unit 541 recording medium 55 memory 56 storage unit 57 computer program 58, 581, 582 first learning model 581a, 582a input layer 581b , 582b intermediate layer 581c, 582c output layer 59, 591 second learning model 591a input layer 591b intermediate layer 591c output layer

Claims (12)

to the computer,
acquiring medical image data showing a cross-sectional image of a hollow organ;
When medical image data representing a cross-sectional image of a hollow organ is input, the acquired medical image data is input to a first learning model that outputs position data of a predetermined portion of the hollow organ to determine the position of the predetermined portion. get the data,
When medical image data representing a cross-sectional image of a hollow organ and position data of a predetermined portion of the hollow organ are input, a second learning model that outputs the presence or absence of an object in the hollow organ is provided with the acquired medical image data. inputting the image data and the obtained position data of the predetermined part, and outputting the presence or absence of the object;
A computer program that causes a process to be performed. to the computer,
identifying region data of a target object candidate region based on the acquired position data of the predetermined portion;
Acquired medical image data in the second learning model for outputting presence/absence of an object in the hollow organ when medical image data representing a cross-sectional image of the hollow organ and region data of the object candidate region are input. and inputting the region data of the identified object candidate region and outputting the presence or absence of the object;
2. The computer program according to claim 1, causing a process to be executed. to the computer,
weighting the region data differently inside and outside the object candidate region to generate weighting information;
3. The computer program according to claim 2, causing a process to be executed. to the computer,
weighting the acquired medical image data based on the generated weighting information;
inputting the weighted medical image data into the second learning model;
output the presence or absence of an object,
4. The computer program according to claim 3, causing a process to be executed. to the computer,
causing the second learning model to perform convolution processing;
Weighting the feature map after the convolution process based on the generated weighting information,
output presence/absence of objects using weighted feature maps;
4. The computer program according to claim 3, causing a process to be executed. The predetermined site includes a lumen area,
the object includes at least one of dissection, protrusion, and thrombus;
Computer program according to any one of claims 1 to 5. The predetermined site includes regions of lumens and blood vessels,
the object comprises at least one of calcified and attenuating plaque;
Computer program according to any one of claims 1 to 6. The predetermined site includes a lumen area,
the object comprises a guidewire;
Computer program according to any one of claims 1 to 7. The predetermined site includes regions of lumens and blood vessels,
the object comprises a stent;
Computer program according to any one of claims 1 to 8. acquiring medical image data showing a cross-sectional image of a hollow organ;
When medical image data representing a cross-sectional image of a hollow organ is input, the acquired medical image data is input to a first learning model that outputs position data of a predetermined portion of the hollow organ to determine the position of the predetermined portion. get the data,
When medical image data representing a cross-sectional image of a hollow organ and position data of a predetermined portion of the hollow organ are input, a second learning model that outputs the presence or absence of an object in the hollow organ is provided with the acquired medical image data. inputting the image data and the obtained position data of the predetermined part, and outputting the presence or absence of the object;
Information processing methods. a first acquisition unit that acquires medical image data representing a cross-sectional image of a hollow organ;
inputting the medical image data obtained by the first obtaining unit into a first learning model for outputting position data of a predetermined portion of the hollow organ when medical image data representing a cross-sectional image of the hollow organ is inputted; a second acquisition unit that acquires position data of the predetermined part by
When medical image data representing a cross-sectional image of a hollow organ and position data of a predetermined portion of the hollow organ are input, a second learning model for outputting the presence or absence of an object in the hollow organ is provided with the first and an output unit that inputs the medical image data acquired by the acquisition unit and the position data of the predetermined part acquired by the second acquisition unit, and outputs the presence or absence of an object. acquiring first training data including medical image data representing a cross-sectional image of a hollow organ and position data of a predetermined portion of the hollow organ;
generating a first learning model based on the first training data so as to output position data of a predetermined portion of the hollow organ when medical image data representing a cross-sectional image of the hollow organ is input;
acquiring second training data including medical image data representing a cross-sectional image of a hollow organ and data representing the presence or absence of an object in the hollow organ;
inputting medical image data into the first learning model based on the second training data to acquire position data of a predetermined part;
Based on the second training data, the presence or absence of an object in the hollow organ is output when medical image data representing a cross-sectional image of the hollow organ and the acquired position data of the predetermined site are input. generate a second learning model at
Learning model generation method.

Images