JP2018089301A - Organism image processing device, output image production method, learning result production method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an organism image processing device for generating an organism image from which a target is removed from an organism image including the target.SOLUTION: An organism image processing device 1 comprises: a reception part 11 for receiving a training image for input which is an organism image including a target, a training image for output which is an organism image including no target, and an input image which is an organism image including the target; a learning part 12 for learning a neural network which performs image processing for generating the organism image including no target from the organism image including the target, using the training image for input and training image for output; and an image processing part 13 for generating an output image obtained by removing the target from the input image using image processing of the neural network which is a learning result, in which the neural network has plural convolution layers.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a biological image processing apparatus that performs processing related to a biological image.

  Conventionally, digital differential angiography (hereinafter sometimes referred to as “DSA” (Digital Subtraction Angiography)) has been used to grasp the vascular structure of a subject. In DSA, X-ray images of a subject are acquired before and after contrast medium administration, and a mask image that is an X-ray image before contrast medium administration is subtracted from an original image that is an X-ray image after contrast medium administration. A DSA image for rendering a blood vessel image is obtained. In recent years, three-dimensional DSA employing a three-dimensional imaging method has been performed (see, for example, Patent Document 1).

  In DSA, there is a problem that an appropriate DSA image cannot be acquired when a contrast target or an imaging system moves. For example, when photographing a moving organ such as the heart, it is difficult to prepare a mask image that is the same organ position as the original image because of position movement due to pulsation and respiration. However, there is a problem in that movement due to body position and exercise is difficult, resulting in artifacts in the DSA image. In addition, there is a problem that an appropriate DSA image cannot be acquired even when the imaging system is moved, such as when the imaging system is moved to a range where a mask image is not acquired.

  In the case of a DSA image of the heart, a plurality of mask images corresponding to one cardiac cycle are prepared in advance, a mask image that approximates the photographed original image is selected by a pattern matching method, and the selected mask is selected. A method for acquiring a DSA image using an image is known. For relatively small movements of the subject, pixel shift for manually moving the position of the mask image, warping for moving the position of the mask image linearly or non-linearly by calculation (for example, Patent Document 2).

  In addition, in order to avoid the need to take such a mask image before contrast, a method for obtaining a mask image from the original image by separately obtaining a blurred image, or taking a mask image when photographing the original image The method of doing is also developed (for example, refer patent documents 3 and 4).

JP 2010-193975 A Japanese Unexamined Patent Publication No. 2015-213536 Japanese Patent Laid-Open No. 10-322597 JP 2002-135655 A

However, in a conventional DSA that captures a mask image before contrast enhancement, if there is a large movement in the subject or the imaging system, the background between the original image and the mask image even if pattern matching, pixel shift, warping, or the like is performed. Will not match and artifacts will occur in the DSA image.
In addition, when a mask image is acquired before contrast enhancement, there is a problem that the examination time becomes longer and the load on the subject increases. For example, in the three-dimensional DSA, since the mask image is taken, there is a problem that the inspection time is approximately doubled, and the time for restraining the subject becomes long. In addition, even in DSA that does not capture a mask image before contrast enhancement, a configuration for capturing a mask image, an optical configuration for generating a mask image from an original image, etc. are provided in order to generate a blurred image. There is also a problem that the apparatus becomes more complicated or larger as it becomes necessary.

Therefore, in DSA, it is desired to obtain an appropriate DSA image without taking a mask image before contrast enhancement, and to obtain a mask image without using an optical configuration or the like. There was a request.
Generally speaking, there is a desire to be able to generate a biological image (for example, a DSA mask image) from which a target object is removed directly from a biological image including the target object (for example, contrasted blood vessels). It was.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide a biological image processing apparatus that can generate a biological image in which a target object is directly removed from a biological image including the target object. .

In order to achieve the above object, a biological image processing apparatus according to the present invention includes an input training image that is a biological image including a target object, an output training image that is a biological image that does not include the target object, and a biological image including the target object. An input image that is an input image, and an input training image and an output training image received by the reception unit, and performing image processing that generates a biological image that does not include the target object from the biological image that includes the target object A learning unit that learns a neural network, and an image processing unit that generates an output image obtained by removing an object from an input image received by the receiving unit, using image processing of the neural network that is a learning result of the learning unit. The neural network includes a plurality of convolution layers.
With such a configuration, it becomes possible to learn a neural network having a plurality of convolution layers and generate a biological image in which the target object is removed from the biological image including the target object using the learning result. . As a result, for example, a mask image can be generated from an original image of DSA, or a biological image obtained by removing a stent or the like from a biological image including a stent or a coil can be generated.

The biological image processing apparatus according to the present invention includes a receiving unit that receives an input image that is a biological image including a target object, and a learning result of a neural network that generates a biological image that does not include the target object from the biological image including the target object. An image processing unit that generates an output image obtained by removing an object from an input image received by the receiving unit using image processing, and the neural network has a plurality of convolutional layers. .
With such a configuration, it is possible to generate a biological image obtained by removing the target object from the biological image including the target object, using the learning result of the neural network having a plurality of convolution layers. As a result, for example, a mask image can be generated from an original image of DSA, and a biological image in which a stent or the like is removed from a biological image including a stent or a coil can be generated.

  In the biological image processing apparatus according to the present invention, the intermediate layer and the output layer in the neural network may all be convolutional layers.

In the biological image processing apparatus according to the present invention, the biological image including the target object is an angiographic image that is an X-ray image of the biological body administered with the contrast agent, and the biological image not including the target object is administered with the contrast agent. It is an X-ray image of a living body that is not, and the output image may be a mask image used in digital differential angiography.
With such a configuration, a mask image used in DSA can be generated from an angiographic image. As a result, X-ray imaging for the mask image becomes unnecessary, and the exposure dose of the subject can be reduced. In addition, since the time for photographing the mask image is not required, the subject's restraint time is shortened.

In the biological image processing apparatus according to the present invention, the first intermediate layer in the neural network may be a convolution layer using a filter whose side corresponds to a length of 3 mm or more.
With such a configuration, in the neural network, the contrasted blood vessel that is the object can be appropriately removed from the angiographic image.

Further, in the biological image processing apparatus according to the present invention, a subtraction processing unit that subtracts a mask image that is an output image generated from an angiographic image by an image processing unit from an angiographic image that is an input image received by the receiving unit. And a subtracted image output unit that outputs the subtracted image subtracted by the subtracting unit.
With such a configuration, for example, a subtracted image (DSA image) in which backgrounds other than blood vessels are appropriately removed can be acquired even if the organ is in motion or the imaging system is moved greatly. .

The output image manufacturing method according to the present invention includes a first receiving step of receiving an input training image that is a biological image including a target object, and an output training image that is a biological image not including the target object; A learning step for learning a neural network that performs image processing for generating a biological image not including a target object from a biological image including a target object using the input training image and the output training image received in the receiving step; A second reception step that receives an input image that is a biological image including an object, and an image processing of a neural network that is a learning result in the learning step, and an object is removed from the input image received in the second reception step And an image processing step for generating an output image, wherein the neural network includes a plurality of convolution layers. It has been, is intended.
With such a configuration, it is possible to produce an output image obtained by removing the target object from the input image using the learning result of the neural network.

In addition, the learning result manufacturing method according to the present invention is received in the receiving step for receiving the training image for input that is a biological image including the target object and the training image for output that is a biological image not including the target object. A learning step of acquiring a learning result by learning a neural network that performs image processing for generating a biological image not including a target object from a biological image including a target object using the input training image and the output training image; And the neural network has a plurality of convolutional layers.
With such a configuration, it is possible to manufacture a learning result of a neural network used for image processing for generating an output image obtained by removing an object from an input image.

  According to the biological image processing apparatus and the like according to the present invention, it is possible to generate a biological image in which a target object is removed from a biological image including the target object, using a neural network having a plurality of convolution layers.

The block diagram which shows the structure of the biometric image processing apparatus by embodiment of this invention The flowchart which shows operation | movement of the biometric image processing apparatus by the embodiment The figure which shows an example of the model of the neural network in the embodiment The figure which shows an example of the information of each layer in the embodiment The figure which shows an example of the information of each layer in the embodiment The figure which shows an example of the information of each layer in the embodiment The figure which shows an example of the training data in the embodiment The figure which shows an example of the angiographic image and mask image in the embodiment The figure which shows an example of the angiographic image and mask image in the embodiment The figure which shows an example of the angiographic image and mask image in the embodiment The figure which shows an example of the DSA image by the conventional DSA image and the same embodiment The figure which shows an example of the DSA image by the conventional DSA image and the same embodiment The figure which shows an example of the DSA image by the conventional DSA image and the same embodiment The figure which shows an example of the DSA image by the conventional DSA image and the same embodiment Schematic diagram showing an example of the appearance of the computer system in the embodiment The figure which shows an example of a structure of the computer system in the embodiment

  Hereinafter, a biological image processing apparatus according to the present invention will be described using embodiments. In the following embodiments, components and steps denoted by the same reference numerals are the same or equivalent, and repetitive description may be omitted. The biological image processing apparatus according to the present embodiment learns a neural network having a plurality of convolution layers, and generates an output image obtained by removing an object from a biological image using the learning result.

  FIG. 1 is a block diagram showing a configuration of a biological image processing apparatus 1 according to the present embodiment. The biological image processing apparatus 1 according to the present embodiment includes a receiving unit 11, a learning unit 12, an image processing unit 13, a storage unit 14, a subtraction processing unit 15, and a subtraction image output unit 16. In the present embodiment, first, a case where a subtracted image that is a DSA image is generated by the biological image processing apparatus 1 will be described, and then other cases will be described.

  The reception unit 11 receives training data used for learning and actual data used for actual processing using a learning result. The training data includes an input training image that is a biological image including the target object, and an output training image that is a biological image not including the target object. Usually, many sets of input training images and output training images are received by the receiving unit 11. It is assumed that the input training image and the output training image included in a certain group are the same images, that is, images related to the same position of the living body, except whether or not the object is included. The actual data is an input image that is a biological image including the object. Here, when the target is a contrast-enhanced blood vessel and the living body image is an X-ray image, that is, the living body image including the target is an angiographic image that is an X-ray image of a living body administered with a contrast agent. A case where the living body image that does not include an object is an X-ray image of a living body that is not administered with a contrast agent will be described.

  The reception unit 11 may receive data transmitted via a wired or wireless communication line, for example, and read data read from a predetermined recording medium (for example, an optical disk, a magnetic disk, a semiconductor memory, etc.). It may be accepted. The reception unit 11 may or may not include a device (for example, a modem or a network card) for reception. The receiving unit 11 may be realized by hardware, or may be realized by software such as a driver that drives a predetermined device.

  The learning unit 12 uses the input training image and the output training image received by the receiving unit 11 to learn a neural network that performs image processing for generating a biological image that does not include the target object from the biological image that includes the target object. To do. A neural network that performs image processing performs a predetermined calculation on each pixel value of an input image and outputs each pixel value of an output image. By the learning, the learning unit 12 acquires a learning result. In this learning, each pixel value (parameter value) of each filter in a convolutional neural network (CNN) is calculated. The neural network has a plurality of convolution layers. Therefore, since the neural network has at least one intermediate layer (hidden layer), it can be said that learning of the neural network is deep learning. Note that the neural network may or may not have a pooling layer, may have a fully-connected layer, or It may not have. In the present embodiment, a case will be mainly described in which the intermediate layer and the output layer in the neural network are all convolutional layers and do not have the pooling layer and the total coupling layer. The neural network of the present embodiment may have, for example, three or more convolution layers. In the following embodiments, the case where the neural network is composed of three and five convolutional layers will be described in detail. The number of intermediate layers in the neural network may be one layer, two layers, or three or more layers, for example. Further, the convolutional layer included in the neural network may be two layers, four layers, or six layers or more. In the image processing according to the present embodiment, it is usually preferable that the number of pixels of the input image and the output image are the same, and therefore padding may be appropriately performed in each layer of the neural network. The padding may be, for example, zero padding, may be padding that extrapolates the outermost pixel value of the image, or may be padding that is a pixel value folded at each side of the image. In addition, the stride in each layer is not limited, but for example, the stride in the convolution layer is preferably 1, and when the neural network has a pooling layer, the stride of the pooling layer may be 2 or more. Is preferred. In each layer, a bias may or may not be used. Whether to use a bias may be determined independently for each layer. The bias may be, for example, a layer-by-layer bias or a filter-by-filter bias. In the former case, one bias is used in each layer, and in the latter case, one or more biases (the same number as the filter) are used in each layer. When a bias is used in the convolution layer, a result obtained by multiplying each pixel value by the filter parameter and adding the bias is input to the activation function.

  Each setting in the neural network may be as follows. The activation function may be, for example, a ReLU (normalized linear function), a sigmoid function, or another activation function. ReLU is a function where f (u) = max (u, 0). In the learning in the learning unit 12, the error back-propagation method may be used or the mini-batch method may be used as in the normal deep learning. The loss function (error function) may be a mean square error. The number of epochs is not particularly limited, but may be 100 to 5000, for example.

  The image processing unit 13 generates an output image obtained by removing the target object from the input image received by the receiving unit 11 using the image processing of the neural network that is the learning result by the learning unit 12. This image processing is performed by operating a neural network using the learning result parameters. By the image processing by the image processing unit 13, a mask image used for DSA, that is, an image in which the contrasted blood vessels are removed is generated from an input image that is an angiographic image. The generated mask image is accumulated in the storage unit 14. Note that when a plurality of angiographic images are input, the image processing unit 13 may generate a mask image for each angiographic image. In the conventional DSA, one mask image is usually used for a plurality of angiographic images, but the biological image processing apparatus 1 according to the present embodiment generates a mask image corresponding to each angiographic image. Therefore, DSA with higher accuracy is possible. In the learning by the learning unit 12, when learning about a certain part of the living body (for example, the coronary artery of the heart) is performed, the image processing unit 13 selects the target contrast blood vessel from the input image of the part. It is preferable to perform image processing to be removed. In the learning by the learning unit 12, when learning is performed on a plurality of parts of the living body (for example, the coronary artery and the head of the heart), the image processing unit 13 applies the input image of any part of the learning to the input image. On the other hand, image processing for removing a contrasting blood vessel as a target may be performed.

  Note that when performing neural network image processing, processing may be performed on a divided image obtained by dividing an image to be processed. As in Example 1 described later, for example, a 64 × 64 divided image obtained by dividing a 512 × 512 input image is subjected to neural network image processing, and the divided images are combined after the image processing. The output image may be configured by. In this case, learning is performed using the divided images. In this case, the divided image after division (for example, 64 × 64 image) may be considered as the input image, or the image before division (for example, 512 × 512 image) is the input image. You may think that In the latter case, it may be considered that the image processing in the image processing unit 13 includes the division of the input image and the combination of the images after the image processing by the neural network. In this embodiment, this case will be mainly described. In addition, when dividing | segmenting an image, you may divide | segment so that duplication may arise or may divide | segment so that duplication may not arise. The former is more preferable from the viewpoint that it is difficult to produce lines at the divided portions in the combined image.

  In the storage unit 14, the generated mask image (output image) is stored. For example, the mask image may be accumulated by the image processing unit 13. The storage in the storage unit 14 may be temporary storage in a RAM or the like, or may be long-term storage. The storage unit 14 can be realized by a predetermined recording medium (for example, a semiconductor memory, a magnetic disk, an optical disk, etc.).

  The subtraction processing unit 15 subtracts a mask image that is an output image generated from the angiographic image by the image processing unit 13 from the angiographic image that is the input image received by the receiving unit 11. Therefore, in the subtraction process in DSA, the subtraction process is performed using a certain angiographic image and a mask image generated from the angiographic image. As a result, one DSA image can be acquired from one angiographic image without taking a mask image. When a plurality of input images along the time series are received by the reception unit 11, the subtraction processing unit 15 performs a subtraction process on each input image to obtain a plurality of DSA images along the time series. It may be generated.

  The subtraction image output unit 16 outputs the subtraction image subjected to the subtraction process by the subtraction processing unit 15. Here, the subtraction image is a DSA image. The output by the subtracted image output unit 16 may be, for example, displayed on a display device (for example, a liquid crystal display or an organic EL display), transmitted via a communication line to a predetermined device, or stored in a recording medium. Or it may be delivered to other components. Note that the subtracted image output unit 16 may or may not include a device that performs output (for example, a display device or a communication device). The subtracted image output unit 16 may be realized by hardware, or may be realized by software such as a driver that drives these devices.

Next, the operation of the biological image processing apparatus 1 will be described using the flowchart of FIG.
(Step S101) The receiving unit 11 determines whether a plurality of sets of input training images and output training images have been received. If a plurality of sets of input training images and output training images are received, the process proceeds to step S102. If not, the process of step S101 is repeated until they are received.

  (Step S <b> 102) The learning unit 12 learns a neural network using the input training image and the output training image received by the receiving unit 11. The learning unit 12 may accumulate a plurality of parameters, which are learning results obtained by this learning, in a recording medium (not shown).

  (Step S103) The receiving unit 11 determines whether an input image that is image data to be processed has been received. This input image is an angiographic image. If an input image is received, the process proceeds to step S104. If not, the process of step S103 is repeated until the input image is received.

  (Step S <b> 104) The image processing unit 13 uses the neural network that is a learning result by the learning unit 12 to generate an output image in which the contrasted blood vessels that are target objects are removed from the input image received by the receiving unit 11. The output image is a mask image used in DSA.

  (Step S105) The subtraction processing unit 15 generates a subtraction image by subtracting the output image generated in step S104 from the input image received by the reception unit 11 in step S103. This subtracted image is a DSA image.

  (Step S106) The subtraction image output unit 16 outputs the subtraction image generated in step S105. Then, the process returns to step S103.

  In the flowchart of FIG. 2, a case where a plurality of input images are continuously received is shown. However, in a case where this is not the case, for example, when only one input image is received, the processing after step S106 is performed. The series of processing may be terminated. In the flowchart of FIG. 2, the process is ended by power-off or a process end interrupt.

Next, generation of a DSA image by the biological image processing apparatus 1 according to the present embodiment will be described using an example.
[Example 1]
In this embodiment, a case where the neural network has three convolution layers will be described. FIG. 3 is a diagram showing the configuration of the neural network in this embodiment. The input image of the neural network of this embodiment is a one-channel image of 64 × 64 pixels. Each pixel is 8 bits. In the first convolution layer (conv1), 64 sets of 64 × 64 images were generated using 64 sets of one 9 × 9 filter. In conv1, zero padding was performed to make the input and output image sizes the same. In the second convolutional layer (conv2), 32 sets of 64 × 64 images were generated using 32 sets of 64 1 × 1 filters. In the third convolutional layer (conv3), a set of 32 filters of 5 × 5 was used to generate a 64 × 64 image of 1 channel. Even in conv3, zero padding was performed to make the input and output image sizes the same. The output of conv3 is the final output in the neural network, that is, the output image (mask image). In the image processing by CNN, the size of the filter generally increases as the layers become deeper as in the first layer and the second layer, but the neural network of the present embodiment has the first layer and the second layer. As is evident, it has adjacent convolutional layers that reduce the size of the deeper layer filter. In this example, the stride was set to 1 and no bias was used.

  ReLU was used as the activation function for each layer. The information regarding each layer of the present example is as shown in FIG. 4A. In the first layer, 64 sets of filters for one channel having 81 (= 9 × 9) parameters are required, so the number of parameters that must be determined by learning is 5184 (= 81 × 1 × 64). become. In the second layer, 32 sets of filters for 64 channels having one parameter are required, and the number of parameters that must be determined by learning is 2048 (= 1 × 64 × 32). Similarly, in the third layer, the number of parameters is 800. Therefore, in this embodiment, the total number of parameters determined by learning is 8032. The output map size W × W × K in FIG. 4A indicates that the number of vertical and horizontal pixels of the image is W × W and the number of channels is K.

For training data and actual data, images obtained by photographing coronary arteries using an X-ray imaging device (SIEMENS-CROSKTOP T • O • P PlusAngio system) were used. The information of the video is as follows.
Frame rate: 15-30 (frame / s)
Frame: 512 x 512 (8 bits)
Number of cases: 29

  From the image, 50 sets of a post-angiographic frame and a pre-angiographic frame were acquired. Note that, as shown in FIG. 5, a 64 × 64 input training image and an output training image were cut out from the 512 × 512 post-angiographic frame and the pre-angiographic frame and used. The training image for input and the training image for output are images at the same position in the frame. The alignment between the post-angiographic frame and the pre-angiographic frame was performed manually. In addition, 21025 sets of input training images and output training images were prepared.

The above neural network learning was performed on the 21025 sets, and 8032 parameters were determined. In the learning, a mini batch method was used. Various settings in the learning are as follows.
Batch size: 400
Number of epochs: 5000
Optimization: Adam method Loss function: Mean square error

The system used for learning the neural network is as follows.
CPU: Xeon E5-1650 3.5GHz
GPU: NVIDIA GeForce TITAN X × 1
OS: Linux (registered trademark) (Ubuntu 14.04)
Framework: Chainer ver. 1.8
Programming language: Python

  After learning by the learning unit 12, a mask image (output image) is generated by the image processing unit 13 using angiographic images of coronary arteries that are not used for learning. Also in this case, the angiographic image is divided into a plurality of 64 × 64 images, and each output divided image (64 × 64) is input to generate an output divided image (64 × 64). The generated output divided image The combined image was used as the final mask image (output image). The image was divided so that there was no overlap (overlap) in the divided images. FIG. 6A is a diagram illustrating an example of an angiographic image (input image) and a mask image of the present embodiment. As shown in FIG. 6A, it can be seen that a mask image from which contrast blood vessels have been removed can be acquired by using a neural network that is a learning result. FIG. 7A shows a DSA image (DSA image of the present technique) of the present example using the angiographic image and mask image shown in FIG. 6A and a DSA image using the image before angiography as a mask image (conventional). It is a figure which shows an example with DSA image). As shown in FIG. 7A, it can be seen that the DSA image generated in the present embodiment can more clearly show the contrasted blood vessels than the conventional DSA image. Thus, in the biological image processing apparatus 1 according to the present embodiment, the contrast blood vessels can be shown with high contrast in the DSA image, so that the amount of the contrast agent used once can be reduced. As a result, the burden on the body of the person who receives the contrast agent can be reduced.

  FIG. 8 is a diagram showing another example of a conventional DSA image related to a coronary artery and a DSA image generated by this embodiment. As a result of subjective evaluation by two doctors using the image of FIG. 8, both of the two doctors evaluated that the DSA image of the present method had higher image quality than the conventional DSA image. Therefore, it can be said that the biological image processing apparatus 1 according to the present embodiment is useful for supporting a better diagnosis of a doctor.

[Example 2]
In this example, learning was performed in the same manner as in Example 1 except that the number of convolution layers of the neural network was five, and mask images and DSA images were generated. Information about the five convolutional layers is as shown in FIG. 4B.

  FIG. 6B is a diagram illustrating an example of an angiographic image and a mask image according to the present embodiment. As shown in FIG. 6B, it can be seen that the contrasted blood vessels can be appropriately removed even when the learning result of the five-layer neural network is used. FIG. 7B shows a DSA image (DSA image of the present technique) of the present example using the angiographic image and mask image shown in FIG. 6B and a DSA image using the image before angiography as a mask image (conventional image). It is a figure which shows an example with DSA image). Similar to the first embodiment, it can be seen that the contrast blood vessel can be shown more clearly in the DSA image in this embodiment as well. In this embodiment as well, the neural network is characterized by having adjacent convolutional layers in which the filter size of the deeper layer is reduced.

  In Examples 1 and 2, the filter size of the convolutional layer, which is the first intermediate layer, is set to 9 × 9 so that many of the contrasted blood vessels that are the target are included in the filter size. It is to do. A neural network in which the filter size of the first convolution layer was set to other values was also used. However, when the filter size of the first layer was set to less than 7 × 7, the mask image generated in the DSA of the coronary artery had a poor quality. It was not expensive. Specifically, when the filter size of the first layer is 5 × 5, the number of contrast blood vessels remaining in the mask image is larger than when the filter size of the first layer is 7 × 7. Therefore, when the neural network is used for removing the target contrast blood vessel, it is preferable that the filter size of the first convolution layer is 7 × 7 or more. Practically, it is preferable to use a convolution layer having a filter size of 9 × 9 or more for the first layer. Here, one side (7 pixels) of the 7 × 7 filter corresponds to about half the length of the thickest blood vessel diameter in the coronary artery. It is clear that the filter size of the first convolution layer varies depending on the resolution of the biological image to be processed. For example, when processing a biological image having a double resolution, the filter size of the first convolution layer is preferably 14 × 14 or more. Therefore, the first intermediate layer may be a convolution layer using a filter corresponding to a length of 3 mm or more on one side, or a convolution layer using a filter corresponding to a length of 3.5 mm or more, It may be a convolution layer using a filter corresponding to a length of 3.75 mm or more, or may be a convolution layer using a filter corresponding to a length of 4 mm or more, and a filter corresponding to a length of 5 mm or more was used. It is good also as a convolution layer. The 3 mm or the like is the length in the living body. Therefore, it can be said that it is preferable to use a filter whose length in the living body corresponding to the number of pixels on one side of the filter is 3 mm or more. Since the diameter of the coronary artery is about 3 to 5 mm, when one side of the filter is 3 mm, the length of one side corresponds to about half of the largest diameter of the coronary artery. The first intermediate layer is a layer next to the input layer.

  As an optimization method, the Adam method and the Momentum SGD method were compared. As a result, a higher quality mask image could be generated by using the Adam method. Therefore, when performing DSA, it is considered preferable to optimize by the Adam method.

[Example 3]
In this embodiment, a DSA image is generated for the head using a neural network having five convolution layers. Information about the five convolutional layers is as shown in FIG. 4C. In this embodiment, 352 sets of frames (512 × 512) are used in learning. Each frame is a reduction of the original frame (1024 × 1024). Further, 1267200 sets of input training images and output training images were cut out from the set of frames and used. The input training image and the output training image were 32 × 32 images. Therefore, the input image (512 × 512) is also divided into 32 × 32 divided images, an output divided image (32 × 32) is generated from each input divided image (32 × 32), and the generated output divided image is The final mask image (output image) was generated by combining. The batch size was 1000 and the number of epochs was 100. In this embodiment, a Corei 5-3570 3.4 GHz CPU is used in a system for learning a neural network. Other than that, learning was performed in the same manner as in Example 1.

  FIG. 6C is a diagram illustrating an example of an angiographic image and a mask image of the present embodiment. As shown in FIG. 6C, it can be seen that contrast blood vessels can be removed even in the head DSA using the learning result of the five-layer neural network. FIG. 7C shows a head DSA image (DSA image of the present technique) of the present embodiment using the angiographic image and mask image shown in FIG. 6C and a head using an image before angiography as a mask image. It is a figure which shows an example with the DSA image (conventional DSA image) of a part. In the conventional DSA image, an artifact corresponding to the motion remains as in the range of an ellipse, but it can be seen that such an artifact disappears in the DSA image of this embodiment. Therefore, the effect of the present embodiment is great when there is a large movement of the head of the subject and movements of the mouth and chin. Note that it can be seen that the DSA image of this example can execute DSA with the same degree of accuracy as the conventional DSA image in a portion where no artifact exists. Therefore, it can be seen that the DSA image generation by this method is effective whether or not there is an artifact caused by motion.

  Further, in this embodiment, one side of the filter size 15 × 15 of the convolution layer that is the first intermediate layer is about 7 mm in the living body. Therefore, also in the present embodiment, the first intermediate layer is a convolution layer using a filter whose side corresponds to a length of 3 mm or more.

  Needless to say, the biological image processing apparatus 1 according to the present embodiment is not limited to the above-described examples, and various modifications can be made. For example, in the first to third embodiments, the intermediate layer and the output layer of the neural network are all convolutional layers. Needless to say, the neural network may include layers other than the convolutional layer such as a pooling layer. . For example, when the image size input to the input layer of the neural network is 512 × 512, it is preferable to provide a pooling layer after the convolution layer. Further, the size of each image and the number of bits of each pixel are not limited. In addition, regarding the training data of the first to third embodiments, the input frame and the output frame are manually associated with each other. However, for example, a method such as pattern matching is automatically associated with each other. It may be. Further, a bias may be used in each layer.

  As described above, according to the biological image processing apparatus 1 according to the present embodiment, the mask image obtained by removing the contrasted blood vessels from the blood vessel contrast image including the contrasted blood vessels as the target object is obtained by using the learning result of the neural network. Can be generated. Therefore, since the mask image can be obtained without performing X-ray imaging for the mask image, the exposure dose of the subject can be reduced and the examination time of the subject can be shortened. . In addition, since the mask image is generated from the angiographic image, it is possible to obtain a DSA image in which a background other than blood vessels is appropriately removed even for a moving organ or even if the imaging system is moved greatly. DSA with reduced artifacts can be realized. In addition, since it is possible to generate a mask image without using a configuration for capturing a mask image or an optical configuration for generating a mask image from an original image, the mask image can be generated while avoiding an increase in the size of the apparatus. There is also a merit that can be done.

Since removal of blood vessels can be considered to be similar to removal of noise, the inventors of the present application can use an autoencoder (Denoising Autoencoder) that can remove noise from a DSA angiographic image. Although the mask image was generated, the contrasted blood vessels could not be removed appropriately. Therefore, it can be said that the technique using the neural network is more effective for removing blood vessels and the like than such an autoencoder. For noise removal using autoencoder, refer to the following document, for example.
References: Mizuho Nishio, “Noise removal of ultra-low dose CT using deep learning and its clinical application”, Hyogo Science and Technology Association, Academic Research Grant Results Report, 2016

Hereinafter, the effects of the DSA according to the present embodiment will be described for each imaging region.
Head: A subject who cannot be stationary due to disturbance of consciousness or the like can also perform DSA to observe the form of blood vessels.
Abdomen: In the embolization treatment of hepatic artery feeding blood vessels, in order to reduce the artifacts caused by the movement of the diaphragm due to breathing, imaging and treatment were performed in the state of holding the breath. However, since DSA can be performed, it can be applied to a subject who cannot hold his / her breath.
Coronary arteries: Heart pulsation, ribs, and diaphragm movement are large, and artifacts are generated, so DSA has not been performed in the past, but this method makes it possible to perform DSA. As a result, a narrowed blood vessel can be projected. For example, since the degree of coronary artery stenosis can be evaluated precisely and in detail in occlusive heart disease of the heart, cardiac bypass surgery is required or less invasive treatment such as stent placement can be applied It will contribute to the decision of the treatment policy. It is also possible to observe the fine blood vessels opened after the stent placement. Therefore, it contributes to the patient's prognosis determination.
Lung: This technique makes it possible to perform DSA even on subjects who cannot hold their breath.
Upper extremity / lower extremity: In the upper extremity / lower extremity, since the movement of the imaging system is large, DSA is not performed conventionally, but in this method, DSA can be performed.

  In the above description, the two-dimensional DSA is mainly described. However, the three-dimensional DSA is also rotated in the same manner, and a mask image is obtained by the imaging system that rotates in the same manner. A mask image can be generated even in a three-dimensional DSA that obtains an image by the above method. In addition, as described above, processing other than DSA may be performed using the method according to the present embodiment. The biological images handled by the biological image processing apparatus 1 according to the present embodiment are, for example, simple X-ray photographs, CT images, MRI images, nuclear medicine images (for example, PET images, SPECT images, etc.), ultrasonic images, thermographic images, etc. It may be a living body image. The biological image is normally a human image, but an image of a living organism other than a human may be used as the biological image. For example, the biological image may be an animal image. Examples of the image processing by the image processing unit 13 and the subtraction processing by the subtraction processing unit 15 include the following.

1. Object extraction processing The object is to extract the object from the subject's image. The DSA is also included in this process. The target object may be, for example, bone, blood vessel, ureter, pancreatic duct, bile duct, lung, liver, kidney, pancreas and the like. Further, the target object may be a lesion or an abnormal structure. The lesion or abnormal structure may be, for example, inflammation, tumor, calculus, stenosis, aneurysm or the like.

  When the target is a blood vessel, the DSA is used. In addition, even when the target object is a ureter, pancreatic duct, or bile duct, the X-ray image after the contrast is made into a biological image including the target object, and the X-ray image before the contrast, that is, the living body not administered with the contrast agent By learning an X-ray image as a living body image that does not include a target object, a mask image obtained by removing a contrast portion from an X-ray image after contrast using a neural network after learning, as in the case of the DSA. An output image can be generated. Then, by subtracting the mask image from the contrast-enhanced X-ray image, a subtracted image in which contrast images of the ureter, pancreatic duct, and bile duct are extracted can be generated in the same manner as the DSA.

  When the target object is an organ or bone other than a blood vessel, ureter, pancreatic duct, or bile duct, the biological image may be a CT or MRI tomographic image. In that case, the tomographic image itself becomes a biological image including the target object. Further, an image obtained by deleting the target organ or bone from the tomographic image by manual image processing may be used as a biological image that does not include the target object. Then, an output image, which is a mask image obtained by removing the target organs and bones, can be generated from the tomographic image by a neural network of learning results using them. Thereafter, by subtracting the mask image generated from the tomographic image from the input image that is a tomographic image, a subtracted image in which the target organ or bone is extracted can be generated.

  Even when the target object is a lesion or an abnormal structure, the target object can be extracted in the same manner as when the target object is a bone or the like. In that case, X-ray images, CT images, MRI images, PET images, SPECT images, ultrasound images, thermographic images, and the like including lesions and abnormal structures may be biological images including the target object. Further, an image obtained by deleting the lesion or abnormal structure by manual image processing from the biological image including the lesion or abnormal structure may be a biological image not including the target object. If possible, a biometric image obtained for the same subject before the occurrence of a lesion or an abnormal structure, that is, a biometric image that does not include a lesion or an abnormal structure is used as a biological image that does not include a target object. Also good. Further, the biological image of the first subject having a lesion or abnormal structure is used as a biological image including the target object, and the biological image of the second subject having no lesion or abnormal structure is used as the target object. It may be a biological image not included. The first subject and the second subject are different subjects. Even in such a case, it is preferable that the biological image including the target object and the biological image not including the target object are biological images that approximate the same part. Specifically, when both biological images are biological images such as a liver and a kidney, it is preferable that they are approximated biological images including the liver, the kidney, and the like in the same range from the same direction. In addition, a biological image of a subject's lesion or abnormal structure is taken as a biological image including the target object, and a biological image of a part where the same subject's lesion or abnormal structure does not exist is a biological image not including the target object It is good. Even in that case, it is preferable that the biological image including the target object and the biological image not including the target object are approximated biological images. Then, an output image, which is a mask image obtained by removing lesions and abnormal structures, from a biological image including lesions and abnormal structures, which are target objects, can be generated by a neural network of learning results using them. After that, by subtracting the mask image generated from the biological image from the input image, which is a biological image including the target lesion or abnormal structure, the target lesion or abnormal structure (for example, calculus or tumor) Etc.) can be generated.

2. Removal and addition of target object The purpose is to remove or add the target object from the subject's image. The target object may be, for example, a medical device in the body. Examples of the medical device in the body include a stent, a coil, a clip, a catheter, an artificial bone head, and an implant.

(1) Removal of target object An X-ray image, a CT image, an MRI image, etc. taken after a target stent or the like is implanted in the body is a biological image including the target object. Further, an X-ray image or the like taken before the target stent or the like is implanted in the body may be a biological image that does not include the target object. Further, an image obtained by deleting a target object by manual image processing from a biological image including the target object may be used as a biological image not including the target object. Then, an output image obtained by removing the target stent or the like from the biological image including the target stent or the like can be generated by a learning result neural network. In this case, the biological image processing apparatus 1 may not include the subtraction processing unit 15 and the subtraction image output unit 16. The biological image processing apparatus 1 may include an output unit that outputs an output image. The output unit may be the same as the subtraction image output unit 16 except that the output target is different.

(2) Addition of object Learning is performed in the same manner as in (1) above, and a mask image is generated by generating an output image using the learning result. Then, the mask image is subtracted from the biological image including the stent or the like using the mask image and the biological image including the stent or the like that is the target used in generating the output image. An extracted subtracted image of a stent or the like can be generated.

  Here, the above 1. In this case, when extracting bones and blood vessels, the stent image is included in the mask image, so that a subtracted image from which the stent is removed is generated. Therefore, in that case, the above 1. By synthesizing the extracted subtracted image of the target stent, etc., generated as described above, with the subtracted image, the image of blood vessels, bones, etc. including the target stent, etc. is acquired. Can do. In this case, 1. 1 and 2. In the process (2), the input image is the same image. In addition, 1. 1. Subtracted image of 2. To combine the subtracted image of (2): 1. an object (for example, a blood vessel) included in the subtracted image of 2. This means that both are combined so that the positional relationship with the target object (for example, stent) included in the subtracted image in (2) can be understood. It doesn't matter how. The above 1. In the DSA subtraction image, a stent or the like may remain. For example, when the stent image resembles a blood vessel in an angiographic image, the above 1. When the mask image is generated in step 1, a mask image from which the stent and the like are removed is generated. In this subtracted image, a stent or the like remains. As described above, when a stent or the like remains in the DSA subtraction image, it is not necessary to perform the object addition processing in (2). When the target is a catheter, the catheter image is usually similar to a blood vessel in angiographic images of coronary arteries. When generating a mask image in step 1, a mask image from which the catheter has been removed is generated. A catheter image remains in the DSA subtraction image. Therefore, in the case of a coronary artery DSA in which the object is a catheter, the object addition process of (2) may not be performed.

  As described above, there may be a case where the subtraction process is unnecessary. In such a case, the biological image processing apparatus 1 may not include the subtraction processing unit 15 and the subtraction image output unit 16. In this case, the biological image processing apparatus 1 may include an output unit that outputs the output image generated by the image processing unit 13 as described above.

  In the present embodiment, the case where learning and processing using the learning result are performed as a series of processing has been described, but this need not be the case. The learning and the process using the learning result may be performed separately. The biological image processing apparatus that performs learning generates, for example, a reception unit 11 that receives an input training image and an output training image, and generates a biological image that does not include a target object from a biological image that includes the target object using both training images. And a learning unit 12 that learns a neural network that performs image processing. In addition, the apparatus may include an output unit that outputs a plurality of parameters that are learning results. The output may be, for example, transmission of information, accumulation of information on a recording medium, delivery of information to other components, and the like. The biological image processing apparatus that performs processing using the learning result includes, for example, a reception unit 11 that receives an input image and a learning result of a neural network that generates a biological image that does not include a target object from a biological image that includes the target object. An image processing unit 13 that generates an output image obtained by removing an object from the input image received by the receiving unit 11 using image processing may be provided. The image processing unit 13 performs image processing using a plurality of parameters as learning results from another device. The biological image processing apparatus may further include an output unit that outputs an output image, or may further include a subtraction processing unit 15 and a subtraction image output unit 16.

  Note that, as described above, the biological image that does not include the target object may be, for example, a biological image acquired when there is no target object, or a biological image in which the target object is deleted from the biological image that includes the target object. It may be.

  In the present embodiment, the accepting unit 11 may accept an input image or the like from an angiography apparatus, an apparatus such as PET / SPECT / MRI, or may accept an input image or the like from a recording medium. Good. In the former case, the biological image processing apparatus 1 may generate an output image from the input image by real-time processing. The biological image processing apparatus 1 may generate an output image from an input image by batch processing instead of real-time processing.

  In the above embodiment, the case where the biological image processing apparatus 1 is a stand-alone has been described. However, the biological image processing apparatus 1 may be a stand-alone apparatus or a server apparatus in a server / client system. Good. In the latter case, the receiving unit and the output unit may receive information or output information via a communication line.

  In the above embodiment, each process or each function may be realized by centralized processing by a single device or a single system, or may be distributedly processed by a plurality of devices or a plurality of systems. It may be realized by doing.

  In the above embodiment, the information exchange between the components is performed by one component when, for example, the two components that exchange the information are physically different from each other. It may be performed by outputting information and receiving information by the other component, or when two components that exchange information are physically the same, one component May be performed by moving from the phase of the process corresponding to to the phase of the process corresponding to the other component.

  In the above embodiment, information related to processing executed by each component, for example, information received, acquired, selected, generated, transmitted, or received by each component In addition, information such as threshold values, mathematical formulas, addresses, and the like used by each constituent element in processing may be temporarily or for a long time held in a recording medium (not shown), even if not specified in the above description. Further, the storage of information on the recording medium (not shown) may be performed by each component or a storage unit (not shown). Further, reading of information from the recording medium (not shown) may be performed by each component or a reading unit (not shown).

  In the above embodiment, when information used by each component, for example, information such as a threshold value, an address, and various setting values used by each component may be changed by the user, Even if it is not specified in the description, the user may be able to change the information as appropriate, or may not be so. If the information can be changed by the user, the change is realized by, for example, a not-shown receiving unit that receives a change instruction from the user and a changing unit (not shown) that changes the information in accordance with the change instruction. May be. The change instruction received by the receiving unit (not shown) may be received from an input device, information received via a communication line, or information read from a predetermined recording medium, for example. .

  In the above embodiment, when two or more constituent elements included in the biological image processing apparatus 1 have a communication device, an input device, etc., the two or more constituent elements have a physically single device. Or may have separate devices.

  In the above-described embodiment, each component may be configured by dedicated hardware, or a component that can be realized by software may be realized by executing a program. For example, each component can be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. At the time of execution, the program execution unit may execute the program while accessing the storage unit or the recording medium. The software that realizes the biological image processing apparatus in the above embodiment is the following program. That is, this program accepts a computer to accept an input training image that is a biological image including a target object, an output training image that is a biological image not including the target object, and an input image that is a biological image including the target object. A learning unit that learns a neural network that performs image processing to generate a biological image that does not include a target object from a biological image that includes the target object, using the training image for input and the training image for output received by the reception unit, Using the neural network image processing that is the learning result of the learning unit, the neural network functions as an image processing unit that generates an output image obtained by removing the target object from the input image received by the receiving unit. It is a program that has

  In the program, the functions realized by the program do not include functions that can be realized only by hardware. For example, functions that can be realized only by hardware such as a modem and an interface card in a reception unit that receives information and an output unit that outputs information are not included in at least the functions realized by the program.

  Further, this program may be executed by being downloaded from a server or the like, and a program recorded on a predetermined recording medium (for example, an optical disk such as a CD-ROM, a magnetic disk, a semiconductor memory, or the like) is read out. May be executed by Further, this program may be used as a program constituting a program product.

  Further, the computer that executes this program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

FIG. 9 is a schematic diagram showing an example of the external appearance of a computer that executes the program and realizes the biological image processing apparatus 1 according to the embodiment. The above-described embodiment can be realized by computer hardware and a computer program executed on the computer hardware.
In FIG. 9, the computer system 900 includes a computer 901 including a CD-ROM drive 905, a keyboard 902, a mouse 903, and a monitor 904.

  FIG. 10 is a diagram showing an internal configuration of the computer system 900. In FIG. 10, in addition to the CD-ROM drive 905, a computer 901 is connected to an MPU (Micro Processing Unit) 911, a ROM 912 for storing a program such as a boot-up program, and the MPU 911. A RAM 913 that temporarily stores and provides a temporary storage space, a hard disk 914 that stores application programs, system programs, and data, and a bus 915 that interconnects the MPU 911, the ROM 912, and the like are provided. The computer 901 may include a network card (not shown) that provides connection to a LAN, WAN, or the like.

  A program that causes the computer system 900 to execute the functions of the biological image processing apparatus 1 according to the above-described embodiment may be stored in the CD-ROM 921, inserted into the CD-ROM drive 905, and transferred to the hard disk 914. Instead, the program may be transmitted to the computer 901 via a network (not shown) and stored in the hard disk 914. The program is loaded into the RAM 913 when executed. The program may be loaded directly from the CD-ROM 921 or the network. Further, the program may be read into the computer system 900 via another recording medium (for example, a DVD) instead of the CD-ROM 921.

  The program does not necessarily include an operating system (OS) or a third-party program that causes the computer 901 to execute the functions of the biological image processing apparatus 1 according to the above-described embodiment. The program may include only a part of an instruction that calls an appropriate function or module in a controlled manner and obtains a desired result. How the computer system 900 operates is well known and will not be described in detail.

  Further, the present invention is not limited to the above-described embodiment, and various modifications are possible, and it goes without saying that these are also included in the scope of the present invention.

  As described above, according to the biological image processing apparatus and the like according to the present invention, it is possible to generate a biological image from which a target object is removed from a biological image including the target object, which is useful for generating a DSA mask image, for example. .

DESCRIPTION OF SYMBOLS 1 Living body image processing apparatus 11 Reception part 12 Learning part 13 Image processing part 14 Memory | storage part 15 Subtraction process part 16 Subtraction image output part

Claims (9)

A receiving unit that receives an input training image that is a biological image including a target object, an output training image that is a biological image not including the target object, and an input image that is a biological image including the target object;
A learning unit that learns a neural network that performs image processing for generating a biological image that does not include a target object from a biological image that includes a target object, using the training image for input and the training image for output received by the reception unit;
Using an image processing of a neural network that is a learning result by the learning unit, and an image processing unit that generates an output image by removing a target object from the input image received by the receiving unit,
The neural network has a plurality of convolution layers, and is a biological image processing apparatus. A reception unit that receives an input image that is a biological image including a target;
An image for generating an output image obtained by removing the target object from the input image received by the receiving unit by using image processing of a learning result of the neural network that generates a biological image not including the target object from the biological image including the target object A processing unit,
The neural network has a plurality of convolution layers, and is a biological image processing apparatus. The biological image processing apparatus according to claim 1, wherein the intermediate layer and the output layer in the neural network are all convolutional layers. The living body image including the target object is an angiographic image that is an X-ray image of a living body to which a contrast medium is administered,
A biological image that does not include a target object is an X-ray image of a biological body that is not administered with a contrast agent
The biological image processing apparatus according to claim 1, wherein the output image is a mask image used for digital differential angiography. The biological image processing apparatus according to claim 4, wherein the first intermediate layer in the neural network is a convolution layer using a filter whose side corresponds to a length of 3 mm or more. A subtraction processing unit that subtracts a mask image that is an output image generated from the angiographic image by the image processing unit from an angiographic image that is an input image received by the receiving unit;
The biological image processing apparatus according to claim 4, further comprising: a subtracted image output unit that outputs a subtracted image subtracted by the subtracting unit. A first receiving step for receiving a training image for input that is a biological image including a target object and a training image for output that is a biological image not including the target object;
Learning to learn a neural network that performs image processing for generating a biological image that does not include a target object from a biological image that includes the target object, using the input training image and the output training image received in the first receiving step. Steps,
A second accepting step for accepting an input image that is a biological image including the object;
An image processing step of generating an output image obtained by removing an object from the input image received in the second receiving step using image processing of a neural network that is a learning result in the learning step;
The method for producing an output image, wherein the neural network has a plurality of convolutional layers. An accepting step for receiving an input training image that is a biological image including a target object and an output training image that is a biological image not including the target object;
A learning result by learning a neural network that performs image processing for generating a biological image not including a target object from a biological image including the target object using the input training image and the output training image received in the reception step. And a learning step,
The learning result manufacturing method, wherein the neural network has a plurality of convolutional layers. Computer
A receiving unit that receives a training image for input that is a biological image including the target object, a training image for output that is a biological image that does not include the target object, and an input image that is a biological image including the target object;
A learning unit that learns a neural network that performs image processing to generate a biological image that does not include a target object from a biological image that includes a target object, using the input training image and the output training image received by the reception unit;
Using image processing of a neural network that is a learning result by the learning unit, function as an image processing unit that generates an output image obtained by removing an object from an input image received by the receiving unit,
The neural network has a plurality of convolution layers.