JP2021041089A - Medical image processing device, x-ray image processing system and generation method of learning model - Google Patents

Abstract

To provide a medical image processing device which can suppress the reduction of the accuracy of extraction or removal of a specific portion by using a learning model generated with machine learning even when components included in an X-ray image and a two-dimensional projection image are different from each other, and provide an X-ray image processing system and a generation method of a learning model.SOLUTION: An X-ray image processing system 100 extracts a specific portion Q or removes the specific portion Q on the basis of a generated first learning model M1 with machine learning by using an input teacher image T1 including at least one of an image obtained by adding an object shape image B to a DRR image D and an image obtained by subtracting the object shape image from the DRR image and an output teacher image T2 being an image indicating the specific portion Q in the same region as the input teacher image T1 or an image excluding the specific portion Q.SELECTED DRAWING: Figure 1

Description

The present invention relates to a medical image processing apparatus, an X-ray image processing system, and a method of generating a learning model.

Conventionally, a radiotherapy tracking device that tracks the movement of a specific site in order to detect the position of the specific site from an X-ray fluoroscopic image including the specific site of the subject and irradiate the specific site with a treatment beam has been known. (See, for example, Patent Document 1).

The above-mentioned Patent Document 1 discloses a radiotherapy tracking device including a DRR (Digitally Reconstructed Radiography) image creating unit, a discriminator learning unit, and a specific site region detecting unit. The DRR image creation unit creates a DRR image including a specific site by performing a virtual perspective projection on the CT image data of the region including the specific site created at the time of treatment planning. The discriminator learning unit learns a discriminator for recognizing a region of a specific region by executing machine learning by inputting a DRR image and outputting a teacher label image indicating a region of a specific region as an output. The specific site region detection unit detects the region of the specific region by discriminating the X-ray fluoroscopic image including the specific region by using the learned classifier. The radiotherapy tracking device described in Patent Document 1 learns a classifier by machine learning a DRR image and a teacher label image in advance according to the above configuration, and uses the classifier and an X-ray fluoroscopic image. By executing the identification, the position of a specific part is detected. Then, the radiotherapy tracking device described in Patent Document 1 tracks the movement of a specific site in order to irradiate the treatment beam to the specific site.

International Publication No. 2019/003434

However, in the radiotherapy tracking device described in Patent Document 1, the structure or member (structure included in the X-ray fluoroscopic image) included in the X-ray fluoroscopic image is based on CT image data (three-dimensional data). It may not be included in the DRR image generated in the above. The structure includes, for example, a part of the top plate on which the subject is placed and a cover of the X-ray detector. The member also includes, for example, a guide wire and a catheter that are placed in the body of the subject. In addition, the position or number of markers placed in the body of the subject may change between the time when the DRR image is generated and the time when the fluoroscopic image is taken. In these cases, the X-ray fluoroscopic image is identified using a classifier generated by machine learning based on the DRR image due to the difference in the components contained in the image between the X-ray fluoroscopic image and the DRR image. In some cases, it is considered that the accuracy of identification may decrease. Therefore, even if the components included in the X-ray fluoroscopic image (X-ray image) and the DRR image (two-dimensional projection image) are different, the classifier (learning model) generated by machine learning is used. It is desired to develop a medical image processing apparatus, an X-ray image processing system, and a learning model generation method capable of suppressing a decrease in the accuracy of extraction or removal of a specific part (specific part).

The present invention has been made to solve the above-mentioned problems, and one object of the present invention is to use an X-ray image and a two-dimensional projected image even when the components contained in the images are different. , A medical image processing device capable of suppressing a decrease in the accuracy of extraction or removal of a specific part using a learning model generated by machine learning, an X-ray image processing system, and a method of generating a learning model. Is to provide.

In order to achieve the above object, the medical image processing apparatus according to the first aspect of the present invention relates to a region including a specific part of a subject acquired based on three-dimensional data having pixel value data in three dimensions. An image in which an object shape image including an image relating to a predetermined object separate from the subject is added to the dimensional projection image, and an image in which the object shape image is subtracted from the two-dimensional projection image. A learning image generator that generates an input teacher image including at least one, an image showing a specific part in the same area as the input teacher image, or an output teacher image that is an image excluding the specific part, and an input teacher image and an output teacher image. Based on a learning model generation unit that generates a first learning model for extracting a specific part or removing a specific part by performing machine learning using, and a trained first learning model. , A specific part processing unit that performs either a process of extracting a specific part or a process of removing a specific part of an X-ray image including a specific part of a subject acquired by X-ray imaging. Be prepared. The "X-ray image" includes a fluoroscopic image and an X-ray photographed image.

The X-ray image processing system in the second aspect of the present invention relates to a two-dimensional projected image of a region including a specific part of a subject acquired based on the three-dimensional data having pixel value data in three dimensions. , An input teacher that includes at least one of an image to which an object shape image including an image relating to a predetermined object separate from the subject is added and an image in which the object shape image is subtracted from the two-dimensional projected image. Machine learning using a learning image generator that generates an image and an image showing a specific part in the same area as the input teacher image or an image excluding the specific part, and an input teacher image and an output teacher image. By performing the above, a learning model generation unit that extracts a specific part or generates a first learning model for removing the specific part, and an X-ray image including a specific part of the subject are acquired by X-ray photography. Based on the X-ray imaging unit and the trained first learning model, either the process of extracting the specific part or the process of removing the specific part is performed on the X-ray image including the specific part. It is provided with a specific partial processing unit.

The method of generating a learning model in the third aspect of the present invention is for a two-dimensional projected image of a region including a specific part of a subject acquired based on three-dimensional data having pixel value data in three dimensions. , An input teacher that includes at least one of an image to which an object shape image including an image relating to a predetermined object separate from the subject is added and an image in which the object shape image is subtracted from the two-dimensional projected image. A step to generate an image, a step to generate an output teacher image which is an image showing a specific part in the same area as the input teacher image or an image excluding the specific part based on 3D data, and an input teacher image and an output teacher image. It is provided with a step of extracting a specific part or generating a first learning model for removing the specific part by performing machine learning using the above.

In the medical image processing apparatus in the first aspect and the X-ray image processing system in the second aspect, the learning image generation unit is acquired based on the three-dimensional data having the pixel value data in the three dimensions. An image in which an object shape image including an image of a predetermined object separate from the subject is added to a two-dimensional projected image of a region including a specific part of the sample, and an image of the above object with respect to the two-dimensional projected image. To generate an image with the shape image reduced, an input teacher image containing at least one of them, and an output teacher image that is an image showing a specific part in the same area as the input teacher image or an image excluding the specific part. Configure. Then, the learning model generation unit performs machine learning using the input teacher image and the output teacher image to extract a specific part or generate a first learning model for removing the specific part. It is configured as follows. As a result, at least one of adding and reducing the object shape image to the two-dimensional projected image is performed, so that the components included in the two-dimensional projected image correspond to the components included in the X-ray image. Can be changed. That is, it is possible to generate a learning model that can handle even when there is a difference between the constituents included in the two-dimensional projected image and the constituents included in the X-ray image. For example, a learning model is created by performing machine learning using an image (an image including an object shape image) in which a component contained only in an X-ray image is pseudo-included in a two-dimensional projected image as an input teacher image. Can be generated. Therefore, even when the components contained in the image are different between the X-ray image and the two-dimensional projected image, the learning model generated by machine learning is used to extract a specific part or remove the specific part. It is possible to suppress a decrease in accuracy.

Further, in the method of generating a learning model in the third aspect of the present invention, a two-dimensional projected image of a region including a specific part of a subject acquired based on three-dimensional data having pixel value data in three dimensions is obtained. On the other hand, it includes at least one of an image to which an object shape image including an image relating to a predetermined object separate from the subject is added and an image in which the object shape image is subtracted from the two-dimensional projected image. It includes a step of generating an input teacher image and a step of generating an output teacher image which is an image showing a specific portion in the same region as the input teacher image or an image excluding the specific portion based on three-dimensional data. As a result, at least one of adding and reducing the object shape image to the two-dimensional projected image is performed, so that the components included in the two-dimensional projected image correspond to the components included in the X-ray image. Can be changed. That is, it is possible to generate a learning model that can handle even when there is a difference between the constituents included in the two-dimensional projected image and the constituents included in the X-ray image. For example, machine learning can be performed using a two-dimensional projected image in a state in which a component included in an X-ray image is pseudo-included (a state in which an object shape image is included) as an input teacher image. As a result, a learning model capable of suppressing a decrease in the accuracy of extraction or removal of a specific portion even when the components contained in the image are different between the X-ray image and the two-dimensional projection image is provided. be able to.

It is a block diagram for demonstrating the whole structure of the X-ray image processing system by 1st Embodiment. It is a figure for demonstrating the structure of the X-ray image processing system at the time of radiotherapy by 1st Embodiment. It is a figure for demonstrating the CT image data by 1st Embodiment. It is a block diagram which shows the functional structure of the 1st control part and the 2nd control part by 1st Embodiment. It is a figure for demonstrating the extraction of the specific part by the 1st learning model. It is a figure for demonstrating the generation of a DRR image. It is a figure for demonstrating the difference in composition between a DRR image and an X-ray image. It is a figure for demonstrating the object shape image at the time of imitating an FPD cover. It is a figure for demonstrating the horizontal line noise which occurs in an X-ray fluoroscopic image. It is a figure for demonstrating the object shape image at the time of imitating a marker. It is a figure for demonstrating the extraction of the specific part by the 2nd learning model. It is a flowchart for demonstrating the generation method of the 1st learning model. It is a flowchart for demonstrating the control of the X-ray image processing system by 1st Embodiment. It is a flowchart for demonstrating the use example of the X-ray image processing system by 1st Embodiment. It is a block diagram for demonstrating the whole structure of the X-ray image processing system by 2nd Embodiment. It is a figure for demonstrating the structure of the X-ray image processing system in angiography by 2nd Embodiment. It is a figure for demonstrating the DRR image by 2nd Embodiment. It is a block diagram for showing the functional structure of the 1st control part and the 2nd control part by 2nd Embodiment. It is a figure for demonstrating the input teacher image and the output teacher image by 2nd Embodiment. It is a figure for demonstrating the removal of the bone part by the 3rd learning model. It is a flowchart for demonstrating the control of the X-ray image processing system by 2nd Embodiment.

Hereinafter, embodiments embodying the present invention will be described with reference to the drawings.

[First Embodiment]
The configuration of the X-ray image processing system 100 according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 11.

(Configuration of X-ray image processing system)
As shown in FIG. 1, the X-ray image processing system 100 includes a radiotherapy device 1, a treatment planning device 2, a learning device 3, an X-ray imaging unit 4, and an X-ray image processing device 5. The learning device 3 and the X-ray image processing device 5 are examples of "medical image processing devices" within the scope of the claims.

The X-ray image processing system 100 is a tracking device that tracks a specific part Q whose position changes due to the respiration of the subject P or the like when treating a specific part Q including a tumor such as cancer by the radiotherapy device 1. It is configured.

As shown in FIG. 2, the radiotherapy apparatus 1 includes a top plate 11, a gantry 12, a base 13, and a head 14. A subject P to be irradiated with radiation is placed on the top plate 11. The gantry 12 is configured to be movable with respect to the base 13 arranged on the floor surface. The head 14 is arranged in the gantry 12 and irradiates the subject P with a therapeutic beam. The radiation therapy device 1 is configured so that the irradiation direction of the treatment beam irradiated from the head 14 with respect to the subject P can be changed by rotating the gantry 12 with respect to the base 13.

Further, the radiotherapy device 1 is configured to control the position and timing of irradiating the treatment beam based on a signal from the X-ray image processing device 5 described later. Specifically, when the treatment beam is irradiated to the specific portion Q of the subject P, the position of the specific portion Q changes due to the respiration of the subject P or the like. The radiotherapy apparatus 1 is configured so that the specific portion Q can be irradiated with radiation (treatment beam) by a signal based on the position of the specific portion Q acquired by the X-ray image processing apparatus 5.

The treatment planning device 2 is a device for a doctor to plan radiotherapy for a subject P. That is, a CT (Computed Tomography) device (not shown) generates three-dimensional CT image data C for a region including a specific portion Q. Then, in the treatment planning device 2, the treatment is planned based on the three-dimensional CT image data C and the medical examination information regarding the subject P. That is, the treatment planning device 2 generates treatment planning data based on the three-dimensional CT image data C and the examination information regarding the subject P. The CT image data C is three-dimensional data having pixel value data (CT value) in three dimensions. The CT value is a numerical value indicating the ease with which X-rays can be transmitted. As shown in FIG. 3, the CT image is an image showing the magnitude of the CT value of the subject P in the body in shades of light. Here, the CT image data C is three-dimensional voxel data generated based on a plurality of two-dimensional CT images. Further, in the treatment planning device 2, the pixel portion (voxel) corresponding to the specific portion Q on the CT image data C is designated as the specific portion Q by the doctor. Then, the treatment planning device 2 stores the CT image data C in a state including the information about the designated specific portion Q. The CT image data C is an example of "three-dimensional data" in the claims.

As shown in FIG. 1, the learning device 3 includes a first control unit 30. The first control unit 30 has, for example, a CPU (Central Processing Unit). The learning device 3 has, for example, a CPU, a GPU (Graphics Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like as a hardware configuration. Further, the learning device 3 includes an HDD (Hard Disk Drive), a non-volatile memory, or the like as a storage unit.

As shown in FIG. 4, the first control unit 30 includes a DRR image generation unit 31, a teacher image generation unit 32, and a learning model generation unit 33 as functional configurations. The DRR image generation unit 31, the teacher image generation unit 32, and the learning model generation unit 33 are configured as programs (software). That is, the first control unit 30 is configured to function as a DRR image generation unit 31, a teacher image generation unit 32, and a learning model generation unit 33 by executing a program (software). The DRR image generation unit 31 and the teacher image generation unit 32 are examples of the “learning image generation unit” in the claims.

Further, as shown in FIG. 5, the first control unit 30 performs a process of generating an input teacher image T1 and an output teacher image T2 based on the CT image data C acquired by the treatment planning device 2, and also performs a machine learning process. By performing the learning, a process of generating a first learning model M1 for extracting the specific portion Q and a second learning model M2 (see FIG. 11) for extracting the marker E is performed. The details of the first learning model M1 and the second learning model M2 will be described later.

As shown in FIG. 2, the X-ray imaging unit 4 acquires an X-ray fluoroscopic image A including a specific portion Q of the subject P by X-ray imaging. The X-ray imaging unit 4 includes an X-ray irradiation unit 41 and an X-ray detection unit 42. The X-ray irradiation unit 41 includes a first X-ray tube 41a, a second X-ray tube 41b, a third X-ray tube 41c, and a fourth X-ray tube 41d. Further, the X-ray detector 42 includes a first X-ray detector 42a, a second X-ray detector 42b, a third X-ray detector 42c, and a fourth X-ray detector 42d. The first to fourth X-ray detectors 42a to 42d each have, for example, an FPD (Flat Panel Detector). The first X-ray detector 42a detects the X-rays emitted from the first X-ray tube 41a. The first X-ray tube 41a and the first X-ray detector 42a constitute a first X-ray imaging system. Similarly, the second X-ray detector 42b detects the X-rays emitted from the second X-ray tube 41b. The second X-ray tube 41b and the second X-ray detector 42b form a second X-ray imaging system. Similarly, the third X-ray detector 42c detects the X-rays emitted from the third X-ray tube 41c. The third X-ray tube 41c and the third X-ray detector 42c form a third X-ray imaging system. Similarly, the 4th X-ray detector 42d detects the X-rays emitted from the 4th X-ray tube 41d. The 4th X-ray tube 41d and the 4th X-ray detector 42d constitute a 4th X-ray imaging system. The X-ray fluoroscopic image A is an example of the "X-ray image" in the claims.

When X-ray imaging a region including a specific portion Q of the subject P for radiotherapy, two imaging systems are selected from the four imaging systems of the first to fourth X-ray imaging systems. It is used as an X-ray imaging unit 4.

As shown in FIG. 1, the X-ray image processing device 5 includes a second control unit 50 and an operation unit 53. The second control unit 50 has, for example, a CPU. The X-ray image processing device 5 has, for example, a CPU, a GPU, a ROM, a RAM, and the like as a hardware configuration. Further, the X-ray image processing device 5 includes an HDD, a non-volatile memory, or the like as a storage unit.

As shown in FIG. 4, the second control unit 50 includes a specific partial processing unit 51 and a marker processing unit 52 as functional configurations. The specific partial processing unit 51 and the marker processing unit 52 are configured by a program (software). That is, the second control unit 50 is configured to function as the specific partial processing unit 51 and the marker processing unit 52 by executing the program (software).

As shown in FIG. 5, the X-ray image processing device 5 is photographed by the X-ray photographing unit 4 based on the first learning model M1 and the second learning model M2 (see FIG. 11) generated by the learning device 3. A process for extracting a specific portion Q is performed on the X-ray fluoroscopic image A which is an X-ray image. X-ray fluoroscopy is a method of continuously irradiating a smaller amount of X-rays than general X-ray photography to acquire a moving image of the inside of the subject P while suppressing an increase in the amount of exposure. is there. The X-ray image processing device 5 uses the first learning model M1 and the second learning model M2 generated by machine learning in the learning device 3 as input data of the X-ray fluoroscopic image A captured by the X-ray photographing unit 4. By doing so, the output image Aα for tracking the position of the specific portion Q is acquired. Then, the X-ray image processing device 5 outputs a signal for controlling the position at which the radiotherapy device 1 irradiates the treatment beam and the timing at which the treatment beam is radiated, based on the acquired information regarding the position of the specific portion Q. , Transmit to the radiotherapy device 1.

The specific part processing unit 51 extracts the specific part Q from the fluoroscopic image A including the specific part Q of the subject P acquired by X-ray imaging based on the trained first learning model M1. Perform processing.

The marker processing unit 52 extracts the marker E from the X-ray fluoroscopic image A including the specific portion Q of the subject P acquired by X-ray imaging based on the trained second learning model M2. A process for extracting a specific part Q is performed.

The operation unit 53 accepts an input operation for selecting whether to perform processing by the specific partial processing unit 51 or the marker processing unit 52. Specifically, when at least one of the position and the number of markers E placed in the body of the subject P changes between the CT image data C and the fluoroscopic image A, the position of the marker E And the process by the specific partial processing unit 51 is selected to extract the specific partial Q regardless of the number. Further, when both the position and the number of markers E placed in the body of the subject P are not changed between the CT image data C and the fluoroscopic image A, the markers are used to extract the markers E. Processing by the processing unit 52 is selected. The operation unit 53 includes, for example, a mouse and a keyboard.

(Regarding the generation of the first learning model in the first embodiment)
Here, in the present embodiment, the DRR image generation unit 31 generates and acquires a DRR image D for a region including a specific portion Q of the subject P based on the CT image data C. Then, the teacher image generation unit 32 adds an object shape image B including an image relating to a predetermined object separate from the subject P to the acquired DRR image D, and the DRR image D. The input teacher image T1 including at least one of the image in which the object shape image B is reduced is generated. Further, the teacher image generation unit 32 generates an output teacher image T2 which is an image showing a specific portion Q in the same region as the input teacher image T1. Then, the learning model generation unit 33 generates a first learning model M1 for extracting a specific portion Q by performing machine learning using the input teacher image T1 and the output teacher image T2.

<Regarding DRR image generation>
As shown in FIG. 6, the DRR image generation unit 31 generates a DRR image D including a specific portion Q by a digital reconstruction simulation based on CT image data C which is three-dimensional data obtained by computed tomography. The DRR image D is an example of a "two-dimensional projected image" in the claims.

In generating the DRR image D, the DRR image generation unit 31 uses a virtual X-ray tube 41α and a virtual X-ray detector 42α in a three-dimensional virtual space to X-ray the CT image data C. By arranging so as to perform imaging, a three-dimensional spatial arrangement (imaging geometry) of a virtual X-ray imaging system is generated. The arrangement of these CT image data C and the virtual X-ray imaging system with respect to the CT image data C is the same as the arrangement of the actual subject P, the X-ray irradiation unit 41, and the X-ray detection unit 42 shown in FIG. It is the shooting geometry of. Here, the imaging geometry means the geometrical arrangement relationship between the subject P and the X-ray irradiation unit 41 and the X-ray detection unit 42 in a three-dimensional space.

Then, the DRR image generation unit 31 virtually takes an X-ray image of the CT image data C by irradiating the virtual X-ray detector 42α with X-rays from the virtual X-ray tube 41α. At this time, the DRR image generation unit 31 adds the total CT values in the pixel portions (voxels) that the X-rays emitted from the X-ray tube 41α have passed by the time they reach the virtual X-ray detector 42α. Each pixel value in the DRR image D is calculated by.

As described above, the DRR image generation unit 31 has the same region as the X-ray fluoroscopic image A of the subject P acquired by the X-ray imaging unit 4 with respect to the CT image data C which is the pixel value data in three dimensions. The DRR image D is generated by projecting the images in two dimensions as if they were virtually X-rayed so that the images were taken from the same angle.

<Regarding the generation of input teacher image and output teacher image>
The DRR image D generated based on the CT image data C and the X-ray fluoroscopic image A captured by the X-ray imaging unit 4 may have different components included in the image. For example, as shown in FIG. 7, the cover F of the FPD included in the X-ray detection unit 42 is not included in the DRR image D generated based on the CT image data C, but is included in the X-ray fluoroscopic image A. May be included. As described above, when there is a difference between the component included in the DRR image D and the component included in the fluoroscopic image A, the accuracy of extracting the specific portion Q using the learning model may decrease. .. Therefore, the teacher image generation unit 32 ensures that the specific portion Q is extracted accurately even when there is a difference between the component included in the DRR image D and the component included in the X-ray fluoroscopic image A. , DRR image D is processed. In other words, machine learning is performed using the DRR image D in a state in which the component included in the DRR image D corresponds to the component included in the fluoroscopic image A as the input teacher image T1.

Specifically, as shown in FIG. 8, the teacher image generation unit 32 adds an object shape image B including an image relating to a predetermined object separate from the subject P to the generated DRR image D. The input teacher image T1 including at least one of the image and the image in which the object shape image B is subtracted from the generated DRR image D is generated. The teacher image generation unit 32 adds at least a two-dimensional object shape image B to the generated DRR image D and subtracts the two-dimensional object shape image B from the generated DRR image D. By doing one of them, the input teacher image T1 is generated.

The object shape image B includes at least one of an image imitating a component included in the X-ray fluoroscopic image A and an image obtained by photographing the component included in the X-ray fluoroscopic image A. In the object shape image B, detectors (first to fourth X-ray detectors 42a to 42d) for detecting X-rays and subject P are placed as predetermined objects separate from the subject P. It includes at least one of a structure including at least one of the top plate 11 and a member placed in the body of the subject P including the marker E, the catheter, and at least one of the stents. That is, the object shape image B is an image including components that are not included in the DRR image D but are included in the X-ray fluoroscopic image A. Further, the object shape image B is not limited to the above-mentioned structures and members, but also includes a figure that imitates noise due to scattering of a treatment beam by an irradiation device.

For example, when the cover F of the FPD, which is a part of the X-ray detection unit 42, is reflected on the X-ray fluoroscopic image A as a structure that is an object separate from the subject P, the cover F of the FPD is reflected on the DRR image D. Machine learning is performed using the image to which the object shape image B imitating the above is added as the input teacher image T1. In this case, the object shape image B is represented as a linear image imitating the cover F of the FPD. Further, as shown in FIG. 9, when performing radiotherapy, horizontal noise may be generated in the X-ray fluoroscopic image A due to the influence of scattered rays generated by the treatment beam by the radiotherapy device 1. In this case, the noise caused by the treatment beam can be imitated by adding the object shape image B, which is a linear image, to the DRR image D as a horizontal line.

Further, for example, as shown in FIG. 10, when the marker E made of metal is placed in the body of the subject P, the marker E is reflected in both the fluoroscopic image A and the DRR image D. .. The marker E is placed in the body of the subject P by puncture or the like before acquiring the CT image data C as a mark for specifying the position of the specific portion Q of the subject P when performing radiotherapy. It is a thing. However, between the time when the CT image data C is acquired and the time when the radiotherapy is performed, the marker E indwelled in the body of the subject P falls off, and the position of the marker E moves. , May occur. In such a case, the position and number of markers E included in the DRR image D generated based on the CT image data C and the position and number of markers E included in the fluoroscopic image A will change. .. When there is this change, the accuracy of extracting the specific part Q using the learning model may decrease. Therefore, when the learning model is generated, the number of markers E is changed by performing machine learning using a plurality of DRR images D whose positions and numbers of markers E are changed in advance as input teacher images T1. Also generates a learning model that can extract a specific part Q. In this case, the object shape image B is represented by a circular image imitating the marker E. Then, the input teacher image T1 is generated by adding a circular image imitating the marker E to the DRR image D.

In this way, the teacher image generation unit 32 adds the two-dimensional object shape image B to the generated DRR image D, and adds the two-dimensional object shape image B to the generated DRR image D. It is configured to generate the input teacher image T1 by doing at least one of the reductions.

Further, apart from the case where the two-dimensional object shape image B is used, the CT image data C having the three-dimensional pixel value data is processed three-dimensionally to obtain the component included in the X-ray fluoroscopic image A. , It is also conceivable to generate the input teacher image T1 in a state where the DRR image D is associated with each other. That is, the teacher image generation unit 32 adds the three-dimensional object shape pixel value data to the CT image data C, which is the three-dimensional data obtained by computer tomography, and the three-dimensional object shape to the three-dimensional data. It is configured to generate the input teacher image T1 by projecting it in two dimensions by a digital reconstruction simulation in a state where at least one of the pixel value data is reduced.

For example, consider a case where a DRR image D imitating an X-ray fluoroscopic image A in which the marker E is dropped and moved is generated as an input teacher image T1. In the CT image data C, since the marker E placed in the body of the subject P is a metal, it shows a very high CT value as compared with the body tissue of the subject P. Therefore, based on the CT value, it is possible to specify a pixel portion (voxel) representing the marker E from the CT image data C of the subject P acquired with the marker E placed in the body. .. That is, a pixel portion (voxel) larger than a certain value (for example, a CT value of 1000) is designated as the marker E. Therefore, the pixel portion (voxel) indicating the designated marker E is removed, and the CT value having the same value as the marker E is added to an appropriate position in the CT image data. In this way, the teacher image generation unit 32 adds the object shape pixel value data (CT value for the marker E) to the CT image data C which is three-dimensional data, and reduces the object shape pixel value data. This makes it possible to virtually freely change the position and number of markers E on the CT image data C. Then, the teacher image generation unit 32 generates a plurality of input teacher images T1 in which the position and the number of the markers E are changed by the digital reconstruction simulation in the state where the position and the number of the markers E are changed.

Further, when the teacher image generation unit 32 generates the input teacher image T1, the object shape image B including an image relating to a predetermined object different from the subject P with respect to the DRR image D is randomly generated at a random position. It is configured to perform at least one of adding by size and subtracting the object shape image B to a random position by a random size with respect to the DRR image D. For example, the input teacher image T1 is generated by adding the object shape image B represented by a straight line to the DRR image D while randomly changing the position, angle, line thickness, pixel value, number of lines, and the like. Further, the input teacher image T1 is generated by adding the circular object shape image B to the DRR image D while randomly changing the position, size (radius), pixel value, and number. Further, the input teacher image T1 is generated by randomly reducing the number of markers E included in the DRR image D.

As described above, the teacher image generation unit 32 adds the object shape image B in two dimensions of random size to the random position of the generated DRR image D, and the object shape image B in two dimensions. By randomly combining the reduction, the addition of the three-dimensional object shape pixel value data to the three-dimensional CT image data C, and the reduction of the three-dimensional object shape data, a plurality of pieces are used. Pattern input A teacher image T1 is generated.

As shown in FIG. 5, the teacher image generation unit 32 generates the output teacher image T2 based on the DRR image D in the same manner as the input teacher image T1. Specifically, the DRR image D is generated only for the pixel portion (voxel) related to the specific portion Q registered by the doctor in the CT image data C. That is, virtual X-ray imaging is performed only on the specific portion Q, not on the entire CT image data C. As a result, the obtained DRR image D for the specific portion Q is divided into two regions, a region including the specific portion Q and a region not including the specific portion Q, and a binarized output teacher image T2 is generated. To do. When a plurality of input teacher images T1 are generated based on a plurality of DRR images D, a plurality of output teacher images T2 corresponding to each of the plurality of input teacher images T1 are generated. That is, the output teacher image T2 is generated by using the same shooting geometry as the shooting geometry in each of the plurality of input teacher images T1. In other words, for each of the plurality of input teacher images T1, an image showing a specific portion Q in the same region as the input teacher image T1 is generated as the output teacher image T2.

<Regarding the generation of the first learning model>
As shown in FIG. 5, the learning model generation unit 33 acquires the input teacher image T1 generated by the teacher image generation unit 32 as an input layer. Then, the learning model generation unit 33 uses the output teacher image T2, which is a 2-channel label image generated by the teacher image generation unit 32 so as to represent the region of the specific portion Q registered in the treatment planning device 2, as an output layer. get. Then, the learning model generation unit 33 performs machine learning using the input layer and the output layer. The learning model generation unit 33 generates the first learning model M1 by learning the convolution layer used as the learning model. Then, the specific part processing unit 51 extracts the region of the specific part Q in the X-ray fluoroscopic image A by using the generated first learning model M1. By using the X-ray fluoroscopic image A as an input layer and using the first learning model M1 generated by the learning model generation unit 33, an output image Aα which is a 2-channel label image is generated as an output layer.

In image processing, for example, a specific part Q is extracted by using semantic segmentation, which is a method of classifying (labeling) each pixel in an image into classes. As a method of semantic segmentation, FCN (Full Convolutional Network) is used. The convolutional neural network used in FCN has, for example, the configuration shown in FIG. The intermediate layer is composed of only the convolution layer, and the parameters are determined by machine learning.

(Regarding marker extraction by the second learning model)
Subsequently, the second learning model M2 will be described with reference to FIG. Unlike the first learning model M1 used to extract the specific portion Q, the second learning model M2 is used to extract the marker E placed in the body of the subject P.

As shown in FIG. 11, the second learning model M2 is an input teacher image for marker extraction, which is an input teacher image T1 generated in a state where the marker E for showing the specific portion Q is placed in the body of the subject P. It is generated by performing machine learning by the learning model generation unit 33 using T11 and the output teacher image T3 for marker extraction, which is an image showing the position of the marker E.

The teacher image generation unit 32 generates the output teacher image T3 for marker extraction based on the DRR image D, similarly to the input teacher image T1. Specifically, the marker E is virtually projected onto the pixel portion of the CT image data C corresponding to the marker E by the virtual X-ray tube 41α and the virtual X-ray detector 42α. X-ray photography is performed. As a result, the obtained image is divided into two regions, a marker E and a region other than the marker E, and a binarized output teacher image T3 for marker extraction is generated. When a plurality of marker extraction input teacher images T11 are generated based on a plurality of DRR images D, a plurality of marker extraction input teacher images T11 corresponding to each of the plurality of marker extraction input teacher images T11 are similarly generated as in the output teacher image T2. The output teacher image T3 for marker extraction is generated.

Similar to the generation of the first learning model M1, the learning model generation unit 33 performs machine learning using the marker extraction input teacher image T11 as an input layer and the marker extraction output teacher image T3 as an output layer to generate the marker E. A second learning model M2 for extraction is generated. The algorithm for generating the second learning model M2 is the same as that of the first learning model M1.

The specific portion Q is extracted based on the first learning model M1 and the second learning model M2 generated as described above. Then, a signal for controlling the position to irradiate the radiation and the timing to irradiate the radiation is sent to the radiotherapy device 1 based on the position of the extracted specific portion Q.

(Regarding the generation method of the first learning model)
Next, a method of generating the first learning model M1 by the learning device 3 will be described with reference to FIG.

First, in step 101, CT image data C is acquired.

Next, in step 102, with respect to the DRR image D for the region including the specific portion Q of the subject P acquired based on the CT image data C, an object shape image relating to a predetermined object separate from the subject P is obtained. An input teacher image T1 including at least one of an image in which B is added and an image in which the object shape image B is subtracted from the DRR image D is generated.

Next, in step 103, an output teacher image T2, which is an image showing a specific portion Q in the same region as the DRR image D, is generated based on the CT image data C.

Next, in step 104, the first learning model M1 for extracting the specific portion Q is generated by performing machine learning using the input teacher image T1 and the output teacher image T2.

(Processing for specific partial extraction during radiation therapy)
Next, with reference to FIG. 13, the control processing flow by the X-ray image processing system 100 according to the first embodiment will be described.

First, in step 111, the learning device 3 acquires the CT image data C generated by the treatment planning device 2.

Next, in step 112, the DRR image D is generated by the DRR image generation unit 31.

Next, in step 113, the teacher image generation unit 32 generates the input teacher image T1 and the marker extraction input teacher image T11, and the output teacher image T2 and the marker extraction output teacher image T3.

Next, in step 114, the learning model generation unit 33 generates the first learning model M1 and the second learning model M2.

Next, in step 115, the X-ray imaging unit 4 photographs the X-ray fluoroscopic image A.

Next, in step 116, the specific portion Q is extracted based on the fluoroscopic image A and the first learning model M1 and the second learning model M2.

Next, in step 117, a signal is sent to the radiotherapy device 1 based on the extracted position information of the specific portion Q.

Next, in step 118, the radiation therapy device 1 irradiates the radiation.

(Example of using X-ray image processing system)
Next, an example of using the X-ray image processing system 100 in the first embodiment will be described with reference to FIG. Radiation therapy using the X-ray image processing system 100 according to the first embodiment of the present invention involves irradiating a tumor such as cancer with radiation (for example, high-energy X-ray) to cause a tumor such as cancer. Is a way to treat.

First, a treatment plan and advance preparations are made for radiation therapy. First, in step 121, a marker E, which serves as a mark for indicating the position of the specific portion Q, which is the affected area to be treated, is placed in the body of the subject P by puncture or the like by a doctor. Next, in step 122, the CT image data C of the subject P is acquired by the CT device. Then, in step 123, the doctor plans the treatment in the treatment planning device 2 based on the CT image data C and the clinical examination and examination of the subject P. Then, in step 124, a learning model for extracting a specific portion Q in the X-ray fluoroscopic image A is generated by machine learning based on the acquired CT image data C by the learning device 3. In this way, treatment planning and advance preparation are carried out.

Next, radiation therapy is actually performed. First, in step 125, the X-ray imaging unit 4 performs X-ray imaging on the subject P placed on the top plate 11 of the examination table provided in the radiotherapy apparatus 1, and the X-ray fluoroscopic image A Is obtained. Next, in step 126, the specific portion Q is extracted by the X-ray image processing device 5 by using the above-mentioned learning model for the acquired X-ray fluoroscopic image A. Then, in step 127, radiation therapy is performed by irradiating radiation based on the position of the extracted specific portion Q.

(Effect of the first embodiment)
In the first embodiment, the following effects can be obtained.

In the X-ray image processing system 100 of the first embodiment, as described above, the learning image generation unit (DRR image generation unit 31 and the teacher image generation unit 32) has three-dimensional data (CT) having pixel value data in three dimensions. An object containing an image relating to a predetermined object separate from the subject P with respect to a two-dimensional projected image (DRR image D) about a region including a specific portion Q of the subject P acquired based on the image data C). The input teacher image T1 including at least one of the image to which the shape image B is added and the image in which the object shape image B is subtracted from the DRR image D, and the identification in the same region as the input teacher image T1. It is configured to generate an output teacher image T2 which is an image showing a portion Q or an image excluding a specific portion Q. Then, the learning model generation unit 33 performs machine learning using the input teacher image T1 and the output teacher image T2 to extract the specific portion Q or remove the specific portion Q. It is configured to generate the learning model M1. As a result, at least one of adding and reducing the object shape image B to the DRR image D is performed, so that the components included in the DRR image D correspond to the components included in the fluoroscopic image A. Can be changed. That is, it is possible to generate the first learning model M1 that can handle even when there is a difference between the component included in the DRR image D and the component included in the X-ray fluoroscopic image A. For example, by performing machine learning using an image (an image including an object shape image B) in which a component contained only in the X-ray fluoroscopic image A is pseudo-included in the DRR image D as an input teacher image T1. The first learning model M1 can be generated. Therefore, even when the components included in the X-ray fluoroscopic image A and the DRR image D are different, the specific portion Q can be extracted or the specific portion Q can be extracted by using the first learning model M1 generated by machine learning. It is possible to suppress a decrease in the accuracy of removing the specific portion Q.

Further, in the first embodiment, a further effect can be obtained by configuring as follows.

Further, in the first embodiment, in the object shape image B, a detector for detecting X-rays (first to fourth X-ray detectors 42a to 42d) and a subject P are placed as predetermined objects. At least one of a structure including at least one of the top plate 11 and a member placed in the body of the subject P including at least one of a marker E, a guide wire, and a catheter as a predetermined object is imitated. Includes at least one of an image and an image of at least one of a structure and a member. With this configuration, for example, a structure including at least one of a detector for detecting X-rays (first to fourth X-ray detectors 42a to 42d) and a top plate 11 on which the subject P is placed. And even when at least one of the marker E, the guide wire, and the member placed in the body of the subject P including at least one of the catheters is included in the X-ray image (X-ray fluoroscopic image A). By adding at least one of the structure and the member to the two-dimensional projected image (DRR image D), the input teacher image T1 can be generated so as to correspond to the structure included in the X-ray fluoroscopic image A. .. As a result, even when the components included in the images are different between the X-ray fluoroscopic image A and the DRR image D created based on the three-dimensional data (CT image data C), the first generated by machine learning. The specific part Q can be extracted or the specific part Q can be removed by using the one learning model M1 or the second learning model M2.

Further, in the first embodiment, the marker extraction input teacher image T11, which is the input teacher image T1 generated in a state where the marker E for indicating the specific portion Q is placed in the body of the subject P, and the marker E Obtained by X-ray photography based on the second learning model M2 generated by performing machine learning by the learning model generation unit 33 using the marker extraction output teacher image T3 which is an image showing the position. Either a process of extracting the specific portion Q by extracting the marker E or a process of removing the specific portion Q with respect to the X-ray image (X-ray fluoroscopic image A) including the specific portion Q of the subject P. A marker processing unit 52 for performing one of the two is further provided, and the specific partial processing unit 51 or the marker processing unit 52 is configured to be able to select whether to perform the processing. With this configuration, it is possible to select whether to perform processing of extracting the specific portion Q by extracting the marker E by image processing or directly extracting the specific portion Q by image processing. be able to. As a result, even when the marker E cannot be extracted due to the marker E falling off, either the specific portion Q is directly extracted or the specific portion Q is removed. be able to.

Further, in the first embodiment, the operation unit 53 that accepts the input operation is further provided, and based on the reception of the input of the operation to the operation unit 53, the processing is performed by either the specific partial processing unit 51 or the marker processing unit 52. It is configured so that you can choose whether to do it. With this configuration, the operator can select which of the specific partial processing unit 51 and the marker processing unit 52 is to be used by an input operation to the operation unit 53. As a result, it can be easily selected when selecting which of the output result by the specific partial processing unit 51 and the output result by the marker processing unit 52 is to be used.

Further, in the first embodiment, the first control unit 30 including the learning image generation unit (DRR image generation unit 31 and the teacher image generation unit 32) and the learning model generation unit 33, and the second control unit 30 including the specific partial processing unit 51 are included. A control unit 50 (250) is further provided. As a result, the process of generating the first learning model M1 and the second learning model M2 and the process of performing at least one of extracting the specific portion Q and removing the specific portion Q are performed by a separate control unit. It can be carried out. As a result, since the learning model can be generated in advance at a time point prior to the time point at which the subject P is subjected to radiotherapy, an increase in the time required for radiotherapy can be suppressed.

Further, in the first embodiment, when the learning image generation unit (DRR image generation unit 31 and the teacher image generation unit 32) generates the input teacher image T1, the subject is subjected to the two-dimensional projection image (DRR image D). An object shape image B containing an image relating to a predetermined object separate from P is added to a random position at a random size, and an object shape image B is added to a DRR image D at a random size at a random size. It is configured to do at least one of the reduction with. With this configuration, the DRR image D is configured so that the structures and members included in the X-ray fluoroscopic image A correspond to the X-ray fluoroscopic image A regardless of the position and size. can do. As a result, when the first learning model M1 is generated based on the input teacher image T1, the first learning model M1 can be generated so as to correspond to the X-ray fluoroscopic image A having various configurations, so that X-rays can be generated. Regardless of the configuration of the components included in the perspective image A, it is possible to accurately extract the specific part Q using the first learning model M1 generated by machine learning and to accurately remove the specific part Q. it can.

Further, in the first embodiment, the learning image generation unit (DRR image generation unit 31 and teacher image generation unit 32) is based on the three-dimensional data (CT image data C) obtained by computer tomography, and the two-dimensional projection image (DRR). Image D) is generated by digital reconstruction simulation, and a two-dimensional object shape image B is added to the generated DRR image D, and a two-dimensional object shape image B is added to the generated DRR image D. Is configured to generate the input teacher image T1 by subtracting and at least one of them. With this configuration, when the input teacher image T1 is generated based on the two-dimensional object shape image B that imitates the component included in the X-ray fluoroscopic image A, the object shape image B is linear or curved. , And can be represented by a simple shape such as a circle. Therefore, when the input teacher image T1 is generated by making the DRR image D correspond to the configuration of the X-ray fluoroscopic image A, the input teacher image T1 can be easily generated. As a result, using the first learning model M1 and the second learning model M2 generated by machine learning based on the DRR image D, it is easy to reduce the accuracy when extracting the specific part Q and removing the specific part Q. Can be suppressed.

Further, in the first embodiment, the learning image generation unit (DRR image generation unit 31 and the teacher image generation unit 32) has a three-dimensional object shape pixel with respect to the three-dimensional data (CT image data C) obtained by computer tomography. Input teacher by projecting in two dimensions by digital reconstruction simulation with at least one of adding value data and subtracting three-dimensional object shape pixel value data with respect to CT image data C. It is configured to generate image T1. With this configuration, in the CT image data C, a three-dimensional model imitating a three-dimensional object placed in the body of the subject P is added to the body of the subject P on the CT image data C. The input teacher image T1 can be generated by generating the three-dimensional projection image (DRR image D). Therefore, it is possible to generate an input teacher image T1 that is more compatible with the X-ray fluoroscopic image A obtained when a three-dimensional object (member) is actually placed in the body of the subject P at the time of treatment. As a result, it is possible to further suppress a decrease in accuracy when extracting the specific portion Q using the first learning model M1 and the second learning model M2 generated by machine learning, or when removing the specific portion Q. it can.

[Second Embodiment]
The configuration of the X-ray image processing system 200 according to the second embodiment will be described with reference to FIGS. 15 to 20. This second embodiment is different from the first embodiment in which the specific portion Q is extracted (tracked) in the radiotherapy, and in the angiography, the specific portion Q which is the bone portion H is extracted (tracked) from the X-ray image A2. It is configured to be removed and displayed. In the figure, parts having the same configuration as that of the first embodiment are designated by the same reference numerals, and the description thereof will be omitted. Here, in the second embodiment, the specific portion Q includes the bone portion H.

(Configuration of X-ray image processing system according to the second embodiment)
As shown in FIGS. 15 and 16, the X-ray image processing system 200 according to the second embodiment of the present invention includes a display device 201, a CT device 202, a learning device 203, an X-ray imaging unit 204, and an X-ray image processing. The device 205 is provided. As shown in FIG. 17, the X-ray image processing system 200 was learned to remove the bone H from the X-ray image A2 by machine learning using the DRR image D201 and the DRR image D202 during angiography. Based on the third learning model M3, the bone H is removed from the X-ray image A2. Further, the X-ray image processing system 200 accurately removes the bone portion H even when there is a difference between the components included in the DRR image D201 and the DRR image D202 and the components included in the X-ray image A2. It is configured to generate a third learning model M3 that can be used. The third learning model M3 is an example of the "first learning model" in the claims.

The display device 201 is configured to display the angiographic image K generated by the X-ray image processing device 205.

The CT device 202 generates a CT image of the subject P.

As shown in FIG. 15, the learning device 203 includes a first control unit 230. The first control unit 230 has, for example, a CPU. The learning device 203 has, for example, a CPU, a GPU, a ROM, a RAM, and the like as a hardware configuration. Further, the learning device 203 includes an HDD, a non-volatile memory, or the like as a storage unit. The learning device 203 generates three-dimensional CT image data C based on the two-dimensional CT image generated by the CT device 202. Then, the learning device 203 generates the third learning model M3 based on the generated CT image data C.

As shown in FIG. 18, the first control unit 230 includes a DRR image generation unit 231, a teacher image generation unit 232, and a learning model generation unit 233 as functional configurations. The DRR image generation unit 231, the teacher image generation unit 232, and the learning model generation unit 233 are configured as programs (software) as in the first embodiment. That is, the first control unit 230 is configured to function as a DRR image generation unit 231, a teacher image generation unit 232, and a learning model generation unit 233 by executing a program (software). The DRR image generation unit 231 and the teacher image generation unit 232 are examples of the "learning image generation unit" in the claims.

As shown in FIG. 17, the DRR image generation unit 231 generates the DRR image D201 based on the three-dimensional CT image data C generated by the CT image acquired by the CT apparatus 202 as in the first embodiment. To do. That is, the DRR image D201 is generated by performing a virtual X-ray imaging on the CT image data C with the same imaging geometry as the X-ray imaging performed on the subject P when performing angiography. Further, in the same imaging geometry as when the DRR image D201 was generated, the DRR image D202 is generated in a state where the portion corresponding to the bone portion H is excluded from the pixel portion (voxel) constituting the CT image data C. For example, when the DRR image D202 is generated, the Voxel having a CT value of more than 100 is assumed to be the bone H, and the DRR image D202 is generated so as to exclude the bone H.

As shown in FIG. 19, the teacher image generation unit 232 has an object shape image B including an image relating to a predetermined object separate from the subject P with respect to the generated DRR image D201, as in the first embodiment. The input teacher image T201, which is the added image, is generated. Further, an object shape image B of the same size is added to the DRR image D202 generated so as to exclude the specific portion Q including the bone portion H in the same region as the input teacher image T201 at the same position as the input teacher image T201. The output teacher image T202, which is the image being displayed, is generated. That is, the DRR image generation unit 231 adds an object shape image B of the same size at the same position to the generated DRR image D201 and DRR image D202 acquired by the same shooting geometry in the same area. The images are generated as an input teacher image T201 and an output teacher image T202, respectively.

As shown in FIG. 20, the learning model generation unit 233 performs machine learning with the input teacher image T201 as the input layer and the output teacher image T202 as the output layer to generate the third learning model M3. At this time, the learning model generation unit 233 performs machine learning with a plurality of input teacher images T201 and output teacher images T202 in which the type, number, size, pixel value, density, etc. of the object shape image B are randomly added. ..

As shown in FIG. 16, the X-ray imaging unit 204 includes an X-ray irradiation unit 241, an X-ray detection unit 242, and a top plate 243. The X-ray irradiation unit 241 irradiates the subject P placed on the top plate 243 with X-rays. The X-ray detection unit 242 detects the X-rays emitted by the X-ray irradiation unit 241. The X-ray detector 242 has an FPD. A subject P to be X-rayed is placed on the top plate 243.

As shown in FIG. 15, the X-ray image processing apparatus 205 includes a second control unit 250. The second control unit 250 has, for example, a CPU. The X-ray image processing apparatus 205 has, for example, a CPU, a GPU, a ROM, a RAM, and the like as a hardware configuration. Further, the X-ray image processing device 205 includes an HDD, a non-volatile memory, or the like as a storage unit.

As shown in FIG. 18, the second control unit 250 includes a bone processing unit 251 as a functional configuration. The bone processing unit 251 is composed of a program (software). That is, the second control unit 250 is configured to function as the bone processing unit 251 by executing a program (software).

The X-ray image processing device 205 acquires the third learning model M3 that has been learned by machine learning from the learning device 203. Then, the bone processing unit 251 generates an output image Aβ (see FIG. 20) in which the bone H is removed from the X-ray image A2 captured by the X-ray imaging unit 204 based on the acquired third learning model M3. It is configured to do. Then, the X-ray image processing apparatus 205 performs angiography using the captured output image Aβ to perform angiography in a state where the bone portion H is removed, and angiography which is the result of the angiography. The image K is displayed on the display device 201. The bone processing unit 251 is an example of the "specific partial processing unit" in the claims.

The other configurations of the second embodiment are the same as those of the first embodiment.

(Control processing by the X-ray image processing system 200 of the second embodiment)
Next, with reference to FIG. 21, the control processing flow by the X-ray image processing system 200 according to the second embodiment will be described.

First, in step 211, a CT image is acquired from the CT device 202. Then, the DRR image generation unit 231 generates the three-dimensional CT image data C.

Next, in step 212, the DRR image generation unit 231 generates the DRR image D201 and the DRR image D202.

Next, in step 213, the input teacher image T201 and the output teacher image T202 are generated.

Next, in step 214, the third learning model M3 is generated.

Next, in step 215, the X-ray image A2 is photographed by the X-ray photographing unit 204.

Next, in step 216, the bone portion H is removed from the X-ray image A2 captured by the X-ray image processing device 205.

Next, in step 217, angiography is performed using the X-ray image A2 from which the bone portion H has been removed, and the angiographic image K, which is the result of the angiography, is displayed on the display device 201.

(Effect of the second embodiment)
In the second embodiment, the following effects can be obtained.

In the second embodiment, as described above, the specific portion Q includes the bone portion H, and the learning image generation unit (DRR image generation unit 231 and the teacher image generation unit 232) includes the bone in the same region as the input teacher image T201. An object shape image B of the same size is added to the two-dimensional projected images (DRR image D201 and DRR image D202) generated so as to exclude the specific portion Q including the part H at the same position as the input teacher image T201. An image in which the object shape image B of the same size is reduced at the same position with respect to the DRR image D201 and the DRR image D202 generated so as to exclude the specific portion Q including the bone portion H. The output teacher image T202 is configured to include at least one of the above. With this configuration, even when the DRR image D201 and the DRR image D202 based on the three-dimensional data (CT image data C) are used for both the input teacher image T201 and the output teacher image T202, the X-ray image A2 can be configured. The third learning model M3 can be generated so as to be compatible. As a result, even when the DRR image D201 and the DRR image D202, which are two-dimensional projection images based on the CT image data C, are used for both the input teacher image T201 and the output teacher image T202, the configuration of the input teacher image T201 is an X-ray image. By making it correspond to the configuration of A2, it is possible to suppress a decrease in accuracy when extracting the specific portion Q (bone portion H) and when removing the specific portion Q (bone portion H).

The other effects of the second embodiment are the same as those of the first embodiment.

[Modification example]
It should be noted that the embodiments disclosed this time are exemplary in all respects and are not considered to be restrictive. The scope of the present invention is shown by the scope of claims rather than the description of the above-described embodiment, and further includes all modifications (modifications) within the meaning and scope equivalent to the scope of claims.

For example, in the first embodiment, the X-ray image processing system 100 shows an example including a radiotherapy device 1 and a treatment planning device 2, but the present invention is not limited to this. For example, it may be configured not to include at least one of the radiotherapy device 1 and the treatment planning device 2. Similarly, in the second embodiment, the X-ray image processing system 200 shows an example including the CT device 202, but the present invention is not limited to this. For example, the CT apparatus 202 may not be provided in the X-ray image processing system 200, and may be configured to acquire a CT image generated in advance.

Further, in the second embodiment, an example is shown in which the object shape image B of the same size is added to the same position in both the input teacher image T201 and the output teacher image T202, but the present invention has been added to this. Not limited. For example, the object shape image B of the same size may be reduced at the same position with respect to both the input teacher image T201 and the output teacher image T202. That is, when CT imaging is performed with a catheter or the like placed in the body of the subject P in advance, the image representing the catheter included in the DRR images D201 and D202 is input so as to be subtracted from the DRR images D201 and D202. The teacher image T201 and the output teacher image T202 may be generated. Further, when generating the input teacher image T201 and the output teacher image T202, both the process of adding the object shape image B and the process of reducing the object shape image B may be performed.

Further, in the second embodiment, an example in which the marker processing unit 52 and the operation unit 53 are not provided is shown, but the present invention is not limited to this. For example, in the X-ray image processing system 200 of the second embodiment, the marker E may be placed in the body of the subject P, and the marker processing unit 52 and the operation unit 53 may be provided.

Further, in the first and second embodiments, the example in which the DRR image D (D201, D202) is generated by the learning device 3 (203) is shown, but the present invention is not limited to this. For example, the treatment planning device 2 or the like may be configured to generate DRR images D (D201, D202).

Further, in the first and second embodiments, the object shape image B includes detectors (first to fourth X-ray detectors 42a to 42d) and a subject P for detecting X-rays as predetermined objects. A structure including at least one of the top plate 11 (243) to be placed, and a member to be placed in the body of the subject P including at least one of a marker E, a guide wire, and a catheter as predetermined objects. Although an example including at least one of an image imitating at least one of the above and an image of at least one of the structure and the member taken is shown, the present invention is not limited to this. For example, the object shape image B may be configured to be an image that imitates noise contained only in the X-ray image A2 (X-ray fluoroscopic image A).

Further, in the first embodiment, the marker extraction input teacher image T11, which is the input teacher image T1 generated in a state where the marker E for indicating the specific portion Q is placed in the body of the subject P, and the marker E. Obtained by X-ray photography based on the second learning model M2 generated by performing machine learning by the learning model generation unit 33 using the marker extraction output teacher image T3 which is an image showing the position of. Either a process of extracting the specific portion Q by extracting the marker E or a process of removing the specific portion Q with respect to the X-ray image (X-ray fluoroscopic image A) including the specific portion Q of the subject P. An example has been shown in which a marker processing unit 52 that performs one of the above is further provided, and the specific partial processing unit 51 and the marker processing unit 52 are configured so that the processing can be selected. Not limited to this. For example, the DRR image D may be generated based on the CT imaging performed in the state where the marker E is not placed in the subject P. That is, the marker processing unit 52 may not be provided, and the specific portion Q may be extracted or removed only by the processing by the specific partial processing unit 51.

Further, in the first embodiment, the operation unit 53 that accepts an input operation is further provided, and processing is performed by either the specific partial processing unit 51 or the marker processing unit 52 based on the reception of the input of the operation to the operation unit 53. Although an example is shown in which it is possible to select whether to perform the above, the present invention is not limited to this. For example, if it is detected that the same number of markers E are included in the same position in both the DRR image D and the X-ray fluoroscopic image A (X-ray image) without the operation unit 53, the markers It is possible to automatically control that the processing unit 52 selects to perform processing and that if the same number of markers E are not detected at the same position, the specific partial processing unit 51 selects to perform processing. It may be configured as follows.

Further, in the first and second embodiments, the first control unit 30 including a learning image generation unit (DRR image generation unit 31, 231 and a teacher image generation unit 32, 232) and a learning model generation unit 33 (233). Although (230) and a second control unit 50 (250) including a specific partial processing unit 51 (bone processing unit 251) are further provided, the present invention is not limited to this. For example, one control unit includes a DRR image generation unit 31 (231), a teacher image generation unit 32 (232), a learning model generation unit 33 (233), and a specific partial processing unit 51 (bone processing unit 251). It may be configured as follows. That is, one control unit may be configured to generate the first to third learning models M1 to M3 and extract or remove the specific portion Q. Further, each function of the DRR image generation unit 31 (231), the teacher image generation unit 32 (232), the learning model generation unit 33 (233), and the specific partial processing unit 51 (bone processing unit 251) is individually hardware. Wear may be configured.

Further, in the first and second embodiments, when the learning image generation unit (DRR image generation unit 31, 231 and the teacher image generation unit 32, 232) generates the input teacher image T1 (T201), the two-dimensional projection is performed. An object shape image B containing an image relating to a predetermined object separate from the subject P with respect to the images (DRR images D, D201, D202) is added to a random position in a random size, and a DRR image D (DRR image D, D201, D202) is added. An example is shown in which the object shape image B is reduced to a random position by a random size with respect to D201 and D202), and at least one of them is performed, but the present invention is not limited to this. .. For example, the object shape image B can be added to the corresponding location of the DRR image D (D201, D202) by specifying the position of the component included in the X-ray image A2 (X-ray fluoroscopic image A) in advance. , The object shape image B may be reduced.

Further, in the second embodiment, the specific portion Q includes the bone portion H, and the learning image generation unit (DRR image generation unit 231 and the teacher image generation unit 232) includes the bone portion H in the same region as the input teacher image T201. With respect to the two-dimensional projected images (DRR images D201 and D202) generated so as to exclude the specific portion Q including the above, an image in which an object shape image B of the same size is added at the same position as the input teacher image T201. Includes at least one of the DRR image D201 (D202) generated so as to exclude the specific portion Q including the bone portion H, and the image in which the object shape image B of the same size is reduced at the same position. Although an example configured to generate the output teacher image T202 is shown as described above, the present invention is not limited to this. For example, the specific portion Q may be configured not to include the bone portion H. That is, the specific part Q is extracted by adding the object shape image B of the same size to both the input teacher image T201 and the output teacher image T202 with the tumor of the subject P as the specific part Q. It may be configured as.

Further, in the first and second embodiments, the learning image generation unit (DRR image generation unit 31, 231 and teacher image generation unit 32, 232) is based on three-dimensional data (CT image data C) obtained by computer tomography. Then, the two-dimensional projected images (DRR images D, D201, D202) are generated by the digital reconstruction simulation, and the two-dimensional object shape image B is added to the generated DRR images D (D201, D202). , The input teacher image T1 (T201) is generated by subtracting the two-dimensional object shape image B with respect to the generated DRR image D (D201, D202) and at least one of them. However, the present invention is not limited to this. 3 from the in-vivo image of subject P acquired by MRI (Magnetic Resonance Imaging) instead of generating DRR images D (D201, D202) based on CT image data C by computed tomography. A dimensional model may be acquired and a two-dimensional projected image may be generated based on the acquired three-dimensional model.

Further, in the first and second embodiments, the learning image generation unit (DRR image generation unit 31, 231 and the teacher image generation unit 32, 232) refers to the three-dimensional data (CT image data C) obtained by computer tomography. Then, in a state where at least one of adding the three-dimensional object shape pixel value data and subtracting the three-dimensional object shape pixel value data with respect to the CT image data C is performed, the digital reconstruction simulation is performed in two dimensions. An example is shown in which the input teacher image T1 (T201) is generated by projecting to, but the present invention is not limited to this. For example, at the time of taking a CT image, an object imitating a component included in the X-ray fluoroscopic image A (X-ray image A2) is taken at the same time as the subject P, so that the CT image data C has an object shape from the beginning. It may be configured to include pixel value data. Further, the process of adding the two-dimensional object shape image B to the generated DRR images D (D201, D202) and reducing the two-dimensional object shape image B, and the CT image data C The input teacher image T1 (T201) is generated in a state where both the processing of adding the three-dimensional object shape pixel value data and the processing of reducing the three-dimensional object shape pixel value data are performed. You may.
[Aspect]
It will be understood by those skilled in the art that the above exemplary embodiments are specific examples of the following embodiments.

(Item 1)
An object containing an image relating to a predetermined object separate from the subject with respect to a two-dimensional projected image of a region including a specific part of the subject acquired based on the three-dimensional data having pixel value data in three dimensions. An input teacher image including at least one of an image to which a shape image is added and an image in which the object shape image is subtracted from the two-dimensional projected image, and the identification in the same region as the input teacher image. A learning image generation unit that generates an output teacher image that is an image showing a part or an image excluding the specific part, and
A learning model generation unit that extracts the specific portion or generates a first learning model for removing the specific portion by performing machine learning using the input teacher image and the output teacher image. ,
Based on the trained first learning model, a process of extracting the specific portion and removing the specific portion of the X-ray image including the specific portion of the subject acquired by X-ray imaging. A medical image processing apparatus including a specific partial processing unit that performs either processing.

(Item 2)
The object shape image includes a structure including at least one of a detector for detecting X-rays and a top plate on which the subject is placed as the predetermined object, and a marker and a guide as the predetermined object. At least one image of at least one of a wire and a member placed in the body of the subject including at least one of a catheter, and an image of at least one of the structure and the member. The medical image processing apparatus according to item 1, which includes one.

(Item 3)
An input teacher image for marker extraction, which is an input teacher image generated in a state where a marker for indicating the specific portion is placed in the body of the subject, and a marker extraction output, which is an image showing the position of the marker. An X-ray image including the specific part of the subject acquired by X-ray photography based on the second learning model generated by performing machine learning by the learning model generation unit using the teacher image. On the other hand, a marker processing unit that performs either a process of extracting the specific portion by extracting the marker or a process of removing the specific portion is further provided.
The medical image processing apparatus according to any one of items 1 or 2, wherein the specific partial processing unit and the marker processing unit are configured to select whether to perform processing.

(Item 4)
It also has an operation unit that accepts input operations.
Item 3. The medical use according to item 3, wherein the specific partial processing unit or the marker processing unit can select which of the specific partial processing unit and the marker processing unit performs the processing based on the reception of the input of the operation to the operation unit. Image processing device.

(Item 5)
A first control unit including the learning image generation unit and the learning model generation unit,
The medical image processing apparatus according to any one of items 1 to 4, further comprising a second control unit including the specific partial processing unit.

(Item 6)
When generating the input teacher image, the learning image generation unit randomly performs the object shape image including an image relating to the predetermined object separate from the subject with respect to the two-dimensional projected image at a random position. Items 1 to 5 are configured to perform at least one of adding in a large size and subtracting the object shape image to a random position with a random size with respect to the two-dimensional projected image. The medical image processing apparatus according to any one of the above items.

(Item 7)
The specific part includes the bone part and includes the bone part.
The learning image generation unit has the same size at the same position as the input teacher image with respect to the two-dimensional projected image generated so as to exclude the specific portion including the bone portion in the same region as the input teacher image. The object shape image of the same size is subtracted at the same position with respect to the image to which the object shape image of the above is added and the two-dimensional projection image generated so as to exclude the specific portion including the bone portion. The medical image processing apparatus according to any one of items 1 to 6, which is configured to generate the output teacher image so as to include at least one of the images.

(Item 8)
The learning image generation unit generates the two-dimensional projection image by digital reconstruction simulation based on the three-dimensional data obtained by computer tomography, and the object is two-dimensional with respect to the generated two-dimensional projection image. It is configured to generate the input teacher image by doing at least one of adding a shape image and subtracting the two-dimensional object shape image from the generated two-dimensional projected image. The medical image processing apparatus according to any one of items 1 to 7.

(Item 9)
The learning image generation unit adds the three-dimensional object shape pixel value data to the three-dimensional data obtained by computer tomography, and subtracts the three-dimensional object shape pixel value data from the three-dimensional data. The item according to any one of items 1 to 8, wherein the input teacher image is generated by projecting the input teacher image in two dimensions by a digital reconstruction simulation in a state where at least one of the above is performed. Medical image processing device.

(Item 10)
An object containing an image relating to a predetermined object separate from the subject with respect to a two-dimensional projected image of a region including a specific part of the subject acquired based on the three-dimensional data having pixel value data in three dimensions. An input teacher image including at least one of an image to which a shape image is added and an image in which the object shape image is subtracted from the two-dimensional projected image, and the identification in the same region as the input teacher image. A learning image generation unit that generates an output teacher image that is an image showing a part or an image excluding the specific part, and
A learning model generation unit that extracts the specific portion or generates a first learning model for removing the specific portion by performing machine learning using the input teacher image and the output teacher image. ,
An X-ray imaging unit that acquires an X-ray image including the specific portion of the subject by X-ray imaging, and an X-ray imaging unit.
Based on the trained first learning model, a process of extracting the specific portion and a process of removing the specific portion from the X-ray image including the specific portion photographed by the X-ray photographing unit. An X-ray image processing system including a specific partial processing unit that performs either of the above.

(Item 11)
An object containing an image of a predetermined object separate from the subject with respect to a two-dimensional projected image of a region including a specific part of the subject acquired based on the three-dimensional data having pixel value data in three dimensions. A step of generating an input teacher image including at least one of an image to which a shape image is added and an image in which the object shape image is subtracted from the two-dimensional projected image.
A step of generating an output teacher image which is an image showing the specific portion in the same region as the input teacher image or an image excluding the specific portion based on the three-dimensional data.
It includes a step of extracting the specific portion or generating a first learning model for removing the specific portion by performing machine learning using the input teacher image and the output teacher image. , How to generate a learning model.

100, 200 X-ray image processing system 1 Irradiation device 2 Treatment planning device 3, 203 Learning device (medical image processing device)
4,204 X-ray imaging unit 5,205 X-ray image processing device (medical image processing device)
11,243 Top plate 30, 230 First control unit 31, 231 DRR image generation unit (learning image generation unit)
32, 232 Teacher image generation unit (learning image generation unit)
33, 233 Learning model generation unit 50, 250 Second control unit 51 Specific part processing unit 52 Marker processing unit 53 Operation unit 251 Bone processing unit (specific part processing unit)

Claims (11)

An object containing an image relating to a predetermined object separate from the subject with respect to a two-dimensional projected image of a region including a specific part of the subject acquired based on the three-dimensional data having pixel value data in three dimensions. An input teacher image including at least one of an image to which a shape image is added and an image in which the object shape image is subtracted from the two-dimensional projected image, and the identification in the same region as the input teacher image. A learning image generation unit that generates an output teacher image that is an image showing a part or an image excluding the specific part, and
A learning model generation unit that extracts the specific portion or generates a first learning model for removing the specific portion by performing machine learning using the input teacher image and the output teacher image. ,
Based on the trained first learning model, a process of extracting the specific portion and removing the specific portion of the X-ray image including the specific portion of the subject acquired by X-ray imaging. A medical image processing apparatus including a specific partial processing unit that performs either processing. The object shape image includes a structure including at least one of a detector for detecting X-rays and a top plate on which the subject is placed as the predetermined object, and a marker and a guide as the predetermined object. At least one image of at least one of a wire and a member placed in the body of the subject including at least one of a catheter, and an image of at least one of the structure and the member. The medical image processing apparatus according to claim 1, wherein the medical image processing apparatus includes one of the two. An input teacher image for marker extraction, which is an input teacher image generated in a state where a marker for indicating the specific portion is placed in the body of the subject, and a marker extraction output, which is an image showing the position of the marker. An X-ray image including the specific part of the subject acquired by X-ray photography based on the second learning model generated by performing machine learning by the learning model generation unit using the teacher image. On the other hand, a marker processing unit that performs either a process of extracting the specific portion by extracting the marker or a process of removing the specific portion is further provided.
The medical image processing apparatus according to claim 1 or 2, wherein the specific partial processing unit and the marker processing unit are configured to select whether to perform processing. It also has an operation unit that accepts input operations.
The third aspect of claim 3, wherein the specific partial processing unit or the marker processing unit can select which of the specific partial processing unit and the marker processing unit performs the processing based on the reception of the input of the operation to the operation unit. Medical image processing device. A first control unit including the learning image generation unit and the learning model generation unit,
The medical image processing apparatus according to any one of claims 1 to 4, further comprising a second control unit including the specific partial processing unit. When generating the input teacher image, the learning image generation unit randomly performs the object shape image including an image relating to the predetermined object separate from the subject with respect to the two-dimensional projected image at a random position. 1 to claim 1, which are configured to perform at least one of adding in a large size and subtracting the object shape image to a random position with a random size with respect to the two-dimensional projected image. The medical image processing apparatus according to any one of 5. The specific part includes the bone part and includes the bone part.
The learning image generation unit has the same size at the same position as the input teacher image with respect to the two-dimensional projected image generated so as to exclude the specific portion including the bone portion in the same region as the input teacher image. The object shape image of the same size is subtracted at the same position with respect to the image to which the object shape image of the above is added and the two-dimensional projection image generated so as to exclude the specific portion including the bone portion. The medical image processing apparatus according to any one of claims 1 to 6, which is configured to generate the output teacher image so as to include at least one of the images. The learning image generation unit generates the two-dimensional projection image by digital reconstruction simulation based on the three-dimensional data obtained by computer tomography, and the object is two-dimensional with respect to the generated two-dimensional projection image. It is configured to generate the input teacher image by doing at least one of adding a shape image and subtracting the two-dimensional object shape image from the generated two-dimensional projected image. The medical image processing apparatus according to any one of claims 1 to 7. The learning image generation unit adds the three-dimensional object shape pixel value data to the three-dimensional data obtained by computer tomography, and subtracts the three-dimensional object shape pixel value data from the three-dimensional data. The present invention according to any one of claims 1 to 8, wherein the input teacher image is generated by projecting the input teacher image in two dimensions by a digital reconstruction simulation in a state where at least one of the above is performed. Medical image processing equipment. An object containing an image relating to a predetermined object separate from the subject with respect to a two-dimensional projected image of a region including a specific part of the subject acquired based on the three-dimensional data having pixel value data in three dimensions. An input teacher image including at least one of an image to which a shape image is added and an image in which the object shape image is subtracted from the two-dimensional projected image, and the identification in the same region as the input teacher image. A learning image generation unit that generates an output teacher image that is an image showing a part or an image excluding the specific part, and
A learning model generation unit that extracts the specific portion or generates a first learning model for removing the specific portion by performing machine learning using the input teacher image and the output teacher image. ,
An X-ray imaging unit that acquires an X-ray image including the specific portion of the subject by X-ray imaging, and an X-ray imaging unit.
Based on the trained first learning model, a process of extracting the specific portion and a process of removing the specific portion from the X-ray image including the specific portion photographed by the X-ray photographing unit. An X-ray image processing system including a specific partial processing unit that performs either of the above. An object containing an image of a predetermined object separate from the subject with respect to a two-dimensional projected image of a region including a specific part of the subject acquired based on the three-dimensional data having pixel value data in three dimensions. A step of generating an input teacher image including at least one of an image to which a shape image is added and an image in which the object shape image is subtracted from the two-dimensional projected image.
A step of generating an output teacher image which is an image showing the specific portion in the same region as the input teacher image or an image excluding the specific portion based on the three-dimensional data.
It includes a step of extracting the specific portion or generating a first learning model for removing the specific portion by performing machine learning using the input teacher image and the output teacher image. , How to generate a learning model.

Images