CN112862738B - Method and device for synthesizing multi-mode image, storage medium and processor - Google Patents
Method and device for synthesizing multi-mode image, storage medium and processor Download PDFInfo
- Publication number
- CN112862738B CN112862738B CN202110387317.5A CN202110387317A CN112862738B CN 112862738 B CN112862738 B CN 112862738B CN 202110387317 A CN202110387317 A CN 202110387317A CN 112862738 B CN112862738 B CN 112862738B
- Authority
- CN
- China
- Prior art keywords
- image
- projection
- sequence
- generator
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 230000002194 synthesizing effect Effects 0.000 title claims abstract description 31
- 238000003860 storage Methods 0.000 title claims abstract description 18
- 238000002595 magnetic resonance imaging Methods 0.000 claims abstract description 28
- 238000004364 calculation method Methods 0.000 claims abstract description 24
- 238000002591 computed tomography Methods 0.000 claims abstract description 24
- 238000001959 radiotherapy Methods 0.000 claims abstract description 15
- 238000010586 diagram Methods 0.000 claims description 17
- 238000006243 chemical reaction Methods 0.000 claims description 9
- 238000005070 sampling Methods 0.000 claims description 4
- 230000008485 antagonism Effects 0.000 claims 1
- 125000004122 cyclic group Chemical group 0.000 claims 1
- 238000007408 cone-beam computed tomography Methods 0.000 abstract description 6
- 238000005516 engineering process Methods 0.000 abstract description 6
- 230000006870 function Effects 0.000 description 25
- 230000008569 process Effects 0.000 description 12
- 238000004590 computer program Methods 0.000 description 10
- 230000015572 biosynthetic process Effects 0.000 description 6
- 238000003786 synthesis reaction Methods 0.000 description 6
- 239000002131 composite material Substances 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000001308 synthesis method Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000005855 radiation Effects 0.000 description 2
- 239000011165 3D composite Substances 0.000 description 1
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 1
- 238000013256 Gubra-Amylin NASH model Methods 0.000 description 1
- 210000003484 anatomy Anatomy 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 210000004872 soft tissue Anatomy 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 210000001519 tissue Anatomy 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10081—Computed x-ray tomography [CT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10088—Magnetic resonance imaging [MRI]
Abstract
The application discloses a method, a device, a storage medium and a processor for synthesizing a multi-mode image. The method comprises the following steps: converting the first modality image into a second modality image by a generator; dose calculations are performed from the second modality images to generate a radiation therapy plan. According to the method and the device, the technical problem that clinical application requirements cannot be met due to the fact that dose calculation cannot be conducted on medical images such as MRI (magnetic resonance imaging), CBCT (computed tomography) and the like in the related technology is solved.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method and apparatus for synthesizing a multi-mode image, a storage medium, and a processor.
Background
Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) are two imaging modalities commonly used in radiation therapy. Both images may describe the anatomy of the patient and display tissue properties in three dimensions. In addition, CT images can provide Electron Density (ED) information required for accurate radiation dose calculations. MRI has been deeply integrated into radiation therapy planning workflows due to the advantages of having good soft tissue, functional imaging capabilities, and the need for additional radiation exposure. However, MRI images do not provide ED information directly and therefore cannot be used as a single image set for RT planning and dose calculation. The prior art registers a patient CT image to MRI and then maps ED information onto MRI. This requires separate scans for both CT and MRI, which provides image doses and is time consuming and costly, thus making it difficult to meet the needs of clinical applications.
Aiming at the technical problems that the medical images such as MRI, CBCT and the like cannot be subjected to dose calculation in the related technology, and the clinical application needs cannot be met, no effective solution is proposed at present.
Disclosure of Invention
The main purpose of the present application is to provide a method, an apparatus, a storage medium and a processor for synthesizing multi-mode images, so as to solve the technical problem that the clinical application needs cannot be satisfied due to the fact that the dose calculation cannot be performed on medical images such as MRI and CBCT in the related technology.
In order to achieve the above object, according to one aspect of the present application, there is provided a method of synthesizing a multi-modal image. The method comprises the following steps: converting the first modality image into a second modality image by a generator; dose calculations are performed from the second modality images to generate a radiation therapy plan.
Further, converting the first modality image to the second modality image by the generator includes: inputting an ordered image sequence of the first modality image into a shared downsampling module and a residual group module to obtain residual characteristics; performing copying and sequence adjusting operations on the residual characteristics to obtain the characteristics with the sequence adjusted; inputting the features with the adjusted sequence into a Conv-LSTM network module so that the generator fully learns different spatial information of different sequence image sequences; performing a merging operation on the characteristics of Conv-LSTM output to obtain merged characteristics; the combined features are input to a shared upsampling module to obtain a second modality image.
Further, the method further comprises: respectively projecting the second mode image on a coronal plane, a sagittal plane and a cross section to obtain three projected plane diagrams; and respectively entering the three projected plan views and the first mode image into a corresponding projection discriminator and a 3D discriminator to discriminate authenticity so as to form a countermeasure with the generator.
Further, the loss function of the generator includes at least one of: an contrast loss function, a projection loss function, a loop consistency loss.
Further, the first modality image is a magnetic resonance imaging image and the second modality image is a computed tomography image.
In order to achieve the above object, according to another aspect of the present application, there is provided a multi-modal image synthesizing apparatus. The device comprises: the first conversion unit is used for converting the first mode image into a second mode image through the generator; a first determination unit for performing dose calculations from the second modality images to generate a radiation therapy plan.
Further, the first conversion unit includes: the first acquisition module is used for inputting the ordered image sequence of the first mode image into the shared downsampling module and the residual error group module so as to acquire residual error characteristics; the second acquisition module is used for executing copying and sequence adjusting operations on the residual error characteristics to obtain the characteristics with the adjusted sequence; the first input module is used for inputting the characteristics with the adjusted sequence into the Conv-LSTM network module so as to enable the generator to fully learn different spatial information of different sequence images; the first merging module is used for executing merging operation on the characteristics output by the Conv-LSTM to obtain merged characteristics; and the second input module is used for inputting the combined characteristics into the shared up-sampling module so as to obtain a second mode image.
Further, the apparatus further comprises: the first acquisition unit is used for respectively projecting the second modal image on the coronal plane, the sagittal plane and the cross section to obtain three projected plan views; and the first processing unit is used for respectively entering the three projected plan views and the first mode image into a corresponding projection discriminator and a 3D discriminator to discriminate authenticity so as to form a countermeasure with the generator.
Further, the loss function of the generator includes at least one of: an contrast loss function, a projection loss function, a loop consistency loss.
Further, the first modality image is a magnetic resonance imaging image and the second modality image is a computed tomography image.
Through the application, the following steps are adopted: converting the first modality image into a second modality image by a generator; the dose calculation is performed according to the second mode image to generate a radiotherapy plan, so that the RT plan can be performed through the MRI image, thereby meeting the clinical application requirement, and further solving the technical problem that the clinical application requirement cannot be met due to the fact that the dose calculation cannot be performed on medical images such as MRI, CBCT and the like.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application, illustrate and explain the application and are not to be construed as limiting the application. In the drawings:
FIG. 1 is a flow chart of a method of synthesizing a multimodal image provided in accordance with an embodiment of the present application;
FIG. 2 is a schematic diagram I of an alternative method of synthesizing a multimodal image provided in accordance with an embodiment of the present application;
FIG. 3 is a schematic diagram II of an alternative method of synthesizing a multimodal image provided in accordance with an embodiment of the present application;
FIG. 4 is a schematic diagram III of an alternative method of synthesizing a multimodal image provided in accordance with an embodiment of the present application; and
fig. 5 is a schematic diagram of a multi-modal image synthesizing apparatus according to an embodiment of the present application.
Detailed Description
It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
In order to make the present application solution better understood by those skilled in the art, the following description will be made in detail and with reference to the accompanying drawings in the embodiments of the present application, it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the present application described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
According to an embodiment of the present application, a method for synthesizing a multi-modal image is provided.
Fig. 1 is a flowchart of a method of synthesizing a multimodal image according to an embodiment of the application. As shown in fig. 1, the method comprises the steps of:
step S101, converting the first mode image into a second mode image through a generator;
step S102, performing dose calculation according to the second mode image to generate a radiotherapy plan.
For example, the first modality image is a magnetic resonance imaging image (MRI), the second modality image is a computed tomography image (CT), the MRI image is input into a generator to obtain a CT image, and RT planning and dose calculation are performed on the CT image since the CT image can provide Electron Density (ED) information required for accurate radiation dose calculation. The method that the MRI and the CT of the patient on the same day are needed to be shot respectively and then the multi-mode registration is carried out in the prior art is avoided, the CT is matched with the MRI, and the whole process is very time-consuming and high in cost. By the method for synthesizing the multi-mode images, the medical images such as MRI can be converted, the converted images can be subjected to dose calculation to generate a radiotherapy plan, clinical application requirements are met, and the technical problem that the medical images such as MRI and CBCT cannot be subjected to dose calculation, so that the clinical application requirements cannot be met is solved.
Optionally, in the method for synthesizing a multi-modal image according to the embodiment of the present application, converting the first-modality image into the second-modality image through the generator includes: inputting an ordered image sequence of the first modality image into a shared downsampling module and a residual group module to obtain residual characteristics; performing copying and sequence adjusting operations on the residual characteristics to obtain the characteristics with the sequence adjusted; inputting the features with the adjusted sequence into a Conv-LSTM network module so that the generator fully learns different spatial information of different sequence image sequences; performing a merging operation on the characteristics of Conv-LSTM output to obtain merged characteristics; the combined features are input to a shared upsampling module to obtain a second modality image.
In this application, the training method of the generator is first introduced, as shown in fig. 2, and fig. 2 is a general frame diagram of the training method of the generator. Wherein the MR synthesis CT and the CT synthesis MR process are included in fig. 2. The broken and solid arrows represent the generation and discrimination processes, respectively, and the rectangles in fig. 2 represent the corresponding generator and discriminator, respectively.
A non-paired MR-to-CT image synthesis method is presented in this application. In the process of image synthesis, the proposed method needs to consider not only the corresponding original image, but also the successive front and rear layer images thereof. Specifically, the proposed scheme is as follows: (1) The Conv-LSTM network structure is introduced into the model in the application, the input of the model is an ordered image sequence instead of a single image, and the image synthesis of the mixed space sequence is realized. Therefore, the model in the application not only maintains the characteristics of few parameters, easy optimization and unpaired 2D Cycle-GAN network, but also ensures the spatial continuity of the generated image to a certain extent. (2) 3 discriminators are added to the model in the present application. The composite image is projected on the cross-section, sagittal plane and coronal plane, respectively, and then an additional discriminator is used to further discriminate the authenticity of the composite image. (3) The non-local module is introduced into the discriminator, so that the association degree of the position information of the front layer and the rear layer of the synthesized 3D image is enhanced.
As shown in fig. 3, fig. 3 is a frame of the generator and its sub-modules. Where Conv represents the convolutional layer, BN represents the Batch Normalization layer, relu represents the Relu active layer, de-Conv represents the deconvolution layer, RB module represents the residual block, and CL module represents the convolutional long-short term memory.
The generator in the application is used for establishing the spatial continuity of the synthesized image, the framework of the generator is shown in fig. 4, the input of the generator is an ordered image sequence, and the flow of the generator is as follows:
1) Inputting the ordered image sequence into a shared downsampling and residual group module to obtain residual characteristics; 2) Performing operations of copying and adjusting the order of the encoded features (copying the residual features and transposing their order); 3) Inputting the features with the adjusted sequence into a Conv-LSTM network module, so that the generator fully learns different spatial information of different sequence image sequences; 4) Performing a merging operation on the Conv-LSTM derived features; 5) The combined features are input to a shared upsampling module to obtain the final output.
Optionally, in the method for synthesizing a multimodal image according to the embodiment of the present application, the method further includes: respectively projecting the second mode image on a coronal plane, a sagittal plane and a cross section to obtain three projected plane diagrams; and respectively entering the three projected plan views and the first mode image into a corresponding projection discriminator and a 3D discriminator to discriminate authenticity so as to form a countermeasure with the generator.
As shown in fig. 4, (a) a 3D composite image block (corresponding to the second modality image described above) is projected on the coronal, sagittal, and cross-section planes, respectively. (b) And respectively entering the three projected planes and the original 3D image into a corresponding projection discriminator and a 3D discriminator to discriminate authenticity and form countermeasure with the generator.
In addition, the generated CT image may be subjected to a projection operation (as shown in fig. 4 (a)), and the obtained coronal, sagittal, and cross-sectional projection images may be subjected to three additional discriminators to discriminate authenticity, as shown in fig. 4 (b). The image projection mode can be analogous to X-ray shooting of a projection target, except that the imaging principle is different from that of X-ray, but continuous voxel information of the target at a certain visual angle can be represented.
Optionally, in the method for synthesizing a multimodal image according to the embodiment of the present application, the loss function of the generator includes at least one of: an contrast loss function, a projection loss function, a loop consistency loss.
Specifically, the universal Loss (resistance Loss function): the training process of the Cycle-GAN model can be considered as learning how to generate samples close to the real target distribution through competition between the generator G and the discriminator D, and the ideal generated samples obtained by the generator should make the discriminator unable to distinguish the generated image from the real CT image. The competition process can be as follows:
project Loss: in order to provide the generated image sequence with continuous spatial information, in this application the Cycle-GAN [45 ]]Three projection discriminators are added to the model, see section 2.3 for details. Input image sequenceFirst through generator G x2y Obtaining a generated target image sequence->Then three orthogonal projections (axial plane projection: P) ax Coronal plane projection P co Sagittal plane projection: p (P) sa ) Obtaining projections of the generated imageFinally by three projection discriminators +.>To distinguish the projection of the generated image from the projection of the real image +.> And thereby with G x2y Forming a secondary countermeasure:
cycle Consistency Loss (cycle consistency loss): since the method used in the present application is an unsupervised learning method, pairing is not requiredIn order to secure similarity between the generated image and the input image, cycle Consistency Loss, which is defined as formula (4), is used herein. Will produce the resultGenerator G through y→x y2x The retrieved result should be similar to the original input:
full Loss (all Loss functions): finally, the combination of X-Y and Y-X, the universal loss, the projection loss and cycle consistency loss in the present application:
wherein lambda is cx And lambda (lambda) cy Representing the intensity of the countermeasures loss weights during X-generation Y and Y-generation X, respectively.
It should be noted that, in the present application, a technical solution is provided for synthesizing CT from MRI and performing RT planning and dose calculation, and the innovation in the specific synthesis method includes: (1) The Conv-LSTM network structure is introduced into the model in the application, the input of the model is an ordered image sequence instead of a single image, and the image synthesis of the mixed space sequence is realized. Therefore, the model in the application not only maintains the characteristics of few parameters, easy optimization and unpaired 2D Cycle-GAN network, but also ensures the spatial continuity of the generated image to a certain extent. (2) 3 discriminators are added to the model in the present application. The composite image is projected on the cross-section, sagittal plane and coronal plane, respectively, and then an additional discriminator is used to further discriminate the authenticity of the composite image.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.
The embodiment of the application also provides a device for synthesizing the multi-mode images, and it is to be noted that the device for synthesizing the multi-mode images in the embodiment of the application can be used for executing the method for synthesizing the multi-mode images provided in the embodiment of the application. The following describes a multi-modal image synthesizing apparatus provided in an embodiment of the present application.
Fig. 5 is a schematic diagram of a multi-modal image compositing apparatus according to an embodiment of the application. As shown in fig. 5, the apparatus includes: a first conversion unit 501, a first determination unit 502.
Specifically, the first conversion unit 501 is configured to convert the first modality image into the second modality image through the generator;
a first determination unit 502 for performing dose calculations from the second modality images to generate a radiation therapy plan.
In summary, in the multi-modal image synthesizing apparatus provided in the embodiments of the present application, a first-modal image is converted into a second-modal image by a generator through a first conversion unit 501; the first determining unit 502 performs dose calculation according to the second modality image to generate a radiotherapy plan, so that the medical image such as MRI can be converted, and the converted image can be subjected to dose calculation to generate the radiotherapy plan, thereby meeting the clinical application requirement, and further solving the technical problem that the medical image such as MRI and CBCT cannot be subjected to dose calculation, and thus cannot meet the clinical application requirement.
Alternatively, in the apparatus for synthesizing a multimodal image provided in the embodiment of the present application, the first conversion unit 501 includes: the first acquisition module is used for inputting the ordered image sequence of the first mode image into the shared downsampling module and the residual error group module so as to acquire residual error characteristics; the second acquisition module is used for executing copying and sequence adjusting operations on the residual error characteristics to obtain the characteristics with the adjusted sequence; the first input module is used for inputting the characteristics with the adjusted sequence into the Conv-LSTM network module so as to enable the generator to fully learn different spatial information of different sequence images; the first merging module is used for executing merging operation on the characteristics output by the Conv-LSTM to obtain merged characteristics; and the second input module is used for inputting the combined characteristics into the shared up-sampling module so as to obtain a second mode image.
Optionally, in the apparatus for synthesizing a multimodal image provided in the embodiment of the present application, the apparatus further includes: the first acquisition unit is used for respectively projecting the second modal image on the coronal plane, the sagittal plane and the cross section to obtain three projected plan views; and the first processing unit is used for respectively entering the three projected plan views and the first mode image into a corresponding projection discriminator and a 3D discriminator to discriminate authenticity so as to form a countermeasure with the generator.
Optionally, in the multi-modal image synthesizing apparatus provided in the embodiments of the present application, the loss function of the generator includes at least one of: an contrast loss function, a projection loss function, a loop consistency loss.
Optionally, in the multi-mode image synthesizing device provided in the embodiment of the present application, the first mode image is a magnetic resonance imaging image, and the second mode image is a computed tomography image.
The apparatus for synthesizing a multimodal image includes a processor and a memory, and the first conversion unit 501, the first determination unit 502, and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.
The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. The kernel can be provided with one or more, and the synthesis of the multi-mode image is performed by adjusting kernel parameters.
The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip.
The embodiment of the invention provides a storage medium, on which a program is stored, which when executed by a processor, implements a method of synthesizing a multi-modal image.
The embodiment of the invention provides a processor, which is used for running a program, wherein the program runs to execute a multi-mode image synthesis method.
The embodiment of the invention provides equipment, which comprises a processor, a memory and a program stored in the memory and capable of running on the processor, wherein the processor realizes the following steps when executing the program: converting the first modality image into a second modality image by a generator; dose calculations are performed from the second modality images to generate a radiation therapy plan.
The processor also realizes the following steps when executing the program: inputting an ordered image sequence of the first modality image into a shared downsampling module and a residual group module to obtain residual characteristics; performing copying and sequence adjusting operations on the residual characteristics to obtain the characteristics with the sequence adjusted; inputting the features with the adjusted sequence into a Conv-LSTM network module so that the generator fully learns different spatial information of different sequence image sequences; performing a merging operation on the characteristics of Conv-LSTM output to obtain merged characteristics; the combined features are input to a shared upsampling module to obtain a second modality image.
The processor also realizes the following steps when executing the program: respectively projecting the second mode image on a coronal plane, a sagittal plane and a cross section to obtain three projected plane diagrams; and respectively entering the three projected plan views and the first mode image into a corresponding projection discriminator and a 3D discriminator to discriminate authenticity so as to form a countermeasure with the generator.
The processor also realizes the following steps when executing the program: the loss function of the generator includes at least one of: an contrast loss function, a projection loss function, a loop consistency loss.
The processor also realizes the following steps when executing the program: the first modality image is a magnetic resonance imaging image and the second modality image is a computed tomography image.
The present application also provides a computer program product adapted to perform, when executed on a data processing device, a program initialized with the method steps of: converting the first modality image into a second modality image by a generator; dose calculations are performed from the second modality images to generate a radiation therapy plan.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: inputting an ordered image sequence of the first modality image into a shared downsampling module and a residual group module to obtain residual characteristics; performing copying and sequence adjusting operations on the residual characteristics to obtain the characteristics with the sequence adjusted; inputting the features with the adjusted sequence into a Conv-LSTM network module so that the generator fully learns different spatial information of different sequence image sequences; performing a merging operation on the characteristics of Conv-LSTM output to obtain merged characteristics; the combined features are input to a shared upsampling module to obtain a second modality image.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: respectively projecting the second mode image on a coronal plane, a sagittal plane and a cross section to obtain three projected plane diagrams; and respectively entering the three projected plan views and the first mode image into a corresponding projection discriminator and a 3D discriminator to discriminate authenticity so as to form a countermeasure with the generator.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: the loss function of the generator includes at least one of: an contrast loss function, a projection loss function, a loop consistency loss.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: the first modality image is a magnetic resonance imaging image and the second modality image is a computed tomography image.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.
Claims (8)
1. A method of synthesizing a multi-modal image, comprising:
converting the first modality image into a second modality image by a generator;
performing dose calculations from the second modality image to generate a radiation therapy plan;
the second mode image is projected on a coronal plane, a sagittal plane and a cross section respectively, and three projected plane diagrams are obtained;
respectively entering the three projected plan views and the first mode image into a corresponding projection discriminator and a 3D discriminator to discriminate authenticity so as to form countermeasure with the generator;
wherein the step of respectively entering the three projected plan views and the first mode image into a corresponding projection discriminator and 3D discriminator to discriminate authenticity, so as to form countermeasure with the generator comprises the following steps:
input image sequenceFirst through generator G x2y Obtaining a generated target image sequence->Then through the projection P of the axial plane ax Coronal plane projection P co And sagittal plane projection P sa Projection of the resulting image is obtained> Andfinally by three projection discriminators +.>To distinguish the projection of the generated image from the projection of the real image +.>And->And thereby with G x2y Again, a challenge is formed.
2. The method of claim 1, wherein converting the first modality image to the second modality image by the generator comprises:
inputting the ordered image sequence of the first mode image into a shared downsampling module and a residual group module to obtain residual characteristics;
performing copying and sequence adjusting operations on the residual characteristics to obtain the characteristics with the sequence adjusted;
inputting the features with the adjusted sequence into a Conv-LSTM network module so that the generator fully learns different spatial information of different sequence image sequences;
executing a merging operation on the characteristics output by the Conv-LSTM to obtain merged characteristics;
and inputting the combined features into a shared up-sampling module to obtain the second mode image.
3. The method of claim 1, wherein the loss function of the generator comprises at least one of: an antagonism loss function, a projection loss function, and a cyclic consistency loss; wherein the projection loss function is
4. The method of claim 1, wherein the first modality image is a magnetic resonance imaging image and the second modality image is a computed tomography image.
5. A multi-modal image compositing apparatus, comprising:
the first conversion unit is used for converting the first mode image into a second mode image through the generator;
a first determination unit for performing dose calculations from the second modality images to generate a radiotherapy plan;
wherein the apparatus further comprises:
the first acquisition unit is used for respectively projecting the second modal image on the coronal plane, the sagittal plane and the cross section to obtain three projected plan views;
the first processing unit is used for respectively enabling the three projected plan views and the first modal image to enter a corresponding projection discriminator and a 3D discriminator to discriminate authenticity so as to form countermeasure with the generator;
the first processing unit is also used for inputting a sequence of imagesFirst through generator G x2y Obtaining a generated target image sequenceThen through the projection P of the axial plane ax Coronal plane projection P co And sagittal plane projection P sa Obtaining projections of the generated imageAnd->Finally by three projection discriminators +.>To distinguish the projection of the generated image from the projection of the real image +.>And->And thereby with G x2y Again, a challenge is formed.
6. The apparatus of claim 5, wherein the first conversion unit comprises:
the first acquisition module is used for inputting the ordered image sequence of the first mode image into the shared downsampling module and the residual error group module so as to obtain residual error characteristics;
the second acquisition module is used for executing copying and sequence adjusting operations on the residual error characteristics to obtain the characteristics with the adjusted sequence;
the first input module is used for inputting the characteristics with the adjusted sequence into the Conv-LSTM network module so that the generator fully learns different spatial information of different sequence image sequences;
the first merging module is used for executing merging operation on the characteristics output by the Conv-LSTM to obtain merged characteristics;
and the second input module is used for inputting the combined characteristics into the shared up-sampling module so as to obtain the second mode image.
7. A processor for running a program, wherein the program when run performs the method of any one of claims 1 to 4.
8. A storage medium comprising a stored program, wherein the program performs the method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110387317.5A CN112862738B (en) | 2021-04-09 | 2021-04-09 | Method and device for synthesizing multi-mode image, storage medium and processor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110387317.5A CN112862738B (en) | 2021-04-09 | 2021-04-09 | Method and device for synthesizing multi-mode image, storage medium and processor |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112862738A CN112862738A (en) | 2021-05-28 |
| CN112862738B true CN112862738B (en) | 2024-01-16 |
Family
ID=75992393
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110387317.5A Active CN112862738B (en) | 2021-04-09 | 2021-04-09 | Method and device for synthesizing multi-mode image, storage medium and processor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112862738B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR3139225A1 (en) | 2022-08-23 | 2024-03-01 | Compagnie Generale Des Etablissements Michelin | Image synthesis using physical simulation and deep learning methods for early learning of a tire surface evaluation model |
| CN116883247B (en) * | 2023-09-06 | 2023-11-21 | 感跃医疗科技(成都)有限公司 | Unpaired CBCT image super-resolution generation algorithm based on Cycle-GAN |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108351395A (en) * | 2015-10-27 | 2018-07-31 | 皇家飞利浦有限公司 | Virtual CT images from magnetic resonance image |
| CN110930421A (en) * | 2019-11-22 | 2020-03-27 | 电子科技大学 | Segmentation method for CBCT (Cone Beam computed tomography) tooth image |
| CN111436936A (en) * | 2020-04-29 | 2020-07-24 | 浙江大学 | CT image reconstruction method based on MRI |
| CN111789612A (en) * | 2020-08-10 | 2020-10-20 | 上海联影医疗科技有限公司 | Imaging system and method |
| CN112150569A (en) * | 2020-09-24 | 2020-12-29 | 北京羽医甘蓝信息技术有限公司 | Method and device for generating CBCT image into CT image and terminal equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11756160B2 (en) * | 2018-07-27 | 2023-09-12 | Washington University | ML-based methods for pseudo-CT and HR MR image estimation |
-
2021
- 2021-04-09 CN CN202110387317.5A patent/CN112862738B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108351395A (en) * | 2015-10-27 | 2018-07-31 | 皇家飞利浦有限公司 | Virtual CT images from magnetic resonance image |
| CN110930421A (en) * | 2019-11-22 | 2020-03-27 | 电子科技大学 | Segmentation method for CBCT (Cone Beam computed tomography) tooth image |
| CN111436936A (en) * | 2020-04-29 | 2020-07-24 | 浙江大学 | CT image reconstruction method based on MRI |
| CN111789612A (en) * | 2020-08-10 | 2020-10-20 | 上海联影医疗科技有限公司 | Imaging system and method |
| CN112150569A (en) * | 2020-09-24 | 2020-12-29 | 北京羽医甘蓝信息技术有限公司 | Method and device for generating CBCT image into CT image and terminal equipment |
Non-Patent Citations (1)
| Title |
|---|
| Z Xu et al.Satellite Image Prediction Relying on GAN and LSTM Neural Networks.《ICC 2019-2019 IEEE International Conference on Communications》.2019,第1-6页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112862738A (en) | 2021-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ouyang et al. | Ultra‐low‐dose PET reconstruction using generative adversarial network with feature matching and task‐specific perceptual loss | |
| Morris et al. | Cardiac substructure segmentation with deep learning for improved cardiac sparing | |
| CN110047056B (en) | Cross-domain image analysis and composition with depth image to image network and countermeasure network | |
| RU2698997C1 (en) | Neural network for generating synthetic medical images | |
| US20190259153A1 (en) | Cross domain medical image segmentation | |
| US20180225847A1 (en) | Systems and methods for data and model-driven image reconstruction and enhancement | |
| AU2014212517B2 (en) | Method and apparatus for generating a derived image using images of different types | |
| CN112862738B (en) | Method and device for synthesizing multi-mode image, storage medium and processor | |
| US20080317317A1 (en) | Method and Apparatus For Accelerated Elastic Registration of Multiple Scans of Internal Properties of a Body | |
| Liu et al. | Image synthesis-based multi-modal image registration framework by using deep fully convolutional networks | |
| Thomas et al. | Improving automated glioma segmentation in routine clinical use through artificial intelligence-based replacement of missing sequences with synthetic magnetic resonance imaging scans | |
| US11449210B2 (en) | Method for providing an image base on a reconstructed image group and an apparatus using the same | |
| CN111739614A (en) | Medical image enhancement | |
| CN109844815A (en) | The image procossing based on feature is carried out using from the characteristic image of different iterative extractions | |
| Chen et al. | Deep learning based unpaired image-to-image translation applications for medical physics: a systematic review | |
| KR102112706B1 (en) | Method for detecting nodule and apparatus using the same | |
| Xu et al. | Applying cross-modality data processing for infarction learning in medical internet of things | |
| CN114066798A (en) | Brain tumor nuclear magnetic resonance image data synthesis method based on deep learning | |
| Li et al. | DDNet: 3D densely connected convolutional networks with feature pyramids for nasopharyngeal carcinoma segmentation | |
| CN106462988B (en) | The reconstruct of multi-angle of view tomography | |
| CN113012204B (en) | Registration method, registration device, storage medium and processor for multi-mode image | |
| Olberg et al. | Ensemble learning and personalized training for the improvement of unsupervised deep learning‐based synthetic CT reconstruction | |
| Neumann et al. | Deep similarity learning using a Siamese ResNet trained on similarity labels from disparity maps of cerebral MRA MIP pairs | |
| Lei et al. | Deep learning-based 3D image generation using a single 2D projection image | |
| Larsson et al. | Does an ensemble of GANs lead to better performance when training segmentation networks with synthetic images? |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |