CN110559009A

CN110559009A - Method, system and medium for converting multi-modal low-dose CT into high-dose CT based on GAN

Info

Publication number: CN110559009A
Application number: CN201910832520.1A
Authority: CN
Inventors: 苏琬棋; 瞿毅力; 邓楚富; 王莹; 陈志广; 卢宇彤
Original assignee: National Sun Yat Sen University
Current assignee: Sun Yat Sen University; National Sun Yat Sen University
Priority date: 2019-09-04
Filing date: 2019-09-04
Publication date: 2019-12-13
Anticipated expiration: 2039-09-04
Also published as: CN110559009B

Abstract

The invention discloses a method, a system and a medium for converting multi-modal low-dose CT into high-dose CT based on GAN, which comprises the steps of inputting low-dose CT of any modality; carrying out two-dimensional discrete wavelet transform on the low-dose CT to obtain a plurality of decomposition results; and inputting the low-dose CT and a plurality of decomposition results thereof into a trained coder in the GAN network for coding, and decoding the coding results through a decoder in the GAN network to obtain a corresponding high-dose modal image. Based on the wide development of GAN in multi-domain conversion and the decomposition capability of the traditional wavelet transformation, the invention inputs the low-dose CT and the wavelet transformation result thereof into the encoder in the trained GAN network together for encoding, and then decodes the encoding result through the decoder in the GAN network to obtain the corresponding high-dose modal image, thereby conveniently realizing the conversion of the low-dose CT image of any modality to generate the high-dose CT image.

Description

Method, system and medium for converting multi-modal low-dose CT into high-dose CT based on GAN

Technical Field

The invention relates to the field of medical image processing, in particular to a method, a system and a medium for converting multi-modal low-dose CT (computed tomography) into high-dose CT based on GAN (generation countermeasure network), which are used for generating the high-dose CT by generating the countermeasure network conversion according to the low-dose CT of any modality.

Background

As one of the modern mainstream medical images, a detection method of Computed Tomography (CT) has been widely used for clinical diagnosis in various clinical fields. With the popularization and development of CT scanning, more and more people are concerned about the possible radiation hazard of CT scanning to human body. CT scans are generally accompanied by a higher degree of x-ray radiation, and medical studies have shown that exposure to excessive x-ray radiation may induce metabolic abnormalities or cancer, leukemia or other genetic disorders. Researchers wish to reduce x-ray doses to reduce patient risk. The most common method of reducing the radiation dose is to reduce the x-ray flux by reducing the operating current and shortening the exposure time of the x-ray tube. Generally, the weaker the x-ray flux, the more CT noise reconstructed, which can degrade the signal-to-noise ratio and affect diagnostic performance. Therefore, obtaining high quality images that can be used for clinical diagnosis while reducing the dose has become an important direction of research in the CT field in recent years. In order to solve the inherent physical problem, many methods have been designed in the past to improve the image quality of Low-Dose CT (Low-Dose CT, LDCT), and the conventional methods include model-based iterative reconstruction, filtering before reconstruction, and post-reconstruction image processing, the pre-reconstruction processing technique is scanner-specific, and the data of a commercial scanner is not easily provided to researchers, and the post-reconstruction processing technique cannot accurately determine the noise distribution in the image domain, so that the algorithm cannot obtain the best tradeoff between structure preservation and noise reduction.

In recent years, with the development of deep learning in the field of image processing, research on solving the LDCT problem by using a deep learning technique is started, and generation of a countermeasure network is more widely applied to image conversion and image generation. Generating a countermeasure network (GAN) is a flexible deep neural network that can be trained unsupervised as well as supervised. The generation of a countermeasure network generally includes a generator that can generate realistic images by accepting random input and a discriminator that distinguishes between real images and generated images by learning them and thereby directs the generator to generate more realistic images. In addition, some researches combine the wavelet transform and the deep learning technique, which are the common image denoising methods in the traditional digital image processing, into LDCT processing. In some researches, multi-scale decomposition is performed on the CT by using wavelet transformation to obtain scale information and direction information of an image, then a decomposition result is denoised by using a Convolutional Neural Network (CNN), and finally the denoised decomposition result is reconstructed by using wavelet transformation to obtain the denoised CT. In summary, the current LDCT problem research basically involves two-domain conversion from a fixed low dose level to a high dose level, but how to implement multi-domain conversion from multiple low dose levels to high dose levels is still a key technical problem to be solved.

Disclosure of Invention

The technical problems to be solved by the invention are as follows: based on the wide development of GAN in multi-domain conversion and the decomposition capability of traditional wavelet transformation, the invention inputs the low-dose CT and the wavelet transformation result into the encoder in the trained GAN network together for encoding, and then decodes the encoding result through the decoder in the GAN network to obtain the corresponding high-dose modal image, so that the conversion of the low-dose CT image of any modality can be conveniently realized to generate the high-dose CT image.

In order to solve the technical problems, the invention adopts the technical scheme that:

A method for converting multi-modal low-dose CT into high-dose CT based on GAN comprises the following implementation steps:

1) Inputting low-dose CT of any modality;

2) Carrying out two-dimensional discrete wavelet transform on the low-dose CT to obtain a plurality of decomposition results;

3) And inputting the low-dose CT and a plurality of decomposition results thereof into a trained coder in the GAN network for coding, and decoding the coding results through a decoder in the GAN network to obtain a corresponding high-dose modal image.

Optionally, the step 2) of performing two-dimensional discrete wavelet transform on the low-dose CT to obtain a plurality of decomposition results specifically means obtaining an approximate matrix W_i，1Horizontal matrix W_i，1Hvertical matrix W_i，1Vand diagonal matrix W_i，1Dfour results.

Optionally, the GAN network includes an Encoder for encoding images with different dose levels to obtain the same feature space, a Decoder for decoding the encoding result to obtain a converted high dose map, a Discriminator for discriminating whether an image is a real high dose map, and a tag Discriminator for discriminating a reconstructed tag and an original tag_labelAnd a Task processor Task for processing the input image to obtain a Task label.

Optionally, step 3) is preceded by a step of training the GAN network by using a supervised method, and the detailed steps include:

A1) inputting a high dose map h with task labels and label thereof;

A2) Training the Task processor Task to obtain a Task processor Task which completes training; adding i Poisson noises with different levels to the high-dose modal image h to obtain a registered low-dose modal image l with i dose levels_i；

A3) Randomly selecting one mode from the i low-dose modes to carry out a low-dose mode image l_iPerforming conversion training with a high-dose mode image h, and repeating the training guided by a trained Task processor Task to obtain a trained Encoder Encoder and a trained Decoder Decode;

A4) input of test Low dose CT data l_testConverting the test data by using the trained module to obtain converted high-dose CT data, and processing the converted data by using a task processor to obtain a task processing result of the converted data; low dose CT data l_testProcessing the test data by using a task processor to obtain a task processing result of the test data;

A5) Comparing the task processing result of the conversion data with the task processing result of the test data, evaluating whether the conversion result is good, and if the conversion result is good, finishing the training; otherwise, skipping to execute the step A1) and continuing training.

Optionally, the step a2) of training the Task processor Task includes: carrying out first-level decomposition on the high-dose chart h by using two-dimensional discrete wavelet transform to obtain an approximate matrix W_h，1Horizontal matrix W_h，1HVertical matrix W_h，1Vand diagonal matrix W_h，1DHigh dose maps h and W_h，1、W_h，1H、W_h，1V、W_h，1DInputting the data into a Task processor Task together, and processing the data to obtain a reconstructed Task label_rUsing a tag Discriminator_labelFor the Label label and the reconstruction Label label of the high dose map h_rCarrying out discrimination learning, wherein the learning discriminates the former as true and the latter as false;

Low dose modality image l in step a3)_iThe detailed steps of the conversion training with the high dose modality image h include:

A3.1) input of Low dose modality image l_i；

A3.2) for Low dose modality images l_iTwo-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_i，1Horizontal matrix W_i，1HVertical matrix W_i，1VAnd diagonal matrix W_i，1DFour results;

A3.3) mapping of Low dose modality images l_iAnd its approximate matrix W_i，1Horizontal matrix W_i，1HVertical matrix W_i，1VAnd diagonal matrix W_i，1Dinputting the four results into an Encoder Encoder in the GAN network for encoding to obtain an encoding result Code_iThen Code the coding result_iDecoding the coding result by a Decoder in the GAN network to obtain a corresponding converted high-dose modal image h_i，t；

a3.4) converting the Discriminator into a high dose mode image h with the high dose mode image h as a positive sample_i，tPerforming true and false discrimination learning on the negative sample;

A3.5) conversion of high dose modality image h_i，tTwo-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_it，1Horizontal matrix W_it，1HVertical matrix W_it，1VAnd diagonal matrix W_it，1D(ii) a Carrying out two-dimensional discrete wavelet transform on the high-dose modal image h to obtain an approximate matrix W_h，1Horizontal matrix W_h，1HVertical matrix W_h，1VAnd diagonal matrix W_h，1D；

A3.6) conversion of high-dose images h_i，tOf the approximation matrix W_it，1horizontal matrix W_it，1HVertical matrix W_it，1VAnd diagonal matrix W_it，1DApproximate matrix W of high dose modality image h_h，1Horizontal matrix W_h，1HVertical matrix W_h，1VAnd diagonal matrix W_h，1DSolving wavelet self-supervision loss of an approximate matrix, a horizontal matrix, a vertical matrix and a diagonal matrix in a layered mode;

A3.7) approximation matrix W obtained in step A3)_i，1Two-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_i，11For the approximate matrix W obtained in step A5)_it，1Two-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_it，11Second order approximation matrix W for low dose maps and transition maps_i，11And W_it，11Solving the self-supervision loss;

a3.8) conversion of the high dose modality image h obtained in step A3)_i，tAnd its approximate matrix W_it，1horizontal matrix W_it，1HVertical matrix W_it，1Vand diagonal matrix W_it，1DThe Task processor Task obtains the label of the conversion icon_itLabel for finding high dose map and label for conversion icon_itTask tag self-supervision loss of (1);

A3.9) Code results for multiple Low dose modalities_iAnd Code_j，j≠iSolving semantic consistency loss; and solving wavelet consistency loss by layers according to the two-dimensional discrete wavelet transformation results of a plurality of low-dose mode conversion maps.

Optionally, step 3) is preceded by a step of training a GAN network based on the unlabeled low-dose high-dose CT registered dataset a and the task-labeled low-dose CT dataset B, and the detailed steps include:

B1) inputting task-tagged Low dose Modal images l_Band label, executing the training process of the supervised Task processor to obtain the trained Task processor Task, inputting the low-dose mode image l of low-dose high-dose registration_Ahigh dose modal image h_AExecuting a supervised low-dose CT to high-dose CT training process to obtain a trained Encoder Encoder, Decoder Decoder and Discriminator, wherein the low-dose mode image l_AAnd high dose modality image h_AFrom data set A, Low dose modality image l_BAnd label from dataset B;

B2) Recombining the trained Encoder Encoder, Decoder Decoder, Discriminator and Task processor Task to perform unsupervised combination training to obtain the trained Encoder Encoder, Decoder Decoder and Task processor Task;

B3) Input test Low dose CT data l_testconverting by using a trained Encoder Encoder, a Decoder Decode and a Task processor Task to obtain converted high-dose CT data, and processing the converted data by using the Task processor Task to obtain a Task processing result of the converted data; at the same time, test low dose CT data l_testProcessing the test data by using a Task processor Task to obtain a Task processing result of the test data;

B4) And comparing the task processing result of the conversion data with the task processing result of the test data to evaluate whether the conversion data is good, ending the training and exiting if the conversion data is good, and jumping to execute the step B1) if the conversion data is not good.

Optionally, the detailed step of performing the supervised Task processor training procedure in step B1) to obtain the trained Task processor Task includes: low dose modality image l of dataset B with two-dimensional discrete wavelet transform_BPerforming first-order decomposition to obtain an approximate matrix W_B，1Horizontal matrix W_B，1Hvertical matrix W_B，1VAnd diagonal matrix W_B，1Dcombining the low dose modality image of data set B with W_B，1、W_B，1H、W_B，1V、W_B，1DInputting the data into a Task processor Task together, and processing the data to obtain a reconstructed Task label_rUsing a tag Discriminator_labelFor low dose modality images l_BAnd a reconstruction tag label_rCarrying out discrimination learning, wherein the learning discriminates the former as true and the latter as false;

The detailed steps of performing the supervised low-dose CT to high-dose CT training process in step B1) to obtain the trained Encoder, Decoder and Discriminator include:

B1.1) input Low dose modality image l of dataset A_AUsing two-dimensional discrete wavelet transform to low-dose mode image_APerforming first-order decomposition to obtain an approximate matrix W_A，1Horizontal matrix W_A，1Hvertical matrix W_A，1VAnd diagonal matrix W_A，1DFour results;

B1.2) low dose modality image l_AAnd approximation matrix W_A，1Horizontal matrix W_A，1HVertical matrix W_A，1VInputting the codes into an Encoder Encoder together for encoding to obtain a Code of an encoding result_A(ii) a Encoding the resulting Code with the Decoder_ADecoding to obtain a conversion chart h of the converted high-dose mode_A，t；

B1.3) imaging with Discriminator in high dose modality h_Aconvert graph h for positive samples_A，tPerforming true and false discrimination learning on the negative sample;

B1.4) conversion of the plot h by two-dimensional discrete wavelet transform_A，tPerforming first-order decomposition to obtain an approximate matrix W_At，1Horizontal matrix W_At，1Hvertical matrix W_At，1VAnd diagonal matrix W_At，1D(ii) a Two-dimensional discrete wavelet transform for high dose mode image h_APerforming first-order decomposition to obtain an approximate matrix W_h，1horizontal matrix W_h，1HVertical matrix W_h，1VAnd diagonal matrix W_h，1D；

B1.5) converting the graph h_A，tFirst order decomposition result of (a), high dose modality image h_AThe wavelet self-supervision loss of an approximate matrix, a horizontal matrix, a vertical matrix and a diagonal matrix is obtained by layering the first-level decomposition result;

B1.6) to the approximation matrix W_A，1Two-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_A，11To the approximate matrix W_At，1Two-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_At，11second order approximation matrix W for low dose maps and transition maps_A，11And W_At，11Solving the self-supervision loss;

The detailed steps of recombining the trained Encoder Encoder, Decoder Decoder, Discriminator and Task processor Task in the step B2) to obtain the trained Encoder Encoder, Decoder Decoder and Task processor Task through unsupervised combination training include:

B2.1) input of Low dose modality image l of dataset B_BUsing two-dimensional discrete wavelet transform pair l_BPerforming first-order decomposition to obtain an approximate matrix W_B，1Horizontal matrix W_B，1HVertical matrix W_B，1VAnd diagonal matrix W_B，1DFour results;

B2.2) low dose modality image l_Band approximation matrix W_B，1Horizontal matrix W_B，1HVertical matrix W_B，1VAnd diagonal matrix W_B，1DInputting the codes into an Encoder Encoder together for encoding to obtain a Code of an encoding result_B(ii) a Encoding the resulting Code with the Decoder_BDecoding to obtain a converted high dose modal image h_B，t；

B2.3) Discriminator with high dose modality image h of dataset A_Afor a positive sample, low dose map l of data set B_BAnd a transition diagram h_B，tPerforming true and false discrimination learning on the negative sample;

B2.4) conversion of the plot h by two-dimensional discrete wavelet transform_B，tPerforming first-order decomposition to obtain an approximate matrix W_Bt，1Horizontal matrix W_Bt，1HVertical matrix W_Bt，1VAnd diagonal matrix W_Bt，1D；

B2.5) to the approximation matrix W_B，1Performing two-dimensional separationObtaining approximate matrix W by scattered wavelet transform_B，11To the approximate matrix W_Bt，1Two-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_Bt，11For low dose modality images l_BAnd a transition diagram h_B，tSecond order approximation matrix W_B，11And W_Bt，11Solving the self-supervision loss;

B2.6) converting the graph h_B，tAn approximate matrix W obtained by first-order decomposition_Bt，1Horizontal matrix W_Bt，1HVertical matrix W_Bt，1VAnd diagonal matrix W_Bt，1DThe Task processor Task obtains the label of the conversion icon_BtFinding a low dose modality image l_BThe original label and the conversion icon label_BtTask tag consistency loss of (2);

B2.7) low dose modality image l_Band the approximate matrix W obtained by the first-order decomposition_B，1Horizontal matrix W_B，1HVertical matrix W_B，1VAnd diagonal matrix W_B，1DThe Task processor Task is input to obtain a reconstructed Task label_BrFinding a low dose modality image l_BThe original label and the reconstructed label_BrIs self-monitoring for loss.

In addition, the invention also provides a system for converting multi-modal low-dose CT into high-dose CT based on GAN, which comprises:

The input program unit is used for inputting low-dose CT of any modality;

The wavelet transformation program unit is used for carrying out two-dimensional discrete wavelet transformation on the low-dose CT to obtain a plurality of decomposition results;

And the conversion program unit is used for inputting the low-dose CT and a plurality of decomposition results thereof into a trained encoder in the GAN network for encoding, and then decoding the encoding results through a decoder in the GAN network to obtain a corresponding high-dose modal image.

The invention further provides a system for converting multi-modality GAN-based low-dose CT into high-dose CT, comprising a computer device programmed or configured to execute the steps of the method for converting multi-modality GAN-based low-dose CT into high-dose CT, or a computer program stored on a storage medium of the computer device and programmed or configured to execute the method for converting multi-modality GAN-based low-dose CT into high-dose CT.

The present invention further provides a computer readable storage medium having stored thereon a computer program programmed or configured to perform the method of GAN based multi-modality low-dose CT to high-dose CT.

Compared with the prior art, the invention has the following advantages: based on the wide development of GAN in multi-domain conversion and the decomposition capability of the traditional wavelet transformation, the invention inputs the low-dose CT and the wavelet transformation result thereof into the encoder in the trained GAN network together for encoding, and then decodes the encoding result through the decoder in the GAN network to obtain the corresponding high-dose modal image, thereby conveniently realizing the conversion of the low-dose CT image of any modality to generate the high-dose CT image.

Drawings

FIG. 1 is a schematic diagram of a basic process of an embodiment of the present invention.

Fig. 2 is a schematic diagram of a training flow of Task processor Task according to an embodiment of the present invention.

Fig. 3 is a diagram of a GAN network training architecture according to a first embodiment of the present invention.

Fig. 4 is a main flow chart of GAN network training according to a first embodiment of the present invention.

FIG. 5 is a diagram illustrating a training process of Task processor Task according to a second embodiment of the present invention.

Fig. 6 is a diagram of a training architecture for registering low-dose CT to high-dose CT according to a second embodiment of the present invention.

Fig. 7 is a diagram of a combined training architecture according to a second embodiment of the present invention.

fig. 8 is a block diagram of a module reuse architecture according to a second embodiment of the present invention.

Fig. 9 is a main flow chart of GAN network training according to a first embodiment of the present invention.

Detailed Description

the first embodiment is as follows:

As shown in fig. 1, the implementation steps of the method for converting GAN-based multi-modal low-dose CT into high-dose CT in the present embodiment include:

1) Input of low dose CT (denoted as l in the figure) of arbitrary modality_i)；

2) Performing two-dimensional discrete Wavelet transform (represented as Wavelet) on the low-dose CT to obtain a plurality of decomposition results;

3) Inputting the low-dose CT and its multiple decomposition results into the Encoder (EC) in the trained GAN network for encoding (the result is code in the figure)_i) Then (code the coding result)_i) Decoding the encoded result by a decoder (denoted as DC in the figure) in the GAN network to obtain a corresponding high-dose mode image (denoted as h in the figure)_i，t)。

As shown in fig. 1, the step 2) of performing two-dimensional discrete wavelet transform on the low-dose CT to obtain a plurality of decomposition results specifically means obtaining an approximation matrix W_i，1Horizontal matrix W_i，1Hvertical matrix W_i，1VAnd diagonal matrix W_i，1DFour results.

In this embodiment, the GAN network includes an Encoder (abbreviated as EC in the figure) for encoding images with different dose levels to obtain the same feature space, a Decoder (abbreviated as DC in the figure) for decoding the encoding result to obtain a converted high dose map, a Discriminator (abbreviated as D in the figure) for discriminating whether the image is a real high dose map, and a tag Discriminator for discriminating whether a reconstructed tag and an original tag are true or false_label(abbreviated as D in the figure)_label) And a Task processor Task for processing the input image to obtain a Task label.

For the encoder and decoder, the present embodiment employs a smoothing filter for initialization. Smoothing filters are commonly used in conventional digital image processing for preprocessing tasks, where some trivial details in an image are removed before object extraction, pixels are averaged, and the average of pixels in the neighborhood is determined by a filter template to replace the value of each pixel in the image. The embodiment applies the method to the initialization of the encoder and the decoder, and can carry out the blurring processing and the noise reduction on the image while retaining the image information, for example, on a3 × 3 imageSmoothing filter with coefficients of all 1, z_iis the gray value of the corresponding position i of the image covered by the filter, R is the average value of the pixel gray levels in the 3 × 3 neighborhood, and the formula is as follows:

in the embodiment, a supervised learning method is adopted for the GAN network, the training data is high-dose CT data with task labels, a multi-mode low-dose and high-dose registration data set is constructed by adding noise, and low-dose CT images of any mode can be received and further converted to generate high-quality high-dose CT images. Meanwhile, the task processing can be further carried out on the converted high-dose CT image according to the task to which the data set belongs, and the effectiveness of the converted high-dose CT image is verified.

In this embodiment, a training step of Task processor Task and a training step of generating multi-modal transformation of high-dose CT by multi-modal low-dose CT transformation are further included before step 3), where:

In order to verify that the original semantic information of the image cannot be changed after the low-dose map is converted into the high-dose map, the embodiment performs task verification on the high-dose map obtained through conversion so as to ensure that the conversion training can achieve denoising without losing the task information of the image. The training step of the Task processor Task comprises the following steps: as shown in FIG. 2, the high dose map h is decomposed in one level by two-dimensional discrete wavelet transform to obtain an approximate matrix W_h，1Horizontal matrix W_h，1HVertical matrix W_h，1VAnd diagonal matrix W_h，1DHigh dose maps h and W_h，1、W_h，1H、W_h，1V、W_h，1Dinputting the data into a Task processor Task together, and processing the data to obtain a reconstructed Task label_rUsing a tag Discriminator_labelFor the Label label and the reconstruction Label label of the high dose map h_rCarrying out discrimination learning, wherein the learning discriminates the former as true and the latter as false; the trained Task processor Task is used for the conversion map validity verification in the low-dose mode image conversion and high-dose mode image training. The task of the inventionThe processor can directly process the mode diagram and the wavelet transformation result of the mode diagram, and does not process the coding result after the mode diagram is coded, so that the application range is wider.

as shown in fig. 4, before step 3), the present embodiment further includes a step of training a GAN network by using a supervised method, and the detailed steps include:

A1) Inputting a high dose map h with task labels and label thereof;

A3) Randomly selecting one mode from the i low-dose modes to carry out a low-dose mode image l_iPerforming conversion training with a high-dose mode image h, and using a trained Task processor Task to guide training to obtain a trained coder and a trained decoder;

As shown in fig. 3, step a2) the step of training the Task processor Task includes: carrying out first-level decomposition on the high-dose chart h by using two-dimensional discrete wavelet transform to obtain an approximate matrix W_h，1Horizontal matrix W_h，1Hvertical matrix W_h，1Vand diagonal matrix W_h，1DHigh dose maps h and W_h，1、W_h，1H、W_h，1V、W_h，1DInputting the data into a Task processor Task together, and processing the data to obtain a reconstructed Task label_rBy usinglabel Discriminator_labelFor the Label label and the reconstruction Label label of the high dose map h_rAnd performing discrimination learning, wherein the learning discriminates the former as true and the latter as false.

as shown in fig. 3, the low dose modality image/is performed in step a3)_iThe detailed steps of the conversion training with the high dose modality image h include:

a3.1) input of Low dose modality image l_i；

A3.6) conversion of high-dose images h_i，tof the approximation matrix W_it，1Horizontal matrix W_it，1HVertical matrix W_it，1VAnd diagonal matrix W_it，1DOf high dose modality images hApproximation matrix W_h，1Horizontal matrix W_h，1HVertical matrix W_h，1VAnd diagonal matrix W_h，1DSolving Wavelet self-supervision loss (Multi-level Wavelet loss) of an approximate matrix, a horizontal matrix, a vertical matrix and a diagonal matrix in a layered way;

A3.7) approximation matrix W obtained in step A3)_i，1Two-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_i，11For the approximate matrix W obtained in step A5)_it，1Two-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_it，11second order approximation matrix W for low dose maps and transition maps_i，11And W_it，11Finding the self-supervision loss (L2 loss);

A3.8) conversion of the high dose modality image h obtained in step A3)_i，tAnd its approximate matrix W_it，1Horizontal matrix W_it，1Hvertical matrix W_it，1VAnd diagonal matrix W_it，1DThe Task processor Task obtains the label of the conversion icon_itLabel for finding high dose map and label for conversion icon_itTask tag self-supervision loss of (L2 loss);

a3.9) Code results for multiple Low dose modalities_iAnd Code_j，j≠isolving semantic consistency loss; and (3) solving wavelet consistency loss (Multi-level wavelet loss) in a layered manner for the two-dimensional discrete wavelet transformation results of a plurality of low-dose modal transformation maps.

in this embodiment, the loss function in the GAN network is designed as follows:

1. The loss of the training part of the task processor is divided into a tag discriminator loss part and a generator loss part.

The tag discriminator loss is:

in the above formula, loss_{Discriminator，label}For tag Discriminator loss, Discriminator_label(label) is the discrimination result of the label Discriminator for discriminating the label, Discriminator_label(label_r) Identifying a label for a label identifier_rLabel is a label of the high dose chart h, label_rIs the reconstruction tag of the high dose map h.

the generator loss consists of a resistance loss and an unsupervised loss, which can be expressed as:

loss_{Generator，label}＝loss_{Adverserail，label}+loss_{supervision，label}

In the above formula, loss_{Generator，label}Loss of generator_{Adverserail，label}Loss of antagonism, loss_{supervision，label}For self-supervision of losses, Discriminotor_label(label_r) Identifying a label for a label identifier_rLabel is a label of the high dose chart h, label_rIs the reconstruction tag of the high dose map h.

2. The identifier loss of the training framework part is independently updated, and the specific loss is as follows:

In the above formula, loss_{Discriminator}discriminator (h) representing the Discriminator loss of the training architecture part_i，t) Discriminator discrimination conversion of high dose images h_i，tI denotes the mode i, and the Discriminator (h) is the discrimination result of the Discriminator for discriminating the high dose map h.

3. Other modules of the embodiment are updated and trained through an optimizer, and the loss items comprise discriminator guide loss provided by the discriminator, layered self-supervision loss of wavelets, layered consistency loss of wavelets, self-supervision loss of wavelet secondary approximation matrix, semantic consistency loss, task label supervision loss and task label consistency loss.

Discriminator guided loss:

In the above formula, Discriminotor (h)_i，t) Discriminator discrimination conversion of high dose images h_i，tI represents the modality i.

layered self-supervision loss of wavelets:

In the above equation, the matrix W is approximated_it，1Horizontal matrix W_it，1HVertical matrix W_it，1Vand diagonal matrix W_it，1DFor transforming high-dose images h_i，tOf the two-dimensional discrete wavelet transform result of (2), approximation matrix W_h，1Horizontal matrix W_h，1Hvertical matrix W_h，1VAnd diagonal matrix W_h，1DI represents the modality i as a result of the two-dimensional discrete wavelet transform of the high-dose modality image h.

Layered consistency loss of wavelets:

In the above equation, the matrix W is approximated_it，1horizontal matrix W_it，1HVertical matrix W_it，1VAnd diagonal matrix W_it，1DFor transforming high-dose images h_i，tOf the two-dimensional discrete wavelet transform result of (2), approximation matrix W_j，1horizontal matrix W_jt，1HVertical matrix W_jt，1Vand diagonal matrix W_jt，1DFor transforming high-dose images h_j，tI represents the mode i, j represents the mode j, and i is not equal to j.

Self-supervision loss of wavelet second-order approximation matrix:

In the above formula, W_i，1For low dose modality images l_iTwo-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_it，11To approximate a matrix W_it，1And performing two-dimensional discrete wavelet transform to obtain an approximate matrix.

Loss of semantic consistency:

In the above formula, Code_iTo form a low dose modality image l_iand its approximate matrix W_i，1Horizontal matrix W_i，1Hvertical matrix W_i，1VAnd diagonal matrix W_i，1DInputting the four results into an Encoder Encoder in the trained GAN network for encoding to obtain an encoding result, Code_jTo form a low dose modality image l_jAnd its approximate matrix W_j，1Horizontal matrix W_j，1Hvertical matrix W_j，1VAnd diagonal matrix W_j，1DAnd inputting the four results into an Encoder Encoder in the trained GAN network for encoding to obtain an encoding result, wherein i represents a mode i, j represents a mode j, and i is not equal to j.

Task tag supervision loss:

In the above formula, label is the label of the high dose chart h, label_itTo convert a high dose image h_i，tand its approximate matrix W_it，1Horizontal matrix W_it，1HVertical matrix W_it，1VAnd diagonal matrix W_it，1DInputting a Task processor Task to obtain a Task label of the conversion diagram, wherein i represents a mode i;

Task tag consistency loss:

in the above formula, label_itTo convert a high dose image h_i，tAnd its approximate matrix W_it，1horizontal matrix W_it，1HVertical matrix W_it，1VAnd diagonal matrix W_it，1DThe Task processor Task is input to obtain the Task label, label of the conversion chart_jtTo convert a high dose image h_j，tAnd its approximate matrix W_jt，1Horizontal matrix W_jt，1HVertical matrix W_jt，1VAnd diagonal matrix W_jt，1DInputting a Task processor Task to obtain a Task label of the conversion diagram, wherein i represents a mode i, j represents a mode j, and i is not equal to j;

The total loss of each term generator consisting of the encoder-decoder is thus the sum of the above losses, i.e.:

loss_Generator＝loss_Adversarial+loss_{supervision，1}+loss_{consistency，1}+loss_{supervision，2}+loss_{code，consistency}+loss_label+loss_{label，consistency}

The above training process generates training of a high dose modality for complete multi-modal low dose transformation, where the encoder and the decoder are training target products of this embodiment. Unlike some previous LDCT (low dose helical CT) studies that only perform manual empirical physician visual effect quality evaluation on the high dose map converted from the low dose, the present embodiment performs task verification on the converted high dose map by using the task and label to which the data set belongs to ensure that the high dose map obtained by the conversion training is valid. In addition, the method of the present embodiment directly decodes to obtain the converted high dose mode image, and the previous research needs to perform fusion reconstruction on the wavelet image obtained by the recurrent neural network to obtain the converted high dose map, so the image decomposition method of the present embodiment does not require that the decomposition process is reversible.

In addition, the present embodiment further provides a system for converting multi-modality low-dose CT into high-dose CT based on GAN, including:

The input program unit is used for inputting low-dose CT of any modality;

in addition, the present embodiment further provides a system for converting a high-dose CT based on a GAN multi-modal low-dose CT, which includes a computer device programmed or configured to execute the steps of the method for converting a high-dose CT based on a GAN multi-modal low-dose CT according to the present embodiment, or a storage medium of the computer device having stored thereon a computer program programmed or configured to execute the method for converting a high-dose CT based on a GAN multi-modal low-dose CT according to the present embodiment.

Furthermore, the present embodiment also provides a computer readable storage medium, which stores thereon a computer program programmed or configured to execute the method for converting high-dose CT based on GAN multi-modality low-dose CT according to the present embodiment.

example two:

The present embodiment is substantially the same as the first embodiment, and the main differences are as follows: the GAN network is trained differently. The main reasons for this are: because the high-dose CT training data set with task labels required in the first embodiment is difficult to obtain, most of the existing public data sets are: unlabeled low-dose high-dose CT registered dataset (dataset a); task-labeled low dose CT dataset (dataset B). On the basis of continuing to use the modular method of the first embodiment, a supplementary scheme is designed in the first embodiment, a hybrid supervised learning method is adopted, supervised task processor training is carried out on a data set B, supervised low-dose CT to high-dose CT training is carried out on the data set A, then trained modules are combined, unsupervised training is carried out on a labeled data set B, and then high-quality high-dose CT images are generated through conversion.

As shown in fig. 9, this embodiment further includes, before step 3), a step of training a GAN network based on the unlabeled low-dose high-dose CT registered dataset a and the task-labeled low-dose CT dataset B, and the detailed steps include:

B1) Inputting task-tagged Low dose Modal images l_BAnd label, executing the training process of the supervised Task processor to obtain the trained Task processor Task, inputting the low-dose mode image l of low-dose high-dose registration_AHigh dose modal image h_AExecuting a supervised low-dose CT to high-dose CT training process to obtain a trained Encoder Encoder, Decoder Decoder and Discriminator, wherein the low-dose mode image l_AAnd high dose modality image h_AFrom data set A, Low dose modality image l_BAnd label is from dataset B;

B3) Inputting test low-dose CT data, converting by using a trained Encoder Encoder, a Decoder Decode and a Task processor Task to obtain converted high-dose CT data, and processing the converted data by using the Task processor Task to obtain a Task processing result of the converted data; meanwhile, the test low-dose CT data is processed by using a Task processor Task to obtain a Task processing result of the test data;

As shown in fig. 5, the detailed step of executing the supervised Task processor training process in step B1) to obtain the trained Task processor Task includes: performing first-level decomposition on the low-dose modal image of the data set B by using two-dimensional discrete wavelet transform to obtain an approximate matrix W_B，1Horizontal matrix W_B，1HVertical matrix W_B，1VAnd diagonal matrix W_B，1DModulo the low dose of data set BState image and W_B，1、W_B，1H、W_B，1V、W_B，1DInputting the data into a Task processor Task together, and processing the data to obtain a reconstructed Task label_rUsing a tag Discriminator_labelFor low dose modality images l_BAnd a reconstruction tag label_rAnd performing discrimination learning, wherein the learning discriminates the former as true and the latter as false.

As shown in fig. 6, the detailed steps of performing the supervised low-dose CT to high-dose CT training procedure in step B1) to obtain the trained Encoder, Decoder, Discriminator include:

B1.6) to the approximation matrix W_A，1Two-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_A，11To the approximate matrix W_At，1Two-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_At，11Second order approximation matrix W for low dose maps and transition maps_A，11And W_At，11And solving the self-supervision loss.

Supervised low-dose to high-dose CT conversion of GAN networks based on dataset a this part of the training primarily utilizes registered low-dose and high-dose data, training the encoder and decoder to convert the low-dose data to generate high-dose data.

As shown in fig. 7, in step B2), the detailed steps of recombining the trained Encoder, Decoder, Discriminator, and Task processor Task to perform unsupervised combination training to obtain the trained Encoder, Decoder, and Task processor Task in this embodiment include:

B2.5) to the approximation matrix W_B，1Two-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_B，11To the approximate matrix W_Bt，1Two-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_Bt，11second order approximation matrix W for low dose maps and transition maps_B，11And W_Bt，11Solving the self-supervision loss;

B2.6) converting the graph h_B，tAnd the approximate matrix W obtained by the first-order decomposition_Bt，1Horizontal matrix W_Bt，1HVertical matrix W_Bt，1VAnd diagonal matrix W_Bt，1DThe Task processor Task obtains the label of the conversion icon_BtFinding a low dose modality image l_BThe original label and the label of the conversion graph_BtLoss of task tag consistency;

the method is an unsupervised combined training process, a trained Encoder Encoder, a Decoder Decoder and a Task processor Task are combined, the three modules are continuously trained by combining data sets A and B, the low dose and the high dose do not need to be registered in the training process, the low dose and the high dose are derived from different data sets, and the low dose modal image of the data set B can be converted into the high dose modal image by a discriminator loss, a wavelet secondary approximate matrix self-supervision loss, a Task tag consistency loss and a Task tag self-supervision loss constraint Encoder and Decoder. The embodiment uses the Encoder Encoder trained in the previous two training,The Decoder, the Discriminator and the Task processor Task are combined, and the training of the modules is continued, and the low dose data l of the data set B is input_BConverting to obtain high-dose CT image h_B，tTrue high dose CT image h of dataset A_APositive sample, low dose CT image l as discriminator_BAnd transformed high dose CT image h_B，tAs a negative example of the discriminator, unsupervised counterlearning is performed, and in addition, l is paired with a Task processor Task_BAnd h_B，tPerforming label segmentation, learning the segmentation capability of the high-dose chart while keeping the segmentation capability of the Task processor Task on the low-dose chart, and segmenting a result and the low-dose mode image l_BThe label of (1) is compared, so that the Encoder Encode and the Decoder Decode can retain focus information while converting and denoising.

After the training is completed, the present embodiment performs a conversion test on the test data in the data set B, and converts the low-dose test data into high-dose data by using the trained Encoder and Decoder, as shown in fig. 8.

For the test data of the data set B, the embodiment uses the trained Task processor Task to segment the conversion results of the test data and the test data, compares the segmentation results of the test data and the test data, and checks whether the converted data retains the lesion information, thereby verifying whether the converted data is valid. In addition, a trained Task processor Task can be used for carrying out Task processing on the data set A to obtain a Task label with reference value. For the converted data of the second embodiment, the trained Task processor Task is used for validity verification in the second embodiment, the low-dose test data is firstly subjected to Task processing to obtain a processing result, then the test data is converted to obtain high-dose data, the converted high-dose data is subjected to Task processing to obtain a Task processing result of the converted data, finally the two processing results are subjected to similarity evaluation, if the similarity of the two processing results meets the expectation, the generated data is good, otherwise, the network structure of the module is adjusted to be retrained.

The loss function of this embodiment is designed as follows:

1. the design of the supervised task processor training partial loss function is exactly the same as the first embodiment.

2. The loss function of the supervised low-dose CT conversion high-dose CT training part can be divided into two parts, namely discriminator loss and generator loss, and the discriminator loss and the generator loss are updated independently.

2.1, discriminator loss:

In the above formula, Discriminotor (h)_A，t) Low dose modality image/in Discriminator discrimination dataset a_AIs converted into a graph h_A，tthe result of discrimination of (b), Discriminator (h)_A) High dose modality image h representing Discriminator discrimination dataset a_AThe result of the discrimination.

2.2, the generator loss item comprises guide loss provided by a discriminator and self-supervision loss of a wavelet two-level approximate matrix. The specific formula is as follows:

discriminator guided loss:

In the above formula, Discriminotor (h)_A，t) Low dose modality image/in Discriminator discrimination dataset a_AIs converted into a graph h_A，tThe result of the discrimination.

Layered self-supervision loss of wavelets:

in the above equation, the matrix W is approximated_h，1Horizontal matrix W_h，1HVertical matrix W_h，1VAnd diagonal matrix W_h，1DFor low dose modality images l of data set B using two-dimensional discrete wavelet transform_BThe result of the first-order decomposition is approximated by a matrix W_At，1horizontal matrix W_At，1HVertical matrix W_At，1VAnd diagonal matrix W_At，1DFor transforming a graph h by a two-dimensional discrete wavelet transform_A，tThe result obtained by performing the first order decomposition.

Self-supervision loss of wavelet second-order approximation matrix:

In the above formula, W_A，11To approximate a matrix W_A，1two-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_At，11to approximate a matrix W_At，1And performing two-dimensional discrete wavelet transform to obtain an approximate matrix.

the generator loss term is specifically the sum of the above three equations, which can be expressed as:

loss_{Generator，A}＝loss_{Adversarial，A}+loss_{supervision，A，1}+loss_{supervision，A，2}

3. The unsupervised combined training part can be divided into two parts of discriminator loss and generator loss, and the two parts are updated independently.

3.1, discriminator loss:

In the above formula, Discriminotor (h)_B，t) Low dose modality image l representing Discriminator discrimination dataset B_BConverting to obtain high-dose CT image h_B，tThe result of the discrimination. Discriminator (h)_B) Low dose modality image l representing data set B_BThe result of the discrimination. Discriminator (h)_A) Low dose modality image l representing Discriminator discrimination dataset a_Athe result of the discrimination.

3.2, discriminator loss:

The generator loss terms include discriminator-directed losses, wavelet second-order approximation matrix auto-supervision losses, task tag consistency losses, and task tag auto-supervision losses provided by the discriminator. The concrete formula is as follows:

Discriminator guided loss:

Wavelet second-order approximation matrix self-supervision loss:

Task tag consistency loss:

In the above equation, label is the low dose modality image l of the data set B_BOriginal label of (1), label_BtTo convert the graph h_B，tan approximate matrix W obtained by first-order decomposition_Bt，1horizontal matrix W_Bt，1Hvertical matrix W_Bt，1VAnd diagonal matrix W_Bt，1DInputting a Task processor Task to obtain a conversion chart label;

Task tag self-supervision loss:

In the above equation, label is the low dose modality image l of the data set B_BOriginal label of (1), label_Brto form a low dose modality image l_BAnd the approximate matrix W obtained by the first-order decomposition_B，1Horizontal matrix W_B，1HVertical matrix W_B，1VAnd diagonal matrix W_B，1Dinputting the Task processor Task to obtain a reconstructed Task label;

The generator loss is the sum of the above losses, and can be expressed as:

loss_{Generator，B}＝loss_{Adversarial，B}+loss_{supervision，A}+loss_{label，consistency}+loss_label

The input program unit is used for inputting low-dose CT of any modality;

The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments, and all technical solutions belonging to the idea of the present invention belong to the protection scope of the present invention. It should be noted that modifications and embellishments within the scope of the invention may occur to those skilled in the art without departing from the principle of the invention, and are considered to be within the scope of the invention.

Claims

1. A method for converting multi-modal low-dose CT to high-dose CT based on GAN is characterized by comprising the following implementation steps:

1) Inputting low-dose CT of any modality;

2. The method according to claim 1, wherein the step 2) of performing two-dimensional discrete wavelet transform on the low-dose CT to obtain a plurality of decomposition results specifically means obtaining an approximation matrix W_i，1Horizontal matrix W_i，1HVertical matrix W_i，1VAnd diagonal matrix W_i，1DFour results.

3. The method of claim 1, wherein the GAN network comprises an Encoder Encoder for encoding images with different dose levels to obtain the same feature space, a Decoder Decoder for decoding the encoded result to obtain the converted high dose map, a Discriminator for discriminating whether the images are true high dose maps, a tag Discriminator for discriminating between reconstructed tags and original tags_labelAnd a Task processor Task for processing the input image to obtain a Task label.

4. the method for transforming high-dose CT based on GAN multi-modal low-dose CT as claimed in claim 3, wherein step 3) is preceded by the step of training GAN network by supervised method, and the detailed steps comprise:

A1) Inputting a high dose map h with task labels and label thereof;

A4) input of test Low dose CT data l_testConverting the test data by using a trained module to obtain converted high-dose CT data, and processing the converted data by using a Task processor Task to obtain a Task processing result of the converted data; low dose CT data l_testProcessing the test data by using a Task processor Task to obtain a Task processing result of the test data;

5. The method of GAN based multi-modal low-dose CT to high-dose CT according to claim 4, wherein the step A2) of training the Task processor Task comprises: carrying out first-level decomposition on the high-dose chart h by using two-dimensional discrete wavelet transform to obtain an approximate matrix W_h，1horizontal matrix W_h，1HVertical matrix W_h，1VAnd diagonal matrix W_h，1DHigh dose maps h and W_h，1、W_h，1H、W_h，1V、W_h，1DInputting the data into a Task processor Task together, and processing the data to obtain a reconstructed Task label_rUsing a tag Discriminator_labelFor the Label label and the reconstruction Label label of the high dose map h_rto carry outLearning by discrimination, wherein the former is discriminated as true and the latter is discriminated as false;

A3.1) input of Low dose modality image l_i；

A3.6) conversion of high-dose images h_i，tOf the approximation matrix W_it，1horizontal matrix W_it，1HVertical matrix W_it，1VAnd diagonal matrix W_it，1DApproximate matrix W of high dose modality image h_h，1Horizontal matrix W_h，1HVertical matrix W_h，1VAnd diagonal matrix W_h，1DLayered approximationWavelet self-supervision loss of a similar matrix, a horizontal matrix, a vertical matrix and a diagonal matrix;

6. The method for GAN-based multi-modality low-dose CT conversion of high-dose CT according to claim 3, further comprising before step 3) the step of training a GAN network based on unlabeled low-dose high-dose CT registered dataset a, task-labeled low-dose CT dataset B, the detailed steps comprising:

B1) Inputting task-tagged Low dose Modal images l_BAnd label, executing the training process of the supervised Task processor to obtain the trained Task processor Task, inputting the low-dose mode image l of low-dose high-dose registration_AHigh dose modal image h_AExecuting a supervised low-dose CT to high-dose CT training process to obtain a trained Encoder Encoder, Decoder Decoder and Discriminator, wherein the low-dose mode image l_AAnd high dose modality image h_AFrom data set A, Low dose ModemImage l_Band label from dataset B;

7. The method of GAN based multi-modal low-dose CT to high-dose CT as claimed in claim 6, wherein the detailed step of performing the supervised Task processor training procedure in step B1) to obtain the trained Task processor Task comprises: low dose modality image l of dataset B with two-dimensional discrete wavelet transform_Bperforming first-order decomposition to obtain an approximate matrix W_B，1Horizontal matrix W_B，1Hvertical matrix W_B，1VAnd diagonal matrix W_B，1DA low dose modality image l of the data set B_BAnd W_B，1、W_B，1H、W_B，1V、W_B，1Dinputting the data into a Task processor Task together, and processing the data to obtain a reconstructed Task label_rUsing a tag Discriminator_labelFor low dose modality images l_BAnd a reconstruction tag label_rCarrying out discrimination learning, wherein the learning discriminates the former as true and the latter as false;

b2.5) to the approximation matrix W_B，1Two-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_B，11To the approximate matrix W_Bt，1Two-dimensional discrete wavelet transform is carried out to obtain an approximate matrix W_Bt，11For low dose modality images l_BAnd a transition diagram h_B，tSecond order approximation matrix W_B，11And W_Bt，11Solving the self-supervision loss;

B2.6) converting the graph h_B，tand the approximate matrix W obtained by the first-order decomposition_Bt，1Horizontal matrix W_Bt，1HVertical matrix W_Bt，1VAnd diagonal matrix W_Bt，1DThe Task processor Task obtains the label of the conversion icon_BtFinding a low dose modality image l_BThe original label and the conversion icon label_BtTask tag consistency loss of (2);

8. A system for converting multi-modality GAN-based low-dose CT to high-dose CT, comprising:

The input program unit is used for inputting low-dose CT of any modality;

9. A system for converting GAN-based multi-modal low-dose CT into high-dose CT, comprising a computer device, wherein the computer device is programmed or configured to perform the steps of the method for converting GAN-based multi-modal low-dose CT into high-dose CT of any one of claims 1-7, or wherein a storage medium of the computer device has stored thereon a computer program programmed or configured to perform the method for converting GAN-based multi-modal low-dose CT into high-dose CT of any one of claims 1-7.

10. a computer readable storage medium having stored thereon a computer program programmed or configured to perform the method of GAN based multi-modal low-dose CT to high-dose CT as claimed in any of claims 1-7.