CN118397059B - Model training method and registration method for multimodal image enhancement and registration - Google Patents

Abstract

The present invention provides a model training method and a registration method for multimodal image enhancement and registration, the training method comprising: obtaining an MR image sequence to be processed and a CT image sequence to be processed, obtaining M groups of matching image pairs according to the mutual information of the MR image to be processed and the CT image to be processed, and selecting several groups of matching image pairs as training sets; selecting two MR images to be processed and one CT image to be processed in the training set, inputting the selected two MR images to be processed, one CT image to be processed and a Gaussian noise image into a target image sequence enhancement model to be trained, so as to train the target image sequence enhancement model to be trained and obtain a trained target image sequence enhancement model. The trained target image sequence enhancement model obtained by the present invention can generate an MR image matching the CT image, which is conducive to improving the accuracy of three-dimensional reconstruction of the image.

Description

Model training method and registration method for multimodal image enhancement and registration

Technical Field

The present invention belongs to the technical field of image processing, and in particular relates to a model training method and a registration method for multimodal image enhancement and registration.

Background Art

Computer-assisted and medical imaging technology has long been an important part of bone tumor limb-salvage surgery, helping doctors complete surgical approach planning, personalized plan formulation, preoperative simulation, and other tasks, significantly improving the success rate of limb-salvage. Currently, images are generally analyzed by manually or semi-automatically marking tumors, which consumes a lot of surgeons' time and energy. Therefore, it is imperative to use the rapidly developed deep learning technology in recent years to conduct tumor segmentation research.

In this process, there is a contradiction between the high requirements of deep learning technology for the amount of training data and the limited number of medical images. Especially when using multimodal images for analysis, the mismatch of data between different images will lead to a further reduction in the amount of available data, thereby increasing the difficulty of subsequent operations and affecting the tumor segmentation effect. At the same time, image data of different modalities also need to be registered to provide effective information.

Therefore, how to synthesize reliable MR (Magnetic Resonance) images and correspond them in CT (Computed Tomography) image sequences has become a problem that needs to be solved urgently.

Summary of the invention

The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and to propose a model training method and a registration method for multimodal image enhancement and registration. The technical problem to be solved by the present invention is achieved through the following technical solutions:

The present invention provides a model training method for multimodal image enhancement and registration, the training method comprising:

Acquire an MR image sequence to be processed and a CT image sequence to be processed, wherein the MR image sequence to be processed includes M MR images to be processed arranged in sequence, and the CT image sequence to be processed includes N CT images to be processed arranged in sequence, and both M and N are integers greater than 0;

According to the mutual information between the MR image to be processed and the CT image to be processed, M groups of matching image pairs are obtained, and several groups of matching image pairs are selected from the M groups of matching image pairs as training sets, each group of matching image pairs including one MR image to be processed and one CT image to be processed that match each other;

Two MR images to be processed and one CT image to be processed are selected from the training set, and the selected two MR images to be processed, one CT image to be processed and a Gaussian noise image are input into the target image sequence enhancement model to be trained, so as to train the target image sequence enhancement model to be trained and obtain a trained target image sequence enhancement model, and the trained target image sequence enhancement model is used to match the CT image to be registered to obtain a registered MR image; wherein, the two MR images to be processed selected from the training set do not match the selected CT image to be processed; and the target image sequence enhancement model is a model based on a generative adversarial network.

In one embodiment of the present invention, obtaining a to-be-processed MR image sequence and a to-be-processed CT image sequence includes:

Acquire an initial MR image sequence and an initial CT image sequence, wherein the initial MR image sequence includes M initial MR images, and the initial CT image sequence includes N initial CT images;

Converting the brightness values of the initial MR image and the initial CT image into grayscale values to obtain an MR grayscale image and a CT grayscale image;

A histogram equalization algorithm is used to redistribute the grayscale values of the MR grayscale image and the CT grayscale image, respectively, so that the grayscale values of the MR grayscale image and the CT grayscale image are uniformly distributed within a preset range, thereby obtaining an MR uniformly distributed image and a CT uniformly distributed image;

The grayscale values of the MR uniform distribution image and the CT uniform distribution image are respectively normalized to the interval of [0, 255] to obtain the MR image to be processed and the CT image to be processed. All the MR images to be processed form the MR image sequence to be processed, and all the CT images to be processed form the CT image sequence to be processed.

In one embodiment of the present invention, obtaining M groups of matching image pairs according to the mutual information between the MR image to be processed and the CT image to be processed includes:

Acquire the first MR image to be processed in the MR image sequence to be processed;

Based on the mutual information between the first MR image to be processed and the CT image to be processed, selecting a CT image to be processed having the largest mutual information with the first MR image to be processed in the CT image sequence to be processed to form a first group of matching image pairs;

Based on the preset step size, matching CT images to be processed are selected from the second MR image to be processed to the Mth MR image to be processed in the CT image sequence to be processed, to obtain M-1 groups of matching image pairs.

In one embodiment of the present invention, based on a preset step size, matching CT images to be processed are selected from the second MR image to be processed to the Mth MR image to be processed in the CT image sequence to be processed, to obtain M-1 groups of matching image pairs, including:

For the m+1th MR image to be processed, determine whether n+λ is an integer. If it is an integer, select the n+λth CT image to be processed from the CT image sequence to be processed as the image to be matched. If it is a non-integer, select the CT image to be processed that has a larger mutual information with the m+1th MR image to be processed from the two adjacent CT images to be processed that are adjacent to the n+λth CT image to be processed as the image to be matched, wherein the mth MR image to be processed and the nth CT image to be processed form the mth group of matching image pairs, λ is a preset step size, λ=Sm/Sc, Sm is the inter-layer interval of the MR image sequence to be processed, Sc is the inter-layer interval of the CT image sequence to be processed, 1≤m≤M, 1≤n≤N;

Determine whether the mutual information between the CT image to be processed as the image to be matched and the m+1th MR image to be processed is greater than or equal to a preset threshold; if so, the CT image to be processed as the image to be matched and the m+1th MR image to be processed form the m+1th group of matching image pairs; if not, select the CT image to be processed with the largest mutual information with the m+1th MR image to be processed in the CT image sequence to be processed to form the m+1th group of matching image pairs.

In one embodiment of the present invention, the target image sequence enhancement model includes a generator and a discriminator, and the generator includes an encoder, a feature extraction unit and a decoder;

Selecting two MR images to be processed and one CT image to be processed from the training set, inputting the selected two MR images to be processed, one CT image to be processed and a Gaussian noise image into the target image sequence enhancement model to be trained, so as to train the target image sequence enhancement model to be trained, and obtaining a trained target image sequence enhancement model, including:

Selecting two MR images to be processed and one CT image to be processed from the training set, inputting the selected two MR images to be processed, one CT image to be processed and one Gaussian noise image into the encoder, and obtaining a first feature matrix corresponding to the Gaussian noise image, a second feature matrix and a third feature matrix respectively corresponding to the two MR images to be processed, and a fourth feature matrix corresponding to the CT image to be processed;

Inputting the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix into the feature extraction unit to obtain a feature matrix to be decoded;

Inputting the feature matrix to be decoded into the decoder to obtain a first MR enhanced image;

Based on the dynamic differential loss function constructed by the first MR enhanced image and the MR image to be processed to be registered, the parameters of the target image sequence enhancement model to be trained are adjusted in a back-propagation manner to obtain a preliminarily trained target image sequence enhancement model, and a second MR enhanced image output by the preliminarily trained target image sequence enhancement model is obtained; wherein the MR image to be processed to be registered is selected from the training set and is between the two MR images to be processed input to the target image sequence enhancement model to be trained, and the CT image to be processed input to the target image sequence enhancement model to be trained and the MR image to be processed to be registered are in the same group of matching image pairs;

The second MR enhanced image is input into the discriminator to obtain the trained target image sequence enhancement model according to the discrimination result of the discriminator.

In one embodiment of the present invention, the feature extraction unit includes a graph attention module, a channel attention module and a residual structure;

Inputting the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix into the feature extraction unit to obtain a feature matrix to be decoded, comprising:

The first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix are input into the graph attention module, and the graph attention module obtains a fifth feature matrix corresponding to the Gaussian noise image, a sixth feature matrix and a seventh feature matrix corresponding to the two MR images to be processed, and an eighth feature matrix corresponding to the CT image to be processed by capturing similarities between the features;

The fifth feature matrix, the sixth feature matrix, the seventh feature matrix and the eighth feature matrix are input into the channel attention module, and the channel attention module obtains the first weight of the fifth feature matrix, the second weight of the sixth feature matrix, the third weight of the seventh feature matrix, and the fourth weight of the eighth feature matrix through convolution operation, and respectively multiplies the first weight by the fifth feature matrix, the second weight by the sixth feature matrix, the third weight by the seventh feature matrix, and the fourth weight by the eighth feature matrix, and adds all the multiplication results to obtain a ninth feature matrix;

The ninth feature matrix is input into the residual structure, and after a convolution operation, the feature matrix to be decoded is obtained.

In one embodiment of the present invention, the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix are input into the graph attention module, and the graph attention module obtains a fifth feature matrix corresponding to the Gaussian noise image, a sixth feature matrix and a seventh feature matrix corresponding to the two MR images to be processed, and an eighth feature matrix corresponding to the CT image to be processed by capturing similarities between the features, including:

Step 3.211, inputting the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix into the graph attention module, and then dividing the first feature matrix into a plurality of first matrix blocks;

Step 3.212, sequentially select one of the first matrix blocks, and respectively select a second matrix block, a third matrix block and a fourth matrix block in the second characteristic matrix, the third characteristic matrix and the fourth characteristic matrix which have the same position as the first matrix block;

Step 3.213, combining the first matrix block, the second matrix block, the third matrix block and the fourth matrix block into a combined matrix;

Step 3.214, selecting a number of pixel points adjacent to the first matrix block to obtain a pixel matrix;

Step 3.215, combining the combination matrix and the pixel matrix into a splicing matrix;

Step 3.216: According to the connection relationship between the first matrix block, the second matrix block, the third matrix block and the fourth matrix block, weighted summation processing is performed on the first matrix block, the second matrix block, the third matrix block and the fourth matrix block in the spliced matrix to obtain a fifth matrix block, a sixth matrix block, a seventh matrix block and an eighth matrix block respectively;

Step 3.217, using the fifth matrix block, the sixth matrix block, the seventh matrix block and the eighth matrix block to correspondingly replace the first matrix block of the first characteristic matrix, the second matrix block of the second characteristic matrix, the third matrix block of the third characteristic matrix and the fourth matrix block of the fourth characteristic matrix;

Step 3.218, repeat steps 3.212 to 3.217 until all matrix blocks in the first characteristic matrix, the second characteristic matrix, the third characteristic matrix and the fourth characteristic matrix are replaced, and the fifth characteristic matrix, the sixth characteristic matrix, the seventh characteristic matrix and the eighth characteristic matrix are obtained accordingly.

In one embodiment of the present invention, the dynamic differential loss function is expressed as:

;

in, represents the dynamic difference loss function, represents the MR image to be registered. represents the first MR enhanced image, Indicates that the goal of the generator is to arrive The mapping of Indicates the first MR enhanced image Pixels to perform the desired operation, represents the probability distribution of the first MR enhanced image, represents the number of pixels in the MR image to be processed to be registered, Indicates that the generator will register the first The difference in the results after the pixels are mapped to the first MR enhanced image, Indicates all The average value of .

In one embodiment of the present invention, the second MR enhanced image is input to the discriminator to obtain the trained target image sequence enhancement model according to the discrimination result of the discriminator, including:

inputting the second MR enhanced image into the discriminator to obtain a discriminant value;

Determine the relationship between the discrimination value and the discrimination threshold; if the discrimination value is greater than the discrimination threshold and the discrimination result is false, then update the discrimination threshold according to the loss value obtained by the kurtosis loss function, and continue to train the preliminarily trained target image sequence enhancement model until the discrimination value is less than or equal to the latest discrimination threshold, and obtain the trained target image sequence enhancement model; if the discrimination value is less than or equal to the discrimination threshold and the discrimination result is true, then use the preliminarily trained target image sequence enhancement model as the trained target image sequence enhancement model.

The present invention also provides a multi-modal image registration method, comprising:

Acquire the CT image to be registered;

The CT image to be registered is input into the trained target image sequence enhancement model described in any one of the above embodiments to obtain a registered MR image.

Beneficial effects of the present invention:

The present invention first obtains the MR image sequence to be processed and the CT image sequence to be processed, and obtains M groups of matching image pairs according to the mutual information of the MR image to be processed and the CT image to be processed, and selects several groups of matching image pairs from the M groups of matching image pairs as training sets, and each group of matching image pairs includes a mutually matching MR image to be processed and a CT image to be processed, and then selects two MR images to be processed and a CT image to be processed in the training set, and the selected two MR images to be processed, one CT image to be processed and a Gaussian noise image are input into the target image sequence enhancement model to be trained, so as to train the target image sequence enhancement model to be trained, and obtain a trained target image sequence enhancement model, and the obtained trained target image sequence enhancement model can match the CT image to be registered, and obtain a registered MR image. Therefore, the trained target image sequence enhancement model obtained by the present invention can generate an MR image matching the CT image, which is conducive to improving the accuracy of image three-dimensional reconstruction and providing basic technical support for the field of medical image processing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a training method for a model for multimodal image enhancement and registration provided by an embodiment of the present invention;

FIG. 2 is a schematic diagram of a target image sequence enhancement model provided by an embodiment of the present invention.

DETAILED DESCRIPTION

The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

Embodiment 1

Please refer to FIG. 1 , which is a flow chart of a training method for a model for multimodal image enhancement and registration provided by an embodiment of the present invention. The embodiment of the present invention provides a training method for a model for multimodal image enhancement and registration, and the training method includes:

Step 1: Obtain an MR image sequence to be processed and a CT image sequence to be processed, wherein the MR image sequence to be processed includes M MR images to be processed arranged in sequence, and the CT image sequence to be processed includes N CT images to be processed arranged in sequence, and both M and N are integers greater than 0.

Here, the MR image sequence to be processed and the CT image sequence to be processed are both acquired from the same part of the same person. For example, M is 30 and N is 300.

Optionally, the M MR images to be processed and the N CT images to be processed are arranged in spatial order.

In an optional embodiment, step 1 may specifically include:

Step 1.1, obtaining an initial MR image sequence and an initial CT image sequence, wherein the initial MR image sequence includes M initial MR images, and the initial CT image sequence includes N initial CT images.

Here, the initial MR image sequence and the initial CT image sequence are both acquired from the same part of the same person, for example, from the thigh root to the middle of the calf.

Optionally, the M initial MR images and the N initial CT images are arranged in a spatial order from earliest to latest.

Step 1.2: Convert the brightness values of the initial MR image and the initial CT image into grayscale values to obtain an MR grayscale image and a CT grayscale image.

Step 1.3: Use a histogram equalization algorithm to redistribute the grayscale values of the MR grayscale image and the CT grayscale image, respectively, so that the grayscale values of the MR grayscale image and the CT grayscale image are evenly distributed within a preset range, thereby obtaining an MR evenly distributed image and a CT evenly distributed image.

Specifically, the histogram equalization algorithm is used to redistribute the grayscale values of the MR grayscale image and the CT grayscale image, that is, multiple intervals are evenly divided within the preset range, so that the number of pixels in each interval is evenly distributed, thereby obtaining an MR uniform distribution image and a CT uniform distribution image. For example, the preset range is divided into 5 intervals, and the number of pixels evenly distributed to each interval is 10. If 7 pixels in the MR grayscale image fall into the first interval, then 3 pixels with smaller grayscale values can be selected from the second interval in ascending order and placed in the first interval. For example, the grayscale values of the 3 pixels selected from the second interval are 51, 55, and 60, respectively. Then, the grayscale value of each selected pixel is multiplied by A1/A2, where A1 is the largest grayscale value in the first interval, and A2 is the largest grayscale value of the pixels selected from the second interval. For example, 51, 55, and 60 are all multiplied by 50/60, and the multiplication result is placed in the first interval, and so on for other cases.

Here, the preset range of the MR grayscale image is a range consisting of the minimum grayscale value and the maximum grayscale value of the MR grayscale image, and the preset range of the CT grayscale image is a range consisting of the minimum grayscale value and the maximum grayscale value of the CT grayscale image.

Step 1.4, respectively normalize the grayscale values of the MR uniform distribution image and the CT uniform distribution image to the interval [0, 255] to obtain the MR image to be processed and the CT image to be processed. All the MR images to be processed constitute the MR image sequence to be processed, and all the CT images to be processed constitute the CT image sequence to be processed.

In this embodiment, the normalization formula is:

;

in, is the normalized grayscale value, is the gray value before normalization, is the minimum gray value of MR uniform distribution image or CT uniform distribution image, It is the maximum gray value of MR uniform distribution image or CT uniform distribution image.

Step 2: According to the mutual information between the MR image to be processed and the CT image to be processed, M groups of matching image pairs are obtained, and several groups of matching image pairs are selected from the M groups of matching image pairs as training sets, each group of matching image pairs includes one MR image to be processed and one CT image to be processed that match each other.

Specifically, this embodiment performs matching processing on the MR image to be processed and the CT image to be processed through the mutual information between the MR image to be processed and the CT image to be processed, thereby obtaining matching image pairs consisting of M groups of mutually matching MR images to be processed and CT images to be processed, and then selects several groups of matching image pairs as training sets, and the remaining matching image pairs are used as test sets.

For example, 80% of the matching image pairs are randomly selected as the training set, and the remaining 20% are used as the test set.

In an optional embodiment, step 2 may specifically include:

Step 2.1, obtaining the first MR image to be processed in the MR image sequence to be processed.

Step 2.2: Based on the mutual information between the first MR image to be processed and the CT image to be processed, select the CT image to be processed with the largest mutual information with the first MR image to be processed in the CT image sequence to be processed to form a first group of matching image pairs.

In this embodiment, the calculation formula of the mutual information of two images is:

;

in, is the MR image to be processed, is the CT image to be processed, is the mutual information between the MR image to be processed and the CT image to be processed, is the marginal probability distribution of the MR image to be processed, , is the number of gray values a in the MR image to be processed, is the number of pixels in the MR image to be processed, is the marginal probability distribution of the CT image to be processed, , is the gray value of the CT image to be processed The number of is the number of pixels in the CT image to be processed, .

Step 2.3: Based on the preset step size, select matching CT images to be processed from the second MR image to be processed to the Mth MR image to be processed in the CT image sequence to be processed, and obtain M-1 groups of matching image pairs.

Step 2.31. For the m+1th MR image to be processed, determine whether n+λ is an integer. If it is an integer, select the n+λth CT image to be processed from the CT image sequence to be processed as the image to be matched. If it is a non-integer, select the two adjacent CT images to be processed that have a larger mutual information with the m+1th MR image to be processed as the image to be matched, where the mth MR image to be processed and the nth CT image to be processed constitute the mth group of matching image pairs, λ is the preset step size, λ=Sm/Sc, Sm is the inter-layer interval of the MR image sequence to be processed, Sc is the inter-layer interval of the CT image sequence to be processed, 1≤m≤M, 1≤n≤N.

Specifically, assuming that the mth MR image to be processed matches the nth CT image to be processed, forming the mth set of matching image pairs, then first determine whether n+λ is an integer. If n+λ is an integer, the n+λth CT image to be processed can be directly selected as the image to be matched for the m+1th MR image to be processed. If n+λ is not an integer, it is necessary to select the two CT images to be processed that are closest to n+λ. For example, if n+λ is 6.5, the sixth CT image to be processed and the seventh CT image to be processed are selected, and then the CT image to be processed with greater mutual information with the m+1th MR image to be processed is selected from the two selected CT images to be processed as the image to be matched. Here, the inter-layer interval is the distance between two adjacent images.

Step 2.32, determine whether the mutual information between the CT image to be processed as the image to be matched and the m+1th MR image to be processed is greater than or equal to a preset threshold; if so, the CT image to be processed as the image to be matched and the m+1th MR image to be processed are combined into the m+1th matching image pair; if not, the CT image to be processed with the largest mutual information with the m+1th MR image to be processed is selected from the CT image sequence to be processed to form the m+1th matching image pair.

In this embodiment, for other MR images to be processed, the search for matching CT images to be processed is continued in the manner of step 2.31 until all matching image pairs are obtained.

It should be noted that this embodiment does not limit the specific value of the preset threshold, and those skilled in the art can set it according to specific requirements. For example, the preset threshold is 1.05.

Step 3, select two MR images to be processed and one CT image to be processed in the training set, and input the selected two MR images to be processed, one CT image to be processed and a Gaussian noise image into the target image sequence enhancement model to be trained, so as to train the target image sequence enhancement model to be trained and obtain a trained target image sequence enhancement model, and the trained target image sequence enhancement model is used to match the CT image to be registered to obtain a registered MR image; wherein, the two MR images to be processed selected in the training set do not match the selected CT image to be processed; the target image sequence enhancement model is a model based on a generative adversarial network.

Specifically, firstly, three MR images to be processed and one CT image to be processed are selected in the training set, preferably three spatially continuous MR images to be processed, the three MR images to be processed are respectively recorded as MR1, MR2 and MR3, MR2 is spatially located between MR1 and MR3, the CT image to be processed selected in the training set and MR2 form a group of matching image pairs, and then a Gaussian noise image is randomly generated, thereby MR1, MR3, the CT image to be processed matching MR2 and the Gaussian noise image can be input into the target image sequence enhancement model to be trained, so as to train the target image sequence enhancement model to be trained and obtain the trained target image sequence enhancement model.

In this embodiment, the target image sequence enhancement model is a model based on a generative adversarial network, see Figure 2, the target image sequence enhancement model includes a generator and a discriminator, and the generator includes an encoder, a feature extraction unit and a decoder.

In an optional embodiment, step 3 may specifically include:

Step 3.1. Select two MR images to be processed and one CT image to be processed in the training set, input the selected two MR images to be processed, one CT image to be processed and one Gaussian noise image into the encoder, and obtain the first feature matrix corresponding to the Gaussian noise image, the second feature matrix corresponding to the two MR images to be processed, the third feature matrix and the fourth feature matrix corresponding to the CT image to be processed.

Optionally, for images that need to be input into the encoder, a circle of pixels with a gray value of 0 may be filled around these images before these images are input into the encoder.

Optionally, the encoder has multiple layers of image feature extraction layers connected in sequence, and the structures of all image feature extraction layers are the same. The image feature extraction layers include convolution layers, pooling layers, normalization layers, and activation layers connected in sequence. Therefore, after the Gaussian noise image is processed by the multiple layers of image feature extraction layers, a first feature matrix is obtained. After the two MR images to be processed are processed by the multiple layers of image feature extraction layers, a second feature matrix and a third feature matrix are obtained respectively. After the CT images to be processed are processed by the multiple layers of image feature extraction layers, a fourth feature matrix is obtained. For example, the encoder includes 4 layers of image feature extraction layers connected in sequence.

Step 3.2: Input the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix into a feature extraction unit to obtain a feature matrix to be decoded.

In this embodiment, the feature extraction unit includes a graph attention module, a channel attention module and a residual structure.

Step 3.21, input the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix into the graph attention module. The graph attention module captures the similarity between the features to obtain the fifth feature matrix corresponding to the Gaussian noise image, the sixth feature matrix and the seventh feature matrix corresponding to the two MR images to be processed, and the eighth feature matrix corresponding to the CT image to be processed.

Step 3.211, input the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix into the graph attention module, and then divide the first feature matrix into a plurality of first matrix blocks.

For example, the size of the first matrix block is I rows and 1 column, where I is 4, for example.

Step 3.212, select a first matrix block in sequence, and select a second matrix block, a third matrix block and a fourth matrix block having the same position as the first matrix block from the second characteristic matrix, the third characteristic matrix and the fourth characteristic matrix respectively.

Specifically, the first matrix blocks can be selected in sequence from left to right and then from top to bottom. For the currently selected first matrix block, the second matrix block with the same position as the first matrix block is selected in the second characteristic matrix, the third matrix block with the same position as the first matrix block is selected in the third characteristic matrix, and the fourth matrix block with the same position as the first matrix block is selected in the fourth characteristic matrix.

Step 3.213, combine the first matrix block, the second matrix block, the third matrix block and the fourth matrix block into a combined matrix.

Specifically, in order of arrangement from left to right, the first matrix block, the second matrix block, the third matrix block and the fourth matrix block are combined into a combined matrix. The size of the combined matrix is, for example, 4 rows and 4 columns.

Step 3.214, select a number of pixel points adjacent to the first matrix block to obtain a pixel matrix.

Specifically, all the pixels that are most adjacent to the first matrix block are selected in the first feature matrix, and then a number of pixels are randomly selected from these pixels to form a pixel matrix. For example, 8 pixels are selected to form a pixel matrix of 4 rows and 2 columns.

Step 3.215, combine the combination matrix and the pixel matrix into a stitching matrix.

Specifically, the combination matrix and the pixel matrix are spliced to obtain a spliced matrix. The size of the spliced matrix is, for example, 4 rows and 6 columns.

Step 3.216: According to the connection relationship between the first matrix block, the second matrix block, the third matrix block and the fourth matrix block, weighted processing is performed on the first matrix block, the second matrix block, the third matrix block and the fourth matrix block in the spliced matrix, respectively, to obtain a fifth matrix block, a sixth matrix block, a seventh matrix block and an eighth matrix block respectively.

Specifically, for the first matrix block in the spliced matrix, a matrix block connected to the first matrix block is first selected from the second matrix block, the third matrix block and the fourth matrix block, and then the first matrix block, the matrix block connected to the first matrix block, and each column element in the pixel matrix are weighted summed to obtain the fifth matrix block. For example, the second matrix block and the third matrix block are both connected to the first matrix block, and the fourth matrix block is not connected to the first matrix block. Therefore, the second matrix block and the third matrix block are multiplied by a coefficient respectively, and each column element in the pixel matrix is also multiplied by a coefficient respectively, and then the results of their multiplication are added, and then the added result is added to the first matrix block to obtain the fifth matrix block, wherein the value range of each coefficient is 0-1, and the sum of all coefficients is 1.

For the second matrix block in the spliced matrix, first select a matrix block connected to the second matrix block from the first matrix block, the third matrix block and the fourth matrix block, and then perform weighted summation on the second matrix block and the matrix block connected to the second matrix block, that is, multiply the matrix blocks connected to the second matrix block by a coefficient respectively, and then add the results of their multiplication, and then add the added result to the second matrix block to obtain a sixth matrix block. For the third matrix block in the spliced matrix, first select a matrix block connected to the third matrix block from the first matrix block, the second matrix block and the fourth matrix block, and then perform weighted summation on the third matrix block and the matrix block connected to the third matrix block, that is, multiply the matrix blocks connected to the third matrix block by a coefficient respectively, and then add the results of their multiplication, and then add the added result to the third matrix block to obtain a seventh matrix block. For the fourth matrix block in the spliced matrix, a matrix block connected to the fourth matrix block is first selected from the first matrix block, the second matrix block and the third matrix block, and then the fourth matrix block and the matrix blocks connected to the fourth matrix block are weighted summed, that is, the matrix blocks connected to the fourth matrix block are multiplied by a coefficient respectively, and then the results of their multiplication are added, and then the added result is added to the fourth matrix block to obtain an eighth matrix block.

Step 3.217, use the fifth matrix block, the sixth matrix block, the seventh matrix block and the eighth matrix block to correspondingly replace the first matrix block of the first characteristic matrix, the second matrix block of the second characteristic matrix, the third matrix block of the third characteristic matrix and the fourth matrix block of the fourth characteristic matrix.

Specifically, the first matrix block of the first characteristic matrix currently being processed is replaced by the fifth matrix block, the second matrix block of the second characteristic matrix currently being processed is replaced by the sixth matrix block, the third matrix block of the third characteristic matrix currently being processed is replaced by the seventh matrix block, and the fourth matrix block of the fourth characteristic matrix currently being processed is replaced by the eighth matrix block.

Step 3.218, repeat steps 3.212 to 3.217 until all matrix blocks in the first characteristic matrix, the second characteristic matrix, the third characteristic matrix and the fourth characteristic matrix are replaced, to obtain the fifth characteristic matrix, the sixth characteristic matrix, the seventh characteristic matrix and the eighth characteristic matrix.

Specifically, repeat steps 3.212 to 3.217, and according to the above steps, replace all the first matrix blocks in the first characteristic matrix with the fifth matrix block to obtain the fifth characteristic matrix, replace all the second matrix blocks in the second characteristic matrix with the sixth matrix block to obtain the sixth characteristic matrix, replace all the third matrix blocks in the third characteristic matrix with the seventh matrix block to obtain the seventh characteristic matrix, and replace all the fourth matrix blocks in the fourth characteristic matrix with the eighth matrix block to obtain the eighth characteristic matrix.

Step 3.22: Input the fifth feature matrix, the sixth feature matrix, the seventh feature matrix, and the eighth feature matrix into the channel attention module. The channel attention module obtains the first weight of the fifth feature matrix, the second weight of the sixth feature matrix, the third weight of the seventh feature matrix, and the fourth weight of the eighth feature matrix through convolution operation. The first weight is multiplied by the fifth feature matrix, the second weight is multiplied by the sixth feature matrix, the third weight is multiplied by the seventh feature matrix, and the fourth weight is multiplied by the eighth feature matrix. All the multiplication results are added together to obtain the ninth feature matrix.

Specifically, the channel attention module performs convolution operations on the fifth feature matrix, the sixth feature matrix, the seventh feature matrix, and the eighth feature matrix through the convolution kernel, thereby obtaining the first weight, the second weight, the third weight, and the fourth weight, and then multiplies the first weight with the fifth feature matrix, the second weight with the sixth feature matrix, the third weight with the seventh feature matrix, and the fourth weight with the eighth feature matrix. Finally, all the multiplication results are added together to obtain the ninth feature matrix. In this way, the feature matrices can be integrated as much as possible while appropriately preserving the original features.

Step 3.23: Input the ninth feature matrix into the residual structure, and after convolution operation, obtain the feature matrix to be decoded.

Specifically, the ninth feature matrix is input into the residual structure, and then the convolution kernel is used to perform a convolution operation on the ninth feature matrix to restore the number of channels to the original number of channels, thereby obtaining a feature matrix to be decoded, and the feature matrix to be decoded is output to the decoder. The residual structure can ensure the positive effect of the image.

Step 3.3: Input the feature matrix to be decoded into the decoder to obtain the first MR enhanced image.

Optionally, the decoder has multiple image feature reconstruction layers connected in sequence, the number of image feature reconstruction layers and image feature extraction layers is the same, the structure of all image feature reconstruction layers is the same, and the image feature reconstruction layers include sequentially connected convolution layers, deconvolution layers, normalization layers and activation layers. The decoder is used to enhance the perception and extraction of image features.

Step 3.4, based on the dynamic differential loss function constructed based on the first MR enhanced image and the processed MR image to be registered, adjust the parameters of the target image sequence enhancement model to be trained by back propagation to obtain the preliminarily trained target image sequence enhancement model, and obtain the second MR enhanced image output by the preliminarily trained target image sequence enhancement model; wherein, the processed MR image to be registered is selected from the training set and is between the two processed MR images input to the target image sequence enhancement model to be trained, and the processed CT image input to the target image sequence enhancement model to be trained and the processed MR image to be registered are in the same group of matching image pairs.

Specifically, a dynamic differential loss function is constructed using the first MR enhanced image and the MR image to be processed to be registered, so as to obtain a loss value through the dynamic differential loss function, and then the parameters of the target image sequence enhancement model to be trained are adjusted by back propagation until the loss value reaches a minimum or the training reaches a maximum number of training times, thereby obtaining a preliminarily trained target image sequence enhancement model. At this time, the output of the preliminarily trained target image sequence enhancement model is the second MR enhanced image.

Here, the MR image to be registered is spatially located between the two MR images to be processed input to the target image sequence enhancement model to be trained, and matches the CT image to be processed input to the target image sequence enhancement model to be trained.

Alternatively, the dynamic differential loss function is expressed as:

;

in, represents the dynamic difference loss function, represents the MR image to be registered. represents the first MR enhanced image, Indicates that the goal of the generator is to arrive The mapping of Indicates the first MR enhanced image Pixels to perform the desired operation, represents the probability distribution of the first MR enhanced image, represents the number of pixels in the MR image to be processed to be registered, Indicates that the generator will register the first The difference in the results after the pixels are mapped to the first MR enhanced image, Indicates all The average value of .

Step 3.5: input the second MR enhanced image into the discriminator to obtain a trained target image sequence enhancement model according to the discrimination result of the discriminator.

Step 3.51: Input the second MR enhanced image into the discriminator to obtain a discriminant value.

Step 3.52, determine the relationship between the discriminant value and the discriminant threshold. If the discriminant value is greater than the discriminant threshold and the discrimination result is false, then update the discriminant threshold according to the loss value obtained by the kurtosis loss function, and continue to train the preliminarily trained target image sequence enhancement model (that is, continue to train the preliminarily trained target image sequence enhancement model according to the above training steps) until the discriminant value is less than or equal to the latest discriminant threshold, and obtain the trained target image sequence enhancement model. If the discriminant value is less than or equal to the discriminant threshold and the discrimination result is true, then the preliminarily trained target image sequence enhancement model is used as the trained target image sequence enhancement model.

In this embodiment, the updating method of the discrimination threshold is: multiplying the loss value obtained by using the kurtosis loss function by the unupdated discrimination threshold to obtain the updated discrimination threshold.

Alternatively, the kurtosis loss function is expressed as:

;

in, represents the kurtosis loss function, represents the second MR enhanced image, Indicates that the goal of the generator is to arrive The mapping of Indicates the second MR enhanced image Pixels to perform the desired operation, represents the probability distribution of the second MR enhanced image, represents the number of pixels in the second MR enhanced image, Indicates the second MR enhanced image pixels, represents the mean value of all pixels in the second MR enhanced image, represents the standard deviation of the second MR enhanced image, Indicates the first pixels, represents the mean value of all pixels in the MR image to be processed to be registered, Represents the standard deviation of the MR images to be registered.

Optionally, the discriminator includes 4 convolutional layers with sizes ranging from 16×16 to 4×4 and connected in sequence, and also includes a normalization layer, an activation function layer and a fully connected layer after the 4 convolutional layers.

In addition, since the MR image data set is small, it is easy to cause gradient vanishing or overfitting problems. Therefore, norm regularization for image gradients can be added after the discriminator to improve the training effect and obtain more stable judgment results.

In a specific embodiment, after obtaining the trained target image sequence enhancement model, the trained target image sequence enhancement model can also be tested using a test set, and the remaining several groups of matching image pairs obtained above that are not used as training sets can be used as test sets. The trained target image sequence enhancement model can be tested using the test set to determine the effect of the trained target image sequence enhancement model.

The present invention performs sequence registration based on the combination of mutual information and inter-layer spacing information; by introducing a multi-channel input and feature extraction unit in the generator, MR enhanced images that can be used for image fusion and automatic segmentation model training are generated, which effectively alleviates the problem of data shortage in subsequent fusion and segmentation model training, and is beneficial to improving the accuracy of image three-dimensional reconstruction, providing basic technical support for the field of medical image processing.

The trained target image sequence enhancement model obtained by the present invention can generate MR enhanced images, which can be used for the training of subsequent fusion segmentation models, and help solve the problem of lack of medical image data. The graph attention module used can capture the similarity information between images and optimize the details of local features; the channel attention module can adjust the weight value of each layer of input information in the feature extraction process, and work together to make the generated MR enhanced images more realistic, which can provide effective data for subsequent training and other applications.

Embodiment 2

The present invention further provides a multimodal image registration method based on the first embodiment, and the multimodal image registration method includes:

Acquire the CT image to be registered;

The CT image to be registered is input into the trained target image sequence enhancement model described in the first embodiment to obtain a registered MR image.

Specifically, based on the trained target image sequence enhancement model obtained in Example 1, the CT image that actually needs to be registered (i.e., the CT image to be registered) can be input into the trained target image sequence enhancement model, and the generator of the trained target image sequence enhancement model will output the registered MR image.

The implementation principle and technical effect of the multimodal image registration method provided in this embodiment are similar to those of the multimodal image enhancement and registration method provided in Embodiment 1, and will not be described in detail here.

In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims (8)

1. A training method for a model for multi-modal image enhancement and registration, the training method comprising:

Acquiring an MR image sequence to be processed and a CT image sequence to be processed, wherein the MR image sequence to be processed comprises M MR images to be processed which are arranged in sequence, the CT image sequence to be processed comprises N CT images to be processed which are arranged in sequence, and M and N are integers larger than 0;

Obtaining M groups of matched image pairs according to mutual information of the MR image to be processed and the CT image to be processed, and selecting a plurality of groups of matched image pairs from the M groups of matched image pairs as a training set, wherein each group of matched image pairs comprises one MR image to be processed and one CT image to be processed which are matched with each other;

Selecting two MR images to be processed and one CT image to be processed in the training set, inputting the two MR images to be processed, the CT image to be processed and the Gaussian noise image to be processed into a target image sequence enhancement model to be trained together so as to train the target image sequence enhancement model to be trained, and obtaining a trained target image sequence enhancement model, wherein the trained target image sequence enhancement model is used for matching CT images to be registered to obtain registered MR images; wherein, the two MR images to be processed selected in the training set are not matched with the CT images to be processed selected; the target image sequence enhancement model is a model based on generating an countermeasure network, wherein,

The target image sequence enhancement model comprises a generator and a discriminator, wherein the generator comprises an encoder, a feature extraction unit and a decoder;

Selecting two MR images to be processed and one CT image to be processed in the training set, inputting the two selected MR images to be processed, one CT image to be processed and one Gaussian noise image into a target image sequence enhancement model to be trained together so as to train the target image sequence enhancement model to be trained, and obtaining a trained target image sequence enhancement model, wherein the method comprises the following steps:

Selecting two MR images to be processed and one CT image to be processed in the training set, and inputting the two MR images to be processed, the one CT image to be processed and the one Gaussian noise image to the encoder to obtain a first feature matrix corresponding to the Gaussian noise image, a second feature matrix, a third feature matrix and a fourth feature matrix corresponding to the two MR images to be processed respectively;

Inputting the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix to the feature extraction unit to obtain a feature matrix to be decoded;

inputting the feature matrix to be decoded to the decoder to obtain a first MR enhanced image;

Based on the dynamic differential loss function constructed by the first MR enhanced image and the MR image to be processed to be registered, adjusting parameters of the target image sequence enhanced model to be trained in a counter-propagation mode to obtain a primarily trained target image sequence enhanced model, and obtaining a second MR enhanced image output by the primarily trained target image sequence enhanced model; the to-be-registered to-be-processed MR images are selected from the training set and are positioned between two to-be-processed MR images input to the to-be-trained target image sequence enhancement model, and the to-be-processed CT images input to the to-be-trained target image sequence enhancement model and the to-be-registered to-be-processed MR images are positioned in the same group of matched image pairs;

Inputting the second MR enhanced image to the discriminator so as to obtain the trained target image sequence enhanced model according to the discrimination result of the discriminator;

wherein the dynamic differential loss function is expressed as:

；

Wherein, Representing a dynamic differential loss function,Representing the MR images to be processed to be registered,A first MR enhanced image is represented,The goal of the representation generator is to slaveTo the point ofIs used for the mapping of (a),Representing the first MR-enhanced imageThe desired operation is performed by the individual pixels,Representing the probability distribution of the first MR enhanced image,Representing the number of pixels in the MR image to be processed to be registered,The representation generator registers the first MR image to be processedThe difference in the results of the mapping of the individual pixels to the first MR enhanced image,Representing allAverage value of (2).

2. Training method according to claim 1, characterized in that acquiring an MR image sequence to be processed and a CT image sequence to be processed comprises:

acquiring an initial MR image sequence and an initial CT image sequence, wherein the initial MR image sequence comprises M Zhang Chushi MR images, and the initial CT image sequence comprises N initial CT images;

Converting the brightness values of the initial MR image and the initial CT image into gray values to obtain an MR gray image and a CT gray image;

Respectively reassigning the gray values of the MR gray scale image and the CT gray scale image by adopting a histogram equalization algorithm, so that the gray values of the MR gray scale image and the CT gray scale image are uniformly distributed within a preset range, and an MR uniform distribution image and a CT uniform distribution image are obtained;

And respectively homogenizing the gray values of the MR uniform distribution image and the CT uniform distribution image to the interval of [0, 255] to obtain the MR image to be processed and the CT image to be processed, wherein all the MR images to be processed form the MR image sequence to be processed, and all the CT images to be processed form the CT image sequence to be processed.

3. Training method according to claim 1, characterized in that obtaining M sets of matching image pairs from mutual information of the MR image to be processed and the CT image to be processed comprises:

Acquiring a first MR image to be processed in the MR image sequence to be processed;

Selecting a CT image to be processed with the maximum mutual information between the CT image to be processed and the first MR image to be processed from the CT image sequence to be processed based on the mutual information between the first MR image to be processed and the CT image to be processed, and forming a first group of matched image pairs;

And based on a preset step length, selecting matched CT images to be processed from the second MR image to be processed to the M th MR image to be processed in the CT image sequence to be processed, and obtaining M-1 group matching image pairs.

4. A training method according to claim 3, wherein selecting the matched CT images from the CT image sequences to be processed for the second MR image to be processed to the mth MR image to be processed based on a preset step length, to obtain M-1 group of matched image pairs, comprises:

For the (m+1) th to-be-processed MR image, judging whether n+lambda is an integer, if so, selecting the (n+lambda) th to-be-processed CT image from the to-be-processed CT image sequence as an to-be-matched image, if not, selecting the to-be-processed CT image with larger mutual information with the (m+1) th to-be-processed MR image from two to-be-processed CT images adjacent to the (n+lambda) th to-be-processed CT image as the to-be-matched image, wherein the (M) th to-be-processed MR image and the (N) th to-be-processed CT image form an (M) th to-be-matched image pair, lambda is a preset step length, lambda=Sm/Sc, sm is an interlayer interval of the to-be-processed MR image sequence, sc is an interlayer interval of the to-be-processed CT image sequence, M is more than or equal to 1 and M is less than or equal to 1 and N is less than or equal to N;

Judging whether mutual information between the CT image to be matched and the (m+1) th MR image to be processed is larger than or equal to a preset threshold value, if so, forming the CT image to be matched and the (m+1) th MR image to be processed into an (m+1) th group matching image pair, and if not, selecting the CT image to be processed with the largest mutual information between the CT image to be matched and the (m+1) th MR image to be processed from the CT image sequence to be processed, and forming the (m+1) th group matching image pair.

5. The training method of claim 1, wherein the feature extraction unit comprises a graph attention module, a channel attention module, and a residual structure;

Inputting the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix to the feature extraction unit to obtain a feature matrix to be decoded, including:

Inputting the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix into the drawing attention module, wherein the drawing attention module obtains a fifth feature matrix corresponding to the Gaussian noise image, a sixth feature matrix, a seventh feature matrix and an eighth feature matrix corresponding to the MR image to be processed, wherein the sixth feature matrix, the seventh feature matrix and the eighth feature matrix correspond to the CT image to be processed are respectively corresponding to the two MR images to be processed through capturing the similarity among features;

Inputting the fifth feature matrix, the sixth feature matrix, the seventh feature matrix and the eighth feature matrix into the channel attention module, wherein the channel attention module obtains a first weight of the fifth feature matrix, a second weight of the sixth feature matrix, a third weight of the seventh feature matrix and a fourth weight of the eighth feature matrix through convolution operation, multiplies the first weight by the fifth feature matrix, the second weight by the sixth feature matrix, the third weight by the seventh feature matrix, the fourth weight by the eighth feature matrix, and adds all multiplication results to obtain a ninth feature matrix;

And inputting the ninth feature matrix into the residual structure, and obtaining the feature matrix to be decoded after convolution operation.

6. The training method of claim 5, wherein inputting the first feature matrix, the second feature matrix, the third feature matrix, and the fourth feature matrix to the attention module, the attention module obtaining a fifth feature matrix corresponding to the gaussian noise image, a sixth feature matrix corresponding to the two MR images to be processed, a seventh feature matrix, and an eighth feature matrix corresponding to the CT image to be processed by capturing similarities between features, comprises:

step 3.211, inputting the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix to the drawing force module, and then dividing the first feature matrix into a plurality of first matrix blocks;

Step 3.212, sequentially selecting one first matrix block, and respectively selecting a second matrix block, a third matrix block and a fourth matrix block which are the same as the first matrix block in position from the second feature matrix, the third feature matrix and the fourth feature matrix;

Step 3.213, combining the first matrix block, the second matrix block, the third matrix block and the fourth matrix block into a combined matrix;

step 3.214, selecting a plurality of pixel points adjacent to the first matrix block to obtain a pixel matrix;

Step 3.215, combining the combination matrix and the pixel matrix into a splicing matrix;

Step 3.216, respectively performing weighted summation processing on the first matrix block, the second matrix block, the third matrix block and the fourth matrix block in the spliced matrix according to the connection relationship among the first matrix block, the second matrix block, the third matrix block and the fourth matrix block, so as to respectively and correspondingly obtain a fifth matrix block, a sixth matrix block, a seventh matrix block and an eighth matrix block;

Step 3.217, replacing the first matrix block of the first feature matrix, the second matrix block of the second feature matrix, the third matrix block of the third feature matrix, and the fourth matrix block of the fourth feature matrix with the fifth matrix block, the sixth matrix block, the seventh matrix block, and the eighth matrix block;

and step 3.218, repeating the steps 3.212 to 3.217 until all matrix blocks in the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix are replaced, and correspondingly obtaining the fifth feature matrix, the sixth feature matrix, the seventh feature matrix and the eighth feature matrix.

7. The training method of claim 1, wherein inputting the second MR enhanced image to the arbiter to obtain the trained target image sequence enhancement model based on the discrimination result of the arbiter comprises:

Inputting the second MR enhanced image to the discriminator to obtain a discrimination value;

Judging the relation between the judging value and the judging threshold value, if the judging value is larger than the judging threshold value and the judging result is false, updating the judging threshold value according to the loss value obtained by the kurtosis loss function, continuing training the primarily trained target image sequence enhancement model until the judging value is smaller than or equal to the latest judging threshold value, obtaining the trained target image sequence enhancement model, and if the judging value is smaller than or equal to the judging threshold value and the judging result is true, taking the primarily trained target image sequence enhancement model as the trained target image sequence enhancement model.

8. A method for registration of multi-modal images, comprising:

Acquiring CT images to be registered;

Inputting the CT image to be registered into the trained target image sequence enhancement model according to any one of claims 1 to 7, and obtaining a registered MR image.