CN113538693B

CN113538693B - Microwave mammary gland image reconstruction method based on deep learning

Info

Publication number: CN113538693B
Application number: CN202110763044.XA
Authority: CN
Inventors: 张朝霞; 鲁雅; 海泽瑞; 王倩; 王锟锟; 周晓玲
Original assignee: Taiyuan University of Technology
Current assignee: Taiyuan University of Technology
Priority date: 2021-07-06
Filing date: 2021-07-06
Publication date: 2022-10-14
Anticipated expiration: 2041-07-06
Also published as: CN113538693A

Abstract

The invention discloses a microwave mammary gland image reconstruction method based on deep learning, and a neural network can convert a measured microwave signal acquired from a 3GHz antenna array into an image. The invention constructs a first self-encoder network, and expresses a high-resolution image by using a vector; a second self-encoder network is then constructed, aiming at mapping the microwave signal to compressed features (vectors). And combining the two self-encoder networks into a complete composite self-encoder network for reconstruction. This approach reduces the difficulty of training a deep learning network for backscattering. The simulation dataset consisted of longitudinal and transverse slices of the university of wisconsin breast image database. Compared with the traditional deformation Bern iteration method and the contrast source inversion method, the breast imaging method can almost perform real-time breast imaging and has a good imaging effect.

Description

Microwave mammary gland image reconstruction method based on deep learning

Technical Field

The invention relates to the technical field of machine learning, in particular to a microwave mammary gland image reconstruction method based on deep learning.

Background

Microwave breast imaging techniques are generally classified into techniques for generating qualitative images of the internal structure of the breast and techniques for generating quantitative images of the complex permittivity of the breast tissue. Quantitative techniques exploit the different dielectric properties of normal breast tissue (such as skin, fat and fibrous glands) and cancerous tumors to reconstruct the dielectric profile of the interior of the breast to detect pathological conditions. The scattered field data resulting from the irradiation of the breast tissue with microwave energy is acquired by the data acquisition device. From the collected scattered field data, the permittivity of the breast tissue is reconstructed. Mathematically, the inversion of the scattered field data is done by solving an inverse problem to reconstruct the permittivity image. The more general the reconstruction model is, the more difficult it is to solve the inverse problem, often resulting in ill-qualification. Non-compliance means that there may be no or many properties that may result in the distribution of the data (i.e., the absence and non-uniqueness of the solution to the mathematical inverse problem), and that small changes in the data may result in arbitrarily large changes in the inferred properties (i.e., instability of the solution).

The main method for solving the problem of unsuitability and inversion related to quantitative microwave breast imaging is an iteration method with large calculation amount. The traditional method is to use different iteration methods to solve the problem, but the problems of real-time imaging incapability and artifact exist.

Neural networks have recently been combined with microwave breast image reconstruction techniques, most of which use simply shaped electromagnetic parameters to represent scatterers. Also most of the articles using deep convolutional neural networks do not propose direct inversion methods, they do not move directly from the data collected by the receiver to contour estimation, they usually perform super-resolution recovery starting from the original image obtained by the traditional method. One of the most common techniques is to obtain quantitative recovery through preliminary operations, such as obtaining a priori information from scattered field data to contrast source inversion/induced current approximation, or to adopt a contrast-based neural network in combination with a traditional subspace-based optimization method, and then train a U-Net network with the obtained data to reconstruct a mammary gland image.

Disclosure of Invention

The invention overcomes the defects of the prior art, and solves the technical problems that: the traditional method has the technical problems of large calculation amount, artifact and incapability of real-time imaging, and simultaneously solves the problems that in many previous researches, models are formed by uniform cylinders or spheres, the complexity of the mammary gland cannot be accurately simulated, and the prior information obtained by the traditional method cannot be directly inverted.

The technical scheme adopted by the invention for solving the technical problems is as follows: a microwave mammary gland image reconstruction method based on deep learning is constructed, and comprises the following steps:

s1, longitudinally and transversely slicing nine breast models of four types provided by an online repository to obtain 3640 dielectric constant images of the breasts;

s2, placing all mammary gland dielectric constant images in an interested region of a numerical value scattering field generated by a moment method to obtain a scattering data array;

s3, adding extra white Gaussian noise into the scattering data array, synthesizing a noise matrix, regarding the noise matrix as a measured scattering field, and reconstructing a dielectric constant image of the mammary gland;

and S4, constructing a composite self-encoder network, inputting the dielectric constant image and the synthesized noise matrix as the self-encoder network, and outputting the reconstructed mammary gland dielectric constant image.

In step S4, the step of building the self-encoder network includes:

constructing a first self-encoder network, taking the mammary gland dielectric constant image as the encoder input of the first self-encoder network, and outputting the calculated compression weight from the decoder of the first self-encoder network;

and constructing a second self-encoder network, taking the synthesized noise matrix as the input of an encoder of the second self-encoder network, simultaneously inputting the compression weight calculated by a decoder of the first self-encoder network into a decoder of the second self-encoder network, and outputting the compression weight as a reconstructed mammary gland dielectric constant image by the decoder of the second self-encoder network.

Wherein the first self-encoder network consists of 5 convolutional layers, the step size in each convolutional layer is 2, the filter size in the first layer is 6 × 6, the filter sizes in the second and third layers are 5 × 5, the filter size in the fourth layer is 4 × 4, the filter size in the fifth layer is 7 × 7, and no padding is used in the encoder; the decoder starts from two fully connected layers, each containing 2048 neurons and 16384 neurons, followed by two convolutional layers of 3 × 3, and finally one deconvolution layer; the goal is to find a good representation of the original real image in a low dimensional space, i.e. a compressed representation of the original large image.

Wherein the second self-encoder network comprises five convolutional layers and two fully-connected layers; the filter sizes of the first three convolutional layers are all 5 multiplied by 5, the step length is 1, and filling is not used; 2048 neurons are in the first fully-connected layer, and 256 neurons are in the second layer; the decoder formed by the training of the first self-encoder network forms a complete structure.

The mammary gland image reconstructed by deep learning is compared with a traditional method deformation Bern iteration method and a contrast source inversion method.

Compared with the prior art, the invention has the following beneficial effects:

the invention provides microwave mammary gland imaging based on deep learning, which comprises the steps of firstly training an auto-encoder to obtain compressed representations of original large images, then using the compressed representations as labels, and training neural network mapping from microwave signals to the compressed representations; compared with the traditional method for solving the problem of microwave mammary gland electromagnetic backscattering, the method can realize real-time imaging and has small calculated amount; compared with the prior neural network, the invention can directly reconstruct the mammary gland image by the scattered field data, and has more practical significance when the real mammary gland image is used.

Drawings

The invention will be further described with reference to the accompanying drawings and examples, in which:

fig. 1 is a schematic flow diagram of a microwave breast image reconstruction method based on deep learning according to the present invention.

Fig. 2 is a schematic diagram of a dielectric characteristic curve of a normal breast tissue in the microwave breast image reconstruction method based on deep learning provided by the invention.

Fig. 3 is a schematic structural diagram of a numerical electromagnetic field model in the microwave breast image reconstruction method based on deep learning according to the present invention.

Fig. 4 is a schematic structural diagram of a composite self-encoder network model in the microwave breast image reconstruction method based on deep learning provided by the invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described are only for illustrating the present invention and are not to be construed as limiting the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Referring to fig. 1, the microwave breast image reconstruction method based on deep learning provided by the invention comprises the following steps:

s1, longitudinally and transversely slicing nine mammary gland models of four types provided by an online storage library to obtain 3640 dielectric constant images of mammary glands;

In step S4, the step of building the self-encoder network includes:

constructing a first self-encoder network, taking the mammary gland dielectric constant image as the input of an encoder of the first self-encoder network, and outputting the calculated compression weight from a decoder of the first self-encoder network;

Wherein the second self-encoder network comprises five convolutional layers and two fully-connected layers; the filter sizes of the first three convolutional layers are all 5 multiplied by 5, the step length is 1, and filling is not used; 2048 neurons in the first fully-connected layer and 256 neurons in the second layer; the decoder formed by the training of the first self-encoder network forms a complete structure.

The data set employed by the present invention was obtained from models developed and maintained in an online repository at the university of wisconsin computational electromagnetics laboratory. The range of dielectric properties for each tissue type (fat, transition and fibroglandular) was derived from a large-scale dielectric spectroscopy study of freshly excised breast tissue specimens. The breast model was derived from a series of T1-weighted Magnetic Resonance Imaging (MRI) of prone patients. Each model consists of a three-dimensional grid of cubic voxels, where each voxel is 0.5mm x 0.5mm. The mammary gland model included a skin layer approximately 1.5 mm thick, a subcutaneous fat layer 1.5 cm thick at the base of the mammary gland, and a muscular chest wall 0.5 cm thick.

There are 4 different classes of 9 breast models in this repository, all of which are considered in the present invention. Numerical models are classified according to radiology density as defined by the american radiology society as follows: the first category is almost exclusively fat (< 25% glandular tissue), the second category is interspersed fibroglandular tissue (25-50 glands), the third category is non-uniformly dense (51-75% glands), the fourth category is very dense (> 75% glands), these tissue density levels encode 1,2, 3 and 4, respectively, in the "classification" field. The first category has two breast models, the second category has three, the third category has three, the fourth category has one, and 9 breast models are provided. For each phantom, a two-dimensional image of each coronal plane was taken longitudinally, representing the range from the area near the nipple to the area near the muscle, and then a two-dimensional image of each model was taken transversely, yielding 3640 pictures.

Txt provides basic information of the model, including internal identification number, mesh size in mesh units, and classification of mammary gland components. Txt contains a number in the second file mtype, per line, as given in table 1. The immersion medium number is assigned to all voxels except for the mammary gland. Muscle media numbers are assigned to all voxels in the chest wall region. Normal breast models in the breast model were classified into seven tissue types, from the highest water-containing fibroglandular tissue with the highest dielectric properties (medium number = 1.1) to the lowest water-containing adipose tissue with the lowest dielectric properties (3.3). Also a transition zone (number of media = 2) has medium dielectric properties. Txt in the third file p-values for each voxel representing normal breast tissue or subcutaneous fat are in the range 0,1, with lower p-values corresponding to lower dielectric property values for a given tissue type and higher p-values corresponding to higher dielectric property values.

FIG. 2 is a graph showing the dielectric constant of normal breast tissue. The seven tissue type regions bounded by the curves in the figure are labeled with the machine numbers in table 1. From top to bottom, the curves correspond to the following eight regions: maximum, group1-high, group 1-mean, group1-low, group3-high, group 3-mean, group3-low, and minium.

Tissue type	Media number
		Immersion medium	-1
Skin	-2
		Muscle	-4
Fibroconnective/glandular-1	1.1
		Fibroconnective/glandular-2	1.2
Fibroconnective/glandular-3	1.3
		Transitional	2
Fatty-1	3.1
		Fatty-2	3.2
Fatty-3	3.3

TABLE 1 organization type and corresponding Medium numbering

Txt file, the media number of the selected voxel is first obtained, corresponding to the tissue type in table 1. The adjacent upper and lower curves are found in FIG. 1, and the εs of the corresponding adjacent upper and lower curves are found in Table 1 _∞ Δ ε and σ _s The value is obtained. Then, the p value corresponding to the p in pval.

cc_selected＝p*cc_upper+(1-p)*cc_lower (1)

Respectively obtain epsilon corresponding to the selected voxels _∞ Δ ε and σ _s The value is obtained. The relative permittivity of the mammary tissue was then obtained using a unipolar debye model. For allThe relaxation time constant of τ is set to 15ps. Epsilon ₀ Has a value of 1.

Table 2 is the maximum, group1-high, group 1-mean, group1-low, group3-high, group 3-mean, group3-low and minium curves associated with normal breast tissue and unipolar debye parameters for skin and muscle.

	ε _∞	Δε	τ(ps)	σ _s (Sm)
					minimum	2.309	0.092	13.00	0.005
group3-low	2.848	1.104	13.00	0.005
					group3-median	3.116	1.592	13.00	0.050
group3-high	3.987	3.545	13.00	0.080
					group1-low	12.99	24.40	13.00	0.397
group1-median	13.81	35.55	13.00	0.738
					group1-high	14.20	40.49	13.00	0.824
maximum	23.20	46.05	13.00	1.306
					skin	15.93	23.83	13.00	0.831
muscle	21.66	33.24	13.00	0.886

TABLE 2 unipolar Debye parameters for skin and chicken in counties associated with normal breast tissue

The numerical electromagnetic field model studied is shown in FIG. 3, considering the two-dimensional transverse magnetic case, with the longitudinal direction along

In a free space context, a non-magnetic scatterer is located in the region of interest D. Is formed by being located at

N of (C) _i Individual line source illumination p =1,2 _i . For each incidence, the scattered field is defined by the position

N of (A) _r Antenna measurement, q =1,2 _r 。

The forward problem is described by two equations. The first is the electric field integral equation, also known as the Lippmann-Schwigger equation.

Wherein E ^t (r) and E ⁱ (r) represents the total electric field and the incident electric field, respectively.

For a wavenumber of homogeneous medium background, g (r, r ') is a two-dimensional free-space green's function. The contrast current density I (r) is defined as I (r) = ξ (r) E ^t (r) contrast is ξ (r) = ε _r (r) -1. The second equation for the forward problem is described as

E ^s (r) is a scattered field of the measurement surface S. The first equation in the forward problem describes the interaction of waves with scatterers in the domain D, commonly referred to as the equation of state. The second equation describes the fringe field as re-radiation of the induced contrast current, referred to as the data equation.

For the inverse problem, the goal is N from the measurement _i Incident scattered field

Reconstructed relative dielectric constant ε _r (r) (r. Epsilon. D). Using psi as operator for solving corresponding forward problem, non-linear equation

In the absence of noise, there is no ε _r The exact solution of (a). To solve the above formula, the objective function of the optimization problem is typically constructed as

T(ε _r ) Is a regularization that balances the stability of the data fit and solution. a is a constant regularization coefficient. The above equation is non-linear and non-convex and is difficult to solve due to the presence of local minima. Many iterative optimization algorithms are proposed to solve nonlinear optimization methods such as deformation berne iteration and contrast source inversion.

Writing state equations and data equations in discrete formIn particular, the domain D is discretized into M × M subunits by using a moment method in combination with a pulse basis function and a delta check function, and the subunits are centered at r _n N =1,2,. M ² . Obtaining discrete forms of (3) and (4):

wherein the vector

And

is M ² Dimension, n-th elements are respectively

And

matrix array

And

is M ² ×M ² And N _r ×M ² Dimension, n elements are respectively

And

here, A is _n' Is the nth' subunit region, and n =1,2 ² ，n'＝1,2,...,M ² And q =1,2,. N _r 。

Contrast xi (r) of domain D is discretized into M ² Dimension vector

For convenience, another discretized contrast form is adopted, namely a matrix of dimensions M × M

Direct conversion to

Furthermore, from the definition of the contrast current density I (r), the discrete form of I (r) can be calculated as

Consider a dimension of 2 x 2m ² The domain of interest D of (a), discretizes the domain into 150 × 150 pixels. On a circle with a radius of 3 meters centered at (0, 0), there are 28 line sources and 28 line receivers. Scatterers in D are mammary gland permittivity images. Of these 3640 images, 2912 were used to train the neural network and 728 were used to test the trained neural network.

Generation of N by moment method _i Incident numerical fringe field and record N _r ×N _i Matrix of sizes

In (1). Then, additional white Gaussian noise

Will be added into

In (1). Synthetic noise matrix

The scattered field, which is considered as a measurement, is used to reconstruct the relative permittivity. The operating frequency was 3GHz.

In order to quantitatively evaluate the performance of the different methods, a relative error R is also defined _e

And

representing true and reconstructed relative permittivity images, M, respectively _t Is the number of tests performed.

Fig. 4 shows a composite self-encoder network constructed by the present invention, wherein a is a first self-encoder network composed of an encoder and a decoder, and b is a second self-encoder network composed of seven convolutional layers and two fully-connected layers. all convolutional layers in a and b contain a two-dimensional convolution operation, batch normalization, followed by a corrective linear unit (ReLU). An autoencoder is a neural network whose goal is to reproduce the input at the output. It typically generates fewer elements in the middle layer and can be considered a compact representation of the input image. The self-encoder structure proposed in c can effectively compress and decompress a two-dimensional image by using the strong imaging readability and image processing capability of the CNN. The encoder consists of 5 convolutional layers, compressing the input image to 256 × 1 pixels. Maximum pooling is not used in the encoder as this would lose information. In contrast, a step size of 2 is used in each convolutional layer of the encoder, the first layer filter size is 6 × 6, the second and third layer filter sizes are 5 × 5, the fourth layer filter size is 4 × 4, the fifth filter size is 7 × 7, and no padding is used in the encoder. The decoder starts with two fully connected layers containing 2048 neurons and 16384 neurons respectively, and then transforms 16384 pixels into 150 × 150, followed by 3 × 3 two convolutional layers, and finally one deconvolution layer. The output of the encoder can be seen as a compact representation of the input image in a low dimensional space. The upsampling in the decoder is performed using fully connected layers, which can also be implemented by transposed convolution of more layers.

In the second self-encoder network, the filter sizes of the first three convolutional layers are all 5 × 5, the step size is 1, and no padding is used. The first fully-connected layer has 2048 neurons and the second layer has 256 neurons. The input to the neural network is the microwave scattering signal (real and imaginary parts) of the mammary gland permittivity image measured by the antenna array, and the output is a 150 x 150 image with 256 elements as a compressed representation. It is therefore followed by a decoder formed in the first stage (from the encoder training) to form a complete structure. Thus, the entire neural network, which receives the microwave signal and reconstructs a 150 × 150 image, includes 5 convolutional layers (3 before and 2 after the fully-connected layers), four fully-connected layers, a shaping layer, and an deconvolution layer.

The moment method was used to simulate the illumination and scatter reception at 3GHz to obtain a 28 x 28 scatter data array, each column storing scatter data for one emitter excitation. Since there are 3640 images, a 28 × 28 × 3640 data array is obtained in total.

From the 3640 image dataset, 2912 images were randomly selected for training the first self-encoder network. The output of the first self-encoder network is a 256 x 1 vector. It is shaped to 16 x 16 because the vector will be used directly as a label to train the second self-encoder network (the output will also be a 256 x 1 vector). The size of the mini-batch is chosen to be 1% of the training set size.

The loss function is a modified Mean Square Equation (MSE). The MSE equation is calculated as follows

I _or Pixel values, I, representing the original image _de Representing pixel values of decoded pictures, N being the number of pixels in a picture, M being the number of picturesAmount (M =2912 for training set).

2912 images in the training set were sent to the encoder to obtain the corresponding compression features. The resulting compressed representation will be used as a label to train the second self-encoder network. Since the output has only 256 elements (smaller than the input size), the training is expected to converge quickly.

In the second self-encoder network, the first concealment layer contains 128 × 5 filters, which will generate 256 channels at the next layer. The first hidden layer is intended to extract some low-level features such as correlation of the scatter signal in local regions. Each filter contains a specific feature, which means that each filter learns a specific transmission rule, an insufficient number of filters results in a large MSE value, and an excessive number of filters increases unnecessary computational burden. Such studies are performed for each layer in the network design process.

The three convolutional layers have 128, 256, 192 channels, respectively. Each convolutional layer is followed by a rectifying linear unit (ReLU) before being connected to the next layer. The third convolutional layer is followed by two fully-connected layers containing 2048 neurons and 256 neurons, respectively. During the training process, 2912 scattered field data arrays and corresponding compression features were used as input markers.

The network (momentum = 0.9) was trained using a momentum stochastic gradient descent algorithm with a minimum batch of 5, with 100 iterations in one epoch. L2 regularization is used to avoid overfitting. The minimized loss function is a half mean square error function.

The test data sets were used to calculate the real-time performance of the network after each epoch by importing 728 scatter field data sets into the network and calculating the Root Mean Square Error (RMSE) between the network output and the expected one-dimensional vector. RMSE is defined as:

wherein I _tg Representing the pixel value, I, in the desired vector _rc Representing the output. N =256 isThe number of pixels in the feature is compressed. M =728 is the number of tests.

The proposed deep learning based method is compared to other non-linear methods such as the deformed born iteration method and the contrast source inversion method, which are iterative, which is time consuming for each frequency and for each iteration with respect to computation time (e.g. the deformed born iteration method has to run a forward solver in each inversion iteration, performing at least 5 hours of simulation for 15 iterations).

While the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A microwave mammary gland image reconstruction method based on deep learning is characterized by comprising the following steps:

s4, constructing a composite self-encoder network, inputting the dielectric constant image and the synthesized noise matrix as the self-encoder network, and outputting a reconstructed mammary gland dielectric constant image;

in step S4, the step of building the self-encoder network includes:

2. The microwave breast image reconstruction method based on deep learning of claim 1, wherein the first self-encoder network is composed of 5 convolutional layers, step size is 2 in each convolutional layer, filter size is 6 x 6 in the first layer, filter size is 5 x 5 in the second and third layers, filter size is 4 x 4 in the fourth layer, filter size is 7 x 7 in the fifth layer, no padding is used in the encoder; the decoder starts from two fully-connected layers, which respectively comprise 2048 neurons and 16384 neurons, followed by two convolution layers of 3 × 3 and finally a deconvolution layer; the goal is to find a good representation of the original real image in a low dimensional space, i.e. a compressed representation of the original large image.

3. The microwave breast image reconstruction method based on deep learning of claim 2, wherein the second self-encoder network comprises five convolutional layers and two fully-connected layers; the filter sizes of the first three convolutional layers are all 5 multiplied by 5, the step length is 1, and filling is not used; 2048 neurons in the first fully-connected layer and 256 neurons in the second layer; the decoder formed by the training of the first self-encoder network forms a complete structure.

4. The microwave mammary image reconstruction method based on deep learning of claim 1, wherein the mammary image reconstructed by deep learning is compared with a traditional method of modified Bern iteration method and a contrast source inversion method.