CN116342444B - Dual-channel multi-mode image fusion method and electronic equipment - Google Patents

Abstract

The present invention provides a dual-channel multimodal image fusion method and a fusion imaging terminal, which relate to the field of medical imaging technology. The source image is decomposed into a structure channel and an energy channel by JBF transformation; the structure channel is fused with small edge and small scale detail information such as tissue fiber by using a local gradient energy operator, and the energy channel is fused with the edge intensity, texture features and grayscale change of the organ by using a local entropy detail enhancement operator, PCNN and phase-consistent NSCT; and a fused image is obtained by inverse JBF transformation. The present invention can enhance the detail information of the fused image on the basis of maintaining the edge, noise reduction and smoothness, and improve the fusion of medical images with a similar degree to multimodal medical images. The improved local gradient energy operator is used for the structure channel to further improve the expression of the detail information of the fused image.

Description

Dual-channel multimodal image fusion method and electronic device

Technical Field

The present invention relates to the field of medical imaging technology, and in particular to a dual-channel multi-modal image fusion method and a fusion imaging terminal.

Background technique

With the application and development of sensor technology and computer technology, medical imaging technology plays an increasingly important role in modern medical diagnosis and treatment. Due to imaging mechanism and technical limitations, different images obtained by a single sensor can only reflect the local features of the lesion. Therefore, if you want to observe all the features of the part in one image, you need to extract the useful information of the target modality medical image and fuse the complementary information of multiple original medical images so that the fused image can provide a more comprehensive and reliable description of the lesion, which helps doctors make a more accurate and comprehensive diagnosis of the lesion.

In the existing technology, image fusion technology has been widely studied in the medical field, and many scholars have proposed a large number of image fusion algorithms, which are roughly divided into spatial domain technology and frequency domain technology. Spatial domain technology refers to the direct fusion operation of the source image pixel level or color space. Currently, the common ones include image pixel maximum method, image pixel weighted average method, principal component analysis (PCA) and Brovey transform. Spatial domain technology can effectively retain the spatial information of medical images, but it often causes image detail loss, contrast reduction, partial spectral information loss and spectral degradation during fusion.

The introduction of frequency domain technology has significantly improved the above problems. Currently, common frequency domain technologies include pyramid transform, wavelet transform, multi-scale transform (MST), etc. Among them, MST-related work has made breakthrough progress in recent years, including three steps: multi-scale decomposition (MSD), high- and low-frequency coefficient selection under specific methods, and inverse MSD reconstruction. As a representative of multi-scale geometric analysis, non-subsampled contourlet transform (NSCT) introduces the idea of non-subsampling on the basis of traditional contourlet transform (CT) to overcome the directional aliasing and pseudo-Gibbs phenomenon in traditional contourlet transform. However, as a frequency domain technology, NSCT lacks the expression of spatial neighborhood information such as the similarity and depth distance between pixels, which limits its ability to maintain edges, reduce noise and smooth.

At the same time, with the development of bilateral filtering theory, joint bilateral filter (JBF) is being widely used in the field of medical image fusion as a new signal processing method. Different from the fusion rules of traditional linear filters, JBF, as a nonlinear filter, introduces the Euclidean distance between pixels as weighting, calculates according to the comprehensive characteristics of spatial weight and similarity weight, effectively extracts the structural features between pixels, and solves the problem of global blurring and unsatisfactory edge structural features when using traditional average filtering and low-pass filtering to separate basic details. However, its limited number of decomposition layers and directions make the fused image still insufficient in terms of the decomposition degree of structural information and details, which restricts its further application. This work still faces huge challenges in improving the multi-feature performance and texture quality of each modality image.

Summary of the invention

The dual-channel multimodal image fusion method provided by the present invention can not only solve the problems of global blur and unsatisfactory edge structure features, but also ensure the requirements of the decomposition degree of structural information and details of the fused image, improve the multi-feature expression and texture quality of each modality image, and meet the use requirements.

The method comprises: step 1, decomposing the source image into a structure channel and an energy channel by JBF transformation;

Step 2: Use the local gradient energy operator to fuse the structural channel with the small edge and small scale detail information such as tissue fibers, and use the local entropy detail enhancement operator, PCNN and phase-consistent NSCT to fuse the energy channel with the organ edge intensity, texture features and grayscale changes;

Step 3: Obtain the fused image through inverse JBF transformation.

It should be further explained that step 1 also includes: performing global blur processing on the input image I, that is,

R _m =G _m *I (11)

Among them, R _m represents the smoothing result under the standard deviation of σ; G _m represents the Gaussian filter with variance of σ ² , and the Gaussian filter G _m at (x, y) is defined as:

The weighted average Gaussian filter is used to generate the global blurred image G, that is,

Where I represents the input image; N(j) represents the set of adjacent pixels of pixel i; represents the variance of pixel values; Z _j represents the normalization operation, that is,

JBF is used to restore the large-scale structure of the energy channel, that is,

Among them, _gs represents the intensity range function based on the intensity difference between pixels; _gd represents the spatial distance function based on the pixel distance; _Zj represents the normalization operation, that is,

σ _s ,σ _r represent the spatial weight and range weight of the bilateral filter respectively;

Get the energy channel E _I (x, y) of the source images A and B, and obtain the structure channel S _I (x, y) through equation (19);

S _I (x, y) = I (x, y) - E _I (x, y) (19).

It should be further explained that step 1 also includes: constructing a local gradient energy operator, that is,

LGE(x,y)＝NE ₁ (x,y)·ST(x,y) (20)

Among them, ST(x,y) represents the structural tensor saliency image generated by STS;

NE ₁ (x, y) represents the local energy of the image at (x, y), that is,

The size of the neighborhood at (x, y) is (2N+1)×(2N+1), where N is 4;

By comparing the local gradient energy between source images, the decision matrix S _map (x, y) is defined as

Update the decision matrix of structural channel fusion to S _mapi (x, y), that is

Among them, Ω ₁ represents a local area centered at (x, y) with a size of T×T, and T is 21;

The fused structure channel _SF (x,y) is obtained according to the following rules:

Among them, _SA (x, y) and _SB (x, y) are the structural channels of the source images A and B respectively.

It should be further explained that step 1 also includes: configuring high frequency sub-band fusion rules of energy channels;

The steps include: describing the detailed information of the high-frequency sub-band of the energy channel, and the local entropy of the image centered at (x, y) is defined as:

Where S represents a window centered at (x, y) with a size of (2N+1)×(2N+1);

The grayscale change rate at (x, y) is calculated based on the spatial frequency to reflect its detail characteristics, that is,

Where h, w represent the length and width of the source image respectively; CF, RF represent the first-order difference in the x and y directions at (i, j), respectively, and the formula is:

CF(x,y)＝f(x,y)-f(x-1,y) (27)

RF(x,y)=f(x,y)-f(x,y-1) (28)

The amplitude of the edge pixel gradient at (x, y) is calculated based on the edge density, which is specifically defined as:

Among them, s _x and s _y represent the results of Sobel operator convolution in the x and y directions respectively, that is,

s _x =T*h _x (30)

s _y =T*h _y (31)

T represents each pixel point (x, y); h _x , h _y represent the Sobel operator in the x and y directions respectively, that is,

The high frequency sub-bands of the energy channel are fused through the high frequency comprehensive measurement operator HM;

Among them, the parameters α ₁ , β ₁ , and γ ₁ are used to adjust the weights of the local entropy, spatial frequency, and edge density of the image in HM, respectively;

By comparing the size of the high-frequency sub-bands HM of the energy channel, the decision matrix E _Hmap (x, y) of the high-frequency sub-band fusion of the energy channel is obtained, which is defined as

At the same time, the fused image of the 1st to 4th layer high frequency sub-bands is obtained according to the following rules:

in, They represent the high frequency sub-bands of the 1st to 4th energy channels of the source images A and B respectively.

It should be further explained that the method uses PCNN to fuse the high-frequency sub-band of the fifth layer, and obtains the high-frequency sub-band of the fused energy channel by calculating the number of PCNN excitations.

in, They represent the high frequency sub-bands of the 5th layer energy channel of source images A and B respectively; They represent the source image and the PCNN excitation times of the high-frequency sub-band of the fifth energy channel. The formula for _Tij (n) is:

T _ij (n)＝T _ij (n-1)+P _ij (n) (38)

P _ij (n) represents the output model of PCNN.

It should be further explained that in the method, in order to obtain the output model of PCNN, the feed input and link input of the neuron at (x, y) are defined as

D _ij (n) = I _ij (39)

Among them, the parameter VL represents the amplitude of the link input;

_Wijop represents the previous excitation state of the eight neighboring neurons, that is,

Secondly, the exponential decay coefficient _{ηf is} used to calculate the decay size of the previous value of the internal activity term _Uij (n), and the link strength β is used to nonlinearly modulate _Dij (n) and _Cij (n) to obtain the current internal activity term, which is defined as

At the same time, the current dynamic threshold is iteratively updated, that is,

Where η _e and _VE represent the exponential decay coefficient and the amplitude of E _ij (n), respectively;

The current internal activity item _Dij (n) is compared with the dynamic threshold _Eij (n-1) at the n-1th iteration to determine the state of the PCNN output model _Pij (n), which is defined as

According to equations (37) and (44), the fusion result of the fifth layer high frequency subband is obtained;

The high-frequency sub-band of the fused energy channel is obtained according to the following rules Right now

It should be further noted that the method further includes configuring a low frequency sub-band fusion rule of the energy channel;

The specific steps include: The PC value at (x, y) is defined as

Among them, θ _k represents the direction angle at position k; represents the value of the amplitude of the nth Fourier component and the angle _θk ; ω represents the parameter used to remove the phase component in the image signal;

The formula is

represents the convolution result of the image pixel at (x, y), that is,

I _L (x, y) represents the pixel value of the low-frequency subband of the energy channel at (x, y); and Represents an even-odd symmetric filter bank of two-dimensional Log-Gabor of scale n.

It should be further explained that the method reflects the change of local contrast of the image by calculating the change of sharpness in the (x, y) neighborhood, which is specifically defined as:

Among them, M, N takes the value of 3; the SCM formula is

Ω ₂ represents a local region of size 3×3;

Configure local energy NE ₂ ;

Among them, M, N takes the value of 3;

The low-frequency sub-bands of the energy channel are fused through the low-frequency comprehensive measurement operator LM:

Among them, the parameters α ₂ , β ₂ , and γ ₂ are used to adjust the weights of the phase consistency value, the local sharpness change, and the local energy size in LM respectively;

The low-frequency sub-band of the fused energy channel is obtained according to the following rules Right now

in, They represent the low-frequency subbands of the energy channel of the source image respectively; E _Lmap (x, y) represents the decision matrix for the fusion of the low-frequency subbands of the energy channel, which is defined as

R _i (x,y) is defined as

N represents the number of source images; Ω ₃ represents the center of (x, y) and the size is The sliding window, The value is 7;

Use the dual coordinate operator to calculate the high frequency subband and low frequency subband Perform linear reconstruction and realize inverse NSCT transform to obtain the energy channel fusion image _EF .

It should be further explained that the method further includes: generating a structure channel fusion image _SF ((x, y)) and an energy channel fusion image _EF (x, y), and obtaining a final fusion image by superposition:

F(x,y)＝S _F (x,y)+E _F (x,y) (60)

Set the input to source images A, B;

Set the output to be the fused image F;

The specific steps include:

Step 1, read the source images A, B, and use JBF decomposition to generate the structure channel { _SA , _SB } and energy channel { _EA , _EB };

Step 2, the structure channel { _SA , _SB } is fused using the local gradient energy operator of formula (20) to generate the structure channel fused image _SF ;

Step 3, fuse the energy channels { _EA , _EB } to generate the energy channel fusion image _EF ;

Step 3.1. Use NSCT to decompose the energy channel { _EA , _EB } to generate high-frequency sub-bands of the energy channel and energy channel low frequency subband

Step 3.2, the high-frequency sub-bands of the 1st to 4th layers are fused using the high-frequency comprehensive measurement operator HM rule based on LE, SF, and ED in formula (34);

Step 3.3, fuse the fifth layer high frequency sub-band using the PCNN rule of formula (37);

Step 3.4, the low-frequency sub-band is fused using the low-frequency comprehensive measurement operator LM rule based on PC, LSCM, and NE ₂ in formula (56);

Step 3.5: For the fused high and low frequency sub-bands The energy channel E _F is generated by using the inverse NSCT transform;

Step 4: For the fused structure channel _SF and energy channel _EF , use the JBF inverse transform of formula (60) to generate the final fused image F.

The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the dual-channel multi-modal image fusion method when executing the program.

It can be seen from the above technical solutions that the present invention has the following advantages:

The dual-channel multimodal image fusion method of the present invention can enhance the detail information of the fused image on the basis of maintaining the edge, noise reduction and smoothness, and improve the similarity with the multimodal medical image. The present invention also adopts an improved local gradient energy operator for the structure channel, and adopts a low-frequency comprehensive measurement operator composed of phase, local sharpness change and local energy to calculate the low-frequency subband of the energy channel, further improving the expression of the detail information of the fused image. The energy channel generated by the JBF transform is decomposed again by NSCT and fused, which improves the multi-directional and multi-scale characteristics of the framework decomposition; a local entropy-based detail enhancement operator is proposed, which processes the 1st to 4th layer high-frequency subbands of the energy channel NSCT decomposition by calculating the local entropy, spatial frequency and edge density of the image, and uses a pulse coupled neural network (PCNN) to process the 5th layer high-frequency subband, and improves the extraction and utilization of edge contour structure and texture features in the energy channel by combining this deep learning with traditional methods.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the description will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a framework diagram of a dual-channel multimodal image fusion method;

FIG2 is a flow chart of a dual-channel multimodal image fusion method;

FIG3 is a diagram showing the fusion results of MR-T1/MR-T2 images under different values of σ _s ;

FIG4 is a line graph of MR-T1/MR-T2 fusion quality under different values of σ _s ;

FIG5 is a line graph of MR-T1/MR-T2 fusion quality under different S values.

Detailed ways

The dual-channel multimodal image fusion method provided by the present invention involves a dual-channel medical image fusion method combining JBF, NSCT and structural tensor theory with local entropy and gradient energy. Among them, the present invention adopts JBF to effectively utilize the spatial structure of the image, so that the fused medical images of various types can show good edge wrapping while being smooth, including three steps: first, the source images A and B are transformed by JBF to obtain the structural channel { _SA , _SB } and the energy channel { _EA , _EB }; secondly, the structural channel and energy channel information are extracted and fused using specific fusion rules to obtain { _SF , _EF }; finally, the fused image F is obtained by inverse JBF transformation. This process not only reflects the spatial proximity between pixels, but also considers the grayscale similarity between pixels, so as to achieve the purpose of edge preservation and denoising, and has the characteristics of simplicity, non-iteration and locality. However, as a dual-channel fusion technology, JBF has limitations in its decomposition. Incomplete image decomposition causes the energy channel to contain some detail texture information from the structural channel, so that the subsequent fusion rules cannot effectively identify and extract the corresponding information, affecting the image fusion quality. NSCT is multi-scale and anisotropic, which can describe the singularity information of the image and the characteristics of different frequency bands and directions. Considering this, NSCT is implanted into the energy channel, and the structure and detail texture of the energy channel are decomposed and fused again to achieve the purpose of improving the multi-directional and multi-scale of the model.

The NSCT involved in the present invention is based on CT, and adopts upsampling filter sampling to replace the downsampling process in the decomposition process. It consists of two parts: a non-subsampled pyramid (NSP) and a non-subsampled directional filter bank (NSDFB), which respectively perform scale decomposition and directional decomposition on the image, avoid directional aliasing and pseudo-Gibbs phenomenon caused by sampling, ensure translation invariance in the decomposition process, and improve the ability to extract image edge information. The method includes three steps: first, using NSCT to decompose the energy channel { _EA , _EB } of the source image to obtain the high-frequency sub-band of the energy channel and low frequency subband Secondly, the high-frequency and low-frequency sub-band information of the energy channel is extracted and fused through specific rules to obtain Finally, the energy channel image _EF is obtained through inverse NSCT.

For the structural tensor theory of the ^present invention, the change of the image f ₍ x,y) at (x,y) is defined as:

Generally, the local change rate C(α) is used to characterize the local geometric features of an image f(x, y) at (x, y), which is defined as

Among them, S represents the structure tensor, that is,

Represents a semi-positive definite matrix with a second-order moment

Represents the local gradient vector of the image f(x,y), the formula is

λ ₁ and λ ₂ represent the structure tensor The characteristic value of

In summary, the structural tensor significance detection operator (STS) is defined as

Based on the above technology, the source image is decomposed into a structural channel and an energy channel through the JBF transform; the local gradient energy operator is used to fuse the structural channel with the small edge and small scale detail information such as tissue fibers, and the local entropy detail enhancement operator, PCNN and phase-consistent NSCT are used to fuse the energy channel with the edge intensity, texture features and grayscale changes of the organ; the fused image is obtained through the inverse JBF transform. In this way, the fused image can enhance the detail information and improve the similarity with the multimodal medical image on the basis of maintaining the edge and noise reduction smoothness.

The dual-channel multimodal image fusion method of the present invention can also acquire and process the associated data based on artificial intelligence technology. Among them, the dual-channel multimodal image fusion method uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. The theory, method, technology and application device. Of course, the dual-channel multimodal image fusion method includes both hardware-level technology and software-level technology. Hardware technology generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, mechatronics, etc. Software technology mainly includes computer perspective technology, machine learning/deep learning and programming language. Programming languages include but are not limited to object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages.

As shown in Figures 1 and 2, a flow chart of a preferred embodiment of the dual-channel multimodal image fusion method of the present invention is shown. The dual-channel multimodal image fusion method is applied to one or more fusion imaging terminals, which are devices that can automatically perform numerical calculations and/or information processing according to pre-set or stored instructions, and whose hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

The fusion imaging terminal can be any electronic product that can interact with the user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), an interactive network television (IPTV), a smart wearable device, etc.

The fusion imaging terminal may also include network equipment and/or user equipment, wherein the network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

The network where the fusion imaging terminal is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

In order to obtain a fused image with rich details and clear texture, the present invention includes three steps of joint bilateral filter decomposition, structural channel, energy channel fusion and image reconstruction, as shown in Figures 1 and 2; the source image is decomposed into a structural channel and an energy channel through JBF transformation; secondly, the local gradient energy operator is used for the structural channel to fuse the detail information of small edges and small scales such as tissue fibers, and the local entropy detail enhancement operator, PCNN and phase-consistent NSCT are used for the energy channel to fuse the edge intensity, texture characteristics and grayscale changes of organs; finally, the fused image is obtained through inverse JBF transformation.

In an exemplary embodiment, in order to maximize the transmission of the detail texture of the source image, first, the input image I is globally blurred, that is,

R _m =G _m *I (11)

Where _Rm represents the smoothing result under standard deviation σ; _Gm represents the Gaussian filter with variance ^σ2 . The Gaussian filter _Gm at (x, y) is defined as

Then, a weighted average Gaussian filter is used to generate the global blurred image G, namely

Where I represents the input image; N(j) represents the set of adjacent pixels of pixel i; represents the variance of pixel values; Z _j represents the normalization operation, that is,

However, after global blurring, the image intensity information is relatively scattered. If it is directly used as the energy channel, the subsequent fusion rule will not be able to extract the intensity information, resulting in blurred boundaries and artifacts in the fused image. In order to generate relatively concentrated edge intensity information, JBF is used to restore the large-scale structure of the energy channel, that is,

Among them, _gs represents the intensity range function based on the intensity difference between pixels; _gd represents the spatial distance function based on the pixel distance; _Zj represents the normalization operation, that is,

σ _s ,σ _r represent the spatial weight and range weight of controlling the bilateral filter respectively.

In summary, we obtain the energy channel E _I (x, y) of the source images A and B, and the structural channel S _I (x, y) through formula (19).

S _I (x,y)＝I(x,y)-E _I (x,y) (19)

As an embodiment of the present invention, it is necessary to configure the structural channel fusion rules. Specifically, in medical imaging, the quality of detail information expression plays a decisive role in the quality of organ lesion diagnosis. In order to accurately reflect the detail information of small edge structures and fibers in organs and tissues, the local gradient energy operator based on the structural tensor and neighborhood energy is used to extract and fuse the structural channel information. In order to solve the problem that STS cannot detect the tiny detail features in some images lacking intensity functions, a local gradient energy operator (local gradient energy, LGE) is constructed, that is,

LGE(x,y)＝NE ₁ (x,y)·ST(x,y) (20)

Among them, ST(x,y) represents the structural tensor saliency image generated by STS; NE ₁ (x,y) represents the local energy of the image at (x,y), that is,

The size of the neighborhood at (x,y) is (2N+1)×(2N+1), where N is 4.

By comparing the local gradient energy between source images, the decision matrix S _map (x, y) is defined as

To ensure the regional integrity of the target image, the decision matrix of the structural channel fusion is updated to S _mapi (x, y), that is,

Among them, Ω ₁ represents a local area with a size of T×T centered at (x, y), and T is 21.

In summary, the fused structural channel _SF (x,y) is obtained according to the following rules:

Among them, _SA (x, y) and _SB (x, y) are the structural channels of the source images A and B respectively.

The present invention also configures energy channel fusion rules. Here, after JBF decomposition, the energy channel contains organ contour structure and edge strength information. At the same time, the limitations of its decomposition cause the energy channel to contain a small amount of fiber and other texture features. Therefore, it is necessary to decompose the complexity of the energy channel information again through NSCT, and use the local entropy detail enhancement operator, PCNN and phase consistency to extract and fuse texture features, organ skeleton and other contour structures, further improve the utilization rate of energy channel information, and enhance the fusion effect.

The present invention also configures the energy channel high frequency sub-band fusion rules. Specifically, through NSCT decomposition, each decomposition layer of the energy channel high frequency sub-band contains organ contour structures and fiber texture features of different scales, and the quality of decomposition layer information extraction directly affects the image fusion effect. At the same time, each decomposition layer has the situation that the image scale information decreases with the increase of the number of layers, which makes it difficult for general image fusion rules to effectively extract the image information of the highest decomposition layer.

As a neural network model, PCNN has the characteristics of pulse synchronization and global coupling. It can extract effective information from complex backgrounds, which is superior to most traditional methods. It also has significant advantages in edge detection, refinement and recognition of image fusion. Considering this, PCNN is embedded in the processing of high-frequency sub-bands to improve the extraction ability of the fifth-layer high-frequency sub-band information, so as to achieve the purpose of improving the structure and texture features of the fused image. At the same time, the local entropy enhancement detail operator is used to fuse the 1st to 4th layers of high-frequency sub-bands, further improving the degree of restoration of organ contours and fiber textures in the fused image. A large number of experiments have confirmed that the local entropy detail enhancement operator is used in the 1st to 4th layers of the high-frequency sub-bands of the energy channel, and the PCNN method is used in the 5th layer to extract and fuse the structure and texture information, which has significant advantages.

In the present invention, first, the fusion rules of the 1st to 4th layer high-frequency sub-bands are introduced. The image entropy of the present invention is a statistical method for estimating the amount of image information, which reflects the amount of detail information contained in the image. Generally speaking, the greater the entropy, the more detail information the image contains. However, the entropy of the entire image often cannot reflect the local detail information of the image. To solve this problem, the local entropy of the image is introduced to further characterize the detail information of the high-frequency sub-band of the energy channel. The local entropy (LE) of the image centered at (x, y) is defined as

Where S represents a window centered at (x, y) with a size of (2N+1)×(2N+1).

In order to further highlight its texture information, the present invention introduces spatial frequency (SF) and reflects its detail characteristics by calculating the grayscale change rate at (x, y), that is,

Where h, w represent the length and width of the source image respectively; CF, RF represent the first-order difference in the x and y directions at (i, j), respectively, and the formula is:

CF(x,y)＝f(x,y)-f(x-1,y) (27)

RF(x,y)=f(x,y)-f(x,y-1) (28)

However, LE and SF, as estimators describing image detail information, lack the extraction and expression of large-scale structural information such as contours. Therefore, edge density (ED) is introduced to highlight the structure and layering of contour edges by calculating the amplitude of the gradient of edge pixels at (x, y). It is defined as

Among them, s _x and s _y represent the results of Sobel operator convolution in the x and y directions respectively, that is,

s _x =T*h _x (30)

s _y =T*h _y (31)

T represents each pixel point (x, y); h _x , h _y represent the Sobel operator in the x and y directions respectively, that is,

Therefore, the high-frequency sub-bands of the energy channel are fused through the high-frequency comprehensive measurement operator HM.

Among them, the parameters α ₁ , β ₁ , and γ ₁ are used to adjust the weights of the local entropy, spatial frequency, and edge density of the image in HM, respectively.

By comparing the size of the high-frequency sub-bands HM of the energy channel, the decision matrix E _Hmap (x, y) of the high-frequency sub-band fusion of the energy channel is obtained, which is defined as

At the same time, the fused image of the 1st to 4th layer high frequency sub-bands is obtained according to the following rules:

in, They represent the high frequency sub-bands of the 1st to 4th energy channels of the source images A and B respectively.

Secondly, the present invention uses PCNN to fuse the fifth layer of high-frequency sub-bands, and obtains the fused energy channel high-frequency sub-bands by calculating the number of PCNN excitations.

in, They represent the high frequency sub-bands of the 5th layer energy channel of source images A and B respectively; They represent the source image and the PCNN excitation times of the high-frequency sub-band of the fifth energy channel. The formula for _Tij (n) is:

T _ij (n)＝T _ij (n-1)+P _ij (n) (38)

P _ij (n) represents the output model of PCNN.

In PCNN, _Dij (n) and _Cij (n) represent the feed input and link input of the neuron at (x, y) after n iterations, respectively. _Dij (n) is related to the intensity of the input image _Iij during the entire iteration process; the synaptic weight of _Cij (n) is related to the previous excitation state of the eight neighboring neurons. To obtain the output model of PCNN, first, the feed input and link input of the neuron at (x, y) are defined as

D _ij (n) = I _ij (39)

Among them, the parameter V _L represents the amplitude of the link input; _Wijop represents the previous excitation state of the eight-neighborhood neurons, that is,

Secondly, the exponential decay coefficient _{ηf is} used to calculate the decay size of the previous value of the internal activity term _Uij (n), and the link strength β is used to nonlinearly modulate _Dij (n) and _Cij (n) to obtain the current internal activity term, which is defined as

At the same time, the current dynamic threshold is iteratively updated, that is,

Where η _e and _VE represent the exponential decay coefficient and the amplitude of E _ij (n), respectively.

Finally, the current internal activity item _Dij (n) is compared with the dynamic threshold _Eij (n-1) at the n-1th iteration to determine the state of the PCNN output model _Pij (n), which is defined as

In summary, according to equations (37) and (44), the fusion result of the high-frequency sub-band of the fifth layer is obtained. At the same time, the high-frequency sub-band of the fused energy channel is obtained according to the following rules: Right now

As an embodiment of the present invention, an energy channel low-frequency sub-band fusion rule is also configured.

The low-frequency subband contains the changes in the brightness and grayscale of the energy channel pixels. In order to further increase the amount of information in the low-frequency subband, phase congruency is used to enhance the information of the low-frequency subband image. Phase congruency (PC) is a dimensionless measurement that is often used to reflect the clarity of an image and the importance of image features. The PC value at (x, y) is defined as

Among them, θ _k represents the direction angle at position k; represents the value of the amplitude of the nth Fourier component and the angle _θk ; ω represents the parameter used to remove the phase component in the image signal; The formula is

represents the convolution result of the image pixel at (x, y), that is,

I _L (x, y) represents the pixel value of the low-frequency subband of the energy channel at (x, y); and Represents an even-odd symmetric filter bank of two-dimensional Log-Gabor of scale n.

However, PC, as a contrast invariant, cannot reflect the local contrast changes. Therefore, the local sharpness change measure (LSCM) is introduced to reflect the local contrast changes of the image by calculating the sharpness change measure (SCM) of the (x, y) neighborhood, which is defined as

Among them, M, N takes the value of 3, and the SCM formula is

Ω ₂ represents a local region of size 3×3.

Since PC and LSCM cannot fully reflect the local signal intensity, local energy NE ₂ is introduced.

Among them, M and N are both 3.

Therefore, the low-frequency sub-bands of the energy channel are fused through the low-frequency comprehensive measurement operator LM.

Among them, the parameters α ₂ , β ₂ , and γ ₂ are used to adjust the weights of the phase consistency value, the local sharpness change, and the local energy size in LM respectively.

In summary, the low-frequency sub-band of the fused energy channel is obtained according to the following rules: Right now

in, They represent the low-frequency subbands of the energy channel of the source image respectively; E _Lmap (x, y) represents the decision matrix for the fusion of the low-frequency subbands of the energy channel, which is defined as

R _i (x,y) is defined as

N represents the number of source images; Ω ₃ represents the center of (x, y) and the size is The sliding window, The value is 7.

Finally, the high frequency subband is calibrated using the dual coordinate operator and low frequency subband Linear reconstruction is performed to realize the inverse NSCT transform and obtain the energy channel fusion image _EF .

In the embodiment of the present invention, the fused image is also reconstructed. Specifically, the structure channel fused image _SF (x, y) and the energy channel fused image _EF (x, y) are generated through the above steps, and then the final fused image is obtained by superposition:

F(x,y)＝S _F (x,y)+E _F (x,y) (60)

Set the input to source images A, B;

Set the output to be the fused image F;

The specific steps include:

Step 1, read the source images A, B, and use JBF decomposition to generate the structure channel { _SA , _SB } and energy channel { _EA , _EB };

Step 2, the structure channel { _SA , _SB } is fused using the local gradient energy operator of formula (20) to generate the structure channel fused image _SF ;

Step 3, fuse the energy channels { _EA , _EB } to generate the energy channel fusion image _EF ;

Step 3.1. Use NSCT to decompose the energy channel { _EA , _EB } to generate high-frequency sub-bands of the energy channel and energy channel low frequency subband

Step 3.2, the high-frequency sub-bands of the 1st to 4th layers are fused using the high-frequency comprehensive measurement operator HM rule based on LE, SF, and ED in formula (34);

Step 3.3, fuse the fifth layer high frequency sub-band using the PCNN rule of formula (37);

Step 3.4, the low-frequency sub-band is fused using the low-frequency comprehensive measurement operator LM rule based on PC, LSCM, and NE ₂ in formula (56);

Step 3.5: For the fused high and low frequency sub-bands The energy channel E _F is generated by using the inverse NSCT transform;

Step 4: For the fused structure channel _SF and energy channel _EF , use the JBF inverse transform of formula (60) to generate the final fused image F.

In this way, the dual-channel multimodal image fusion method of the present invention can enhance the detail information of the fused image on the basis of maintaining the edge, noise reduction and smoothness, and improve the similarity with the multimodal medical image. The present invention also adopts an improved local gradient energy operator for the structural channel, and adopts a low-frequency comprehensive measurement operator composed of phase, local sharpness change and local energy to calculate the low-frequency subband of the energy channel, further improving the expression of the detail information of the fused image. The energy channel generated by the JBF transform is decomposed again by NSCT and fused, which improves the multi-directional and multi-scale characteristics of the framework decomposition; a local entropy-based detail enhancement operator is proposed, which processes the 1st to 4th layer high-frequency subbands of the energy channel NSCT decomposition by calculating the local entropy, spatial frequency and edge density of the image, and uses a pulse coupled neural network (PCNN) to process the 5th layer high-frequency subband, and improves the extraction and utilization of edge contour structure and texture features in the energy channel by combining this deep learning with traditional methods.

Further, as the experiment and result analysis of the specific implementation of the above embodiment, in order to verify the technical effect of the method of the present invention, the following is further explained with the specific implementation effect: setting of experimental data, setting of test images. In order to fully verify the superiority of this method, a comprehensive and extensive experimental analysis is carried out. Experiments are conducted on four sets of human brain image data sets captured by different imaging mechanisms from the Harvard Medical School website①. The resolution of each test image is set to 256×256, of which 118 pairs of multimodal medical images are used to fully verify the effectiveness of this method. Four pairs of magnetic resonance imaging groups (MR-T1/MR-T2), four pairs of electronic computed tomography and magnetic resonance imaging groups (CT/MR), four pairs of magnetic resonance imaging and single photon emission computed tomography groups (MR/SPECT), and four pairs of magnetic resonance imaging and positron emission computed tomography groups (MR/PET) are randomly selected, and their experimental results are analyzed from visual and objective indicators respectively.

All experiments were written in Matlab 2018, and the running environment was AMD Ryzen 7 5800 with Radeon Graphics 3.20 GHz and RAM was 16.0 GB.

The present invention uses six commonly used metrics to comprehensively and quantitatively evaluate the performance of different fusion methods. First, the present invention uses three metrics, peak signal to noise ratio (PSNR), structural similarity (SSIM), and mutual information (MI), to measure the similarity between the fused image and the source image. The higher the metric, the smaller the distortion produced by the fusion process, and the more similar the source image and the fused image are.

Among them, PSNR measures the similarity between the source image and the fused image by calculating the mean square error between them, SSIM measures the structural similarity between the source image and the fused image, and MI measures the correlation by calculating the information entropy of the fused image and the joint information entropy of the fused image and the source image. Secondly, the present invention uses three indicators, namely spatial frequency (SF), standard deviation (SD) and edge information retention (Qabf), to measure the edge information of the source image, the retention of detail texture and the contrast of the fused image. The higher the index, the more details and texture information the fused image contains, and the better the quality of the visual information obtained from the source image. In addition, in order to further evaluate the performance of image fusion, the present invention also introduces information entropy (EN) and visual information fusion fidelity (VIFF) indicators to further measure the amount of information contained in the fused image and the degree of restoration of the fused image to the source image. The higher the index, the better the fusion performance and the less distortion of the fused image.

The verification method of the present invention is to adjust one parameter by fixing other parameters, using 118 pairs of multimodal medical images to generate a series of fusion results, and evaluate them from aspects such as similarity index measurement and visual effect to determine the optimal value of the parameter. The optimal parameters are analyzed below using MR-T1/MR-T2 image fusion as an example.

1) Gaussian standard deviation σ _s :

As the spatial weight of the bilateral filter, the Gaussian standard deviation _σs determines the quality of the source image spatial information recognition and affects the texture structure of the fused image and the degree of similarity with the source image. Therefore, it is particularly important to set a suitable Gaussian standard deviation _σs .

In order to determine the optimal value of _σs , other parameters are fixed to be between 1 and 6. The experimental results are shown in Figure 3. As can be seen from the close-up area of Figure 3c, the detail information from the source image is severely weakened, there are obvious artifacts in the sulci and gyri, and even serious detail loss occurs in the suprasellar cistern. At the same time, Figure 3c also shows that the grayscale information of the fused image is distorted and does not match the source image information, which is not desirable in medical diagnosis. As can be seen from the close-up area of Figure 3d, this situation has been improved, and the contour structure of the fused image is more distinct than that of Figure 3c, but there is still a clear lack of texture features in the sulci and gyri, and the similarity with the source image is uneven. Its grayscale change cannot accurately reflect the lesion information, which seriously affects the accuracy of medical diagnosis. It can be seen from the close-up areas of Figures 3f, 3g, and 3h that as the value of _σs increases, the fusion energy loss increases, the contrast decreases, and the fiber texture features are significantly weakened. At the same time, the imbalance between the similarity between the fused image and the MR-T1 and MR-T2 images becomes more and more obvious. As the value of _σs increases, the MR-T2 image information contained in the anterior crus of the lateral ventricle in the fused image gradually decreases. Even when the value of _σs is 6, the fused image cannot reflect its MR-T2 information. However, when the value of _σs is 3, the fused image has obvious advantages over other values in terms of texture detail feature expression, source image information restoration, and balancing MR-T1 and MR-T2 image information. As can be seen from Figure 3e, the texture of the sulcus and gyrus of the fused image is clearer, the fiber details change significantly, and the grayscale of the anterior crus of the lateral ventricle is balanced, the edges are clear, and there are no artifacts and distortions. In addition, it can be seen from the objective evaluation indicators in Table 1 that when the value of _σs is 3, the pixel grayscale and structural similarity between the fused image and the source image information is the highest, and the fusion performance is the best.

The data in Table 1 are presented in the form of a line graph, which can more intuitively analyze the performance advantages and disadvantages under the change of _σs , as shown in Figure 4. As can be seen from Figure 4, as _σs increases, the objective index increases accordingly, and reaches a peak when _σs is 3. However, when _σs exceeds 3, the similarity between the fused image and the source image gradually decreases, and the adverse effect increases with the increase of _σs , and the image fusion performance decreases accordingly. Therefore, when _σs is 3, the similarity between the fused image and the source image is the highest, the texture detail information is the most significant, and the fusion performance is the best, whether in subjective analysis or objective indicators. Therefore, the optimal value _{of σs} is adjusted to 3. In addition, through a large number of experiments, it is found that the experimental results will not be affected by the value of the parameter _σr in equation (16), so _σr is taken as 0.05.

Table 1 Objective evaluation of MR-T1/MR-T2 fusion results under different values of σ _s

Note: Bold indicates optimal values.

2) Window size S:

In the energy channel high frequency subband detail enhancement operator, the window size S, as the window size of the local image entropy, determines the source image block situation, and thus depicts the source image information by calculating the entropy value of each block. Therefore, setting a suitable window size S plays a vital role in the quality of the source image information extraction.

By fixing other parameters, the window size S is set between 1 and 6, and the experimental results are shown in Table 2. As can be seen from Table 2, as the window size S increases, the objective indicators all increase until the window size S is 3, the PSNR and MI values reach the best state, at which time the fusion performance is the best and the restoration of the source image information is the highest. However, as the window size S increases, when its value exceeds 3, the similarity index between the fused image and the source image gradually decreases. At the same time, as the window size S increases, the SSIM value also increases. When its value is 2, SSIM reaches the best state, and the best state is maintained. When the window size S exceeds 5, the SSIM value begins to decrease, and the fusion performance and image quality become worse and worse. In summary, when S is 3, the pixel grayscale and structural similarity between the fused image and the source image information is the highest, and the fusion performance is the best.

Similarly, the data in Table 2 are presented in the form of a line graph, as shown in Figure 5. As can be seen from Figure 5, when the window size S is 3, compared with other values, all objective indicators are at peak values. At this time, the fused image reaches the best state in terms of the degree of restoration and similarity of the source image, as well as the expression of detailed texture, which is conducive to the capture and analysis of patient lesion information by medical workers and improves the reliability and authenticity of medical diagnosis. Therefore, the optimal value of S is adjusted to 3.

Table 2 Objective evaluation of MR-T1/MR-T2 fusion results under different S values

Note: Bold indicates optimal values.

The units and algorithm steps of each example described in the embodiment disclosed in the dual-channel multimodal image fusion method provided by the present invention can be implemented by electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

The dual-channel multimodal image fusion method provided by the present invention is a combination of the units and algorithm steps of each example described in the embodiments disclosed herein, and can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims (2)

1. A dual-channel multimodal image fusion method, characterized in that the method comprises: Set the input to the source image , ; Set the output to be the fused image F; Specifically include: Step 1. Read the source image , , using JBF decomposition to generate structural channels and energy channels ; Step 1 also includes: Perform global fuzzy processing, that is (11) in, Indicates that the standard deviation is The smoothing result under The variance is The Gaussian filter, Gaussian filter at defined as: (12) Generate a globally blurred image using a weighted average Gaussian filter , Right now (13) in, represents the input image; Represents pixel The set of adjacent pixels; represents the variance of pixel values; represents the normalization operation, that is, (14) JBF is used to recover the large-scale structure of the energy channel, namely (15) in, represents an intensity range function based on the intensity differences between pixels; represents the spatial distance function based on pixel distance; represents the normalization operation, that is, (16) (17) (18) , Respectively represent the spatial weight and range weight of controlling the bilateral filter; Get the energy channel of the input image , and the structural channel is obtained by formula (19) ; (19); Thus, the structural channels of the input source images A and B are obtained respectively. , and energy channels , ; Step 2: Structural channel The local gradient energy operator fusion of formula (20) is used to generate the structure channel fusion image ; Step 2 also includes: constructing a local gradient energy operator, namely (20) in, represents the structural tensor saliency image produced by STS; Indicated in The local energy of the image at (twenty one) The size of the neighborhood is , The value is 4; By comparing the local gradient energy between source images, the decision matrix is obtained defined as (twenty two) Update the decision matrix of structure channel fusion as , Right now (twenty three) in, Indicates The local area with the center being 21*21 in size; The fused structure channel fusion image is obtained according to the following rules , Right now (twenty four) in, , The source images , Structural channels; Step 3: Energy channel Fusion generates energy channel fusion image ; Step 3.1, Energy channel Use NSCT decomposition to generate high-frequency sub-bands of energy channels and energy channel low frequency subband ; Step 3.2, the high-frequency sub-bands of the 1st to 4th layers are fused using the high-frequency comprehensive measurement operator HM rule based on LE, SF, and ED in formula (34); Step 3.2 also includes: configuring high frequency sub-band fusion rules for energy channels; Configuring the high-frequency sub-band fusion rules of the energy channel includes: describing the detailed information of the high-frequency sub-band of the energy channel, The local entropy of the image centered is defined as: (25) in, Indicates is the center, the size is Window; Based on spatial frequency calculation The grayscale change rate at reflects its detail characteristics, that is, (26) in, , Respectively represent the length and width of the source image; CF, RF represent the The first-order difference in the x and y directions is: (27) (28) Based on edge density calculation The amplitude of the gradient of the edge pixel at is specifically defined as: (29) in, , Respectively , The result of the Sobel operator convolution in the direction is (30) (31) Represents each pixel ; , Respectively , The Sobel operator in the direction, that is (32) (33) The high frequency sub-bands of the energy channel are fused through the high frequency comprehensive measurement operator HM; (34) Among them, the parameters , , They are used to adjust the weights of local entropy, spatial frequency and edge density of the image in HM respectively; By comparing the size of the high-frequency sub-bands HM of the energy channel, the decision matrix of the high-frequency sub-band fusion of the energy channel is obtained. , defined as (35) At the same time, the fused image of the 1st to 4th layer high frequency sub-bands is obtained according to the following rules: ; (36) in, , Represents the source images , High frequency sub-bands of the 1st to 4th layer energy channels; Step 3.3, fuse the fifth layer high frequency sub-band using the PCNN rule of formula (37); In this method, PCNN is used to fuse the high-frequency sub-bands of the fifth layer. By calculating the number of PCNN excitations, the high-frequency sub-bands of the fused energy channel are obtained. ; (37) in, , Represents the source images , The 5th layer energy channel high frequency sub-band; , They represent the source image and the PCNN excitation times of the high-frequency sub-band of the 5th energy channel, The formula is (38) Represents the output model of PCNN; In the method, in order to obtain the output model of PCNN, The feed input and link input of the neuron at is defined as (39) (40) Among them, the parameters represents the amplitude of the link input; represents the previous excitation state of the eight neighboring neurons, that is, (41) Secondly, using the exponential decay coefficient Calculate internal activity items The decay magnitude of the previous value, and by the link strength right and Perform nonlinear modulation to obtain the current internal activity item, which is defined as (42) At the same time, the current dynamic threshold is iteratively updated, i.e. (43) in, and denote the exponential decay coefficient and Amplitude of Using the current internal activity With Dynamic threshold at iteration Compare the size and judge the PCNN output model The state is defined as (44) According to equations (37) and (44), the fusion result of the fifth layer high frequency subband is obtained; Step 3.4, for the low frequency sub-band, use formula (56) based on PC, LSCM, The low-frequency comprehensive measurement operator LM rule is used to fuse them; Method by calculation The change in neighborhood sharpness reflects the change in local contrast of the image, which is specifically defined as: (53) in, , The value is 3; the SCM formula is (54) Indicates size Partial area; Configuring Local Energy ; (55) in, , The value is 3; The low-frequency sub-bands of the energy channel are fused through the low-frequency comprehensive measurement operator LM: (56) Among them, the parameters , , They are used to adjust the weights of phase consistency value, local sharpness change and local energy size in LM respectively; The method also includes configuring a low frequency sub-band fusion rule for the energy channel; Specifically include: The PC value at is defined as (46) in, Indicates that it is located The direction angle at Indicates Fourier components and angles The value of the amplitude; represents a parameter for removing a phase component from an image signal; The formula is (47) (48) (49) , Indicates that it is located The convolution result of the image pixel at is (50) (51) (52) Indicates that it is located The pixel value of the low-frequency subband of the energy channel at; and Indicates the scale size Two-dimensional Log-Gabor even-odd symmetric filter bank; The low-frequency sub-band of the fused energy channel is obtained according to the following rules , Right now (57) in, , They represent the low-frequency subbands of the energy channel of the source image respectively; The decision matrix for the fusion of low-frequency subbands of the energy channel is defined as (58) defined as (59) Indicates the number of source images; Indicates is the center, the size is The sliding window, , The value is 7; Step 3.5: Obtain the high-frequency sub-band of the fused energy channel according to the following rules , Right now (45); Step 3.6: For the fused high and low frequency sub-bands Generate energy channel fusion image using NSCT inverse transform ; Use the dual coordinate operator to calculate the high frequency subband and low frequency subband Perform linear reconstruction, implement NSCT inverse transform, and obtain energy channel fusion image ; Step 4: Fuse the fused structural channel image Fusion image with energy channel ,The JBF inverse transform of formula (60) is used to generate the final fused image F; (60). 2. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the dual-channel multimodal image fusion method as claimed in claim 1 are implemented.