CN113538238A - High-resolution photoacoustic image imaging method and device and electronic equipment - Google Patents
High-resolution photoacoustic image imaging method and device and electronic equipment Download PDFInfo
- Publication number
- CN113538238A CN113538238A CN202110779167.2A CN202110779167A CN113538238A CN 113538238 A CN113538238 A CN 113538238A CN 202110779167 A CN202110779167 A CN 202110779167A CN 113538238 A CN113538238 A CN 113538238A
- Authority
- CN
- China
- Prior art keywords
- image
- network
- photoacoustic
- characteristic value
- resolution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 35
- 238000005070 sampling Methods 0.000 claims abstract description 64
- 238000000034 method Methods 0.000 claims abstract description 53
- 230000004927 fusion Effects 0.000 claims abstract description 25
- 238000007781 pre-processing Methods 0.000 claims abstract description 11
- 238000012545 processing Methods 0.000 claims abstract description 10
- 230000006870 function Effects 0.000 claims description 43
- 230000015654 memory Effects 0.000 claims description 21
- 238000012549 training Methods 0.000 claims description 18
- 230000004913 activation Effects 0.000 claims description 9
- 230000017105 transposition Effects 0.000 claims description 8
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 7
- 238000004422 calculation algorithm Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000013527 convolutional neural network Methods 0.000 claims description 4
- 239000000126 substance Substances 0.000 claims description 2
- 239000011159 matrix material Substances 0.000 description 11
- 238000010586 diagram Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 238000000386 microscopy Methods 0.000 description 4
- 238000004590 computer program Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 238000011478 gradient descent method Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000002059 diagnostic imaging Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000010226 confocal imaging Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000005314 correlation function Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000005284 excitation Effects 0.000 description 1
- 230000031700 light absorption Effects 0.000 description 1
- 238000002595 magnetic resonance imaging Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012014 optical coherence tomography Methods 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 230000035515 penetration Effects 0.000 description 1
- 238000010895 photoacoustic effect Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000003325 tomography Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000012285 ultrasound imaging Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
- Ultra Sonic Daignosis Equipment (AREA)
Abstract
The invention discloses a high-resolution photoacoustic image imaging method, a high-resolution photoacoustic image imaging device and electronic equipment, wherein the method comprises the following steps of: acquiring a photoacoustic image with a low sampling rate, preprocessing the photoacoustic image, and generating a preprocessed first photoacoustic image; respectively inputting the first photoacoustic image into an image restoration network and an image super-resolution network, acquiring a first characteristic value output by the image restoration network, and acquiring a second characteristic value output by the image super-resolution network; superposing the first characteristic value and the second characteristic value according to a preset weight to generate a target characteristic value; and converting the target characteristic value into a second photoacoustic image according to the characteristic fusion network, wherein the second photoacoustic image is a high-resolution photoacoustic image. The embodiment of the invention can realize restoration of the photoacoustic image with low sampling rate to generate the photoacoustic image with high resolution, and can acquire more image details and has good image restoration quality by respectively processing the image restoration network and the image super-resolution network.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a high-resolution photoacoustic image imaging method and device and electronic equipment.
Background
The basic principle of photoacoustic imaging is the photoacoustic effect of biological tissues, that is, when the biological tissues are irradiated by light, the biological tissues can absorb light energy to cause the change of internal temperature, and then acoustic signals are generated due to thermal expansion. If biological tissues are irradiated by short pulse laser scanning with the same frequency, then an ultrasonic detector is used for receiving ultrasonic waves generated by the tissues, an initial sound pressure distribution map or a light absorption energy distribution map of the surface of the tissues can be reconstructed by solving an acoustic inverse problem by adopting a proper inversion algorithm, and finally, image information of the tissues can be inverted according to the distribution maps. Compared with the traditional medical imaging methods (CT, optical coherence tomography, MRI, ultrasound imaging, etc.), the photoacoustic imaging has the advantages of non-invasive, high penetration depth and high resolution, and the main research fields at present branch photoacoustic tomography, photoacoustic microscopy, photoacoustic endoscopic imaging, etc., wherein photoacoustic microscopy (PAM) can achieve effects beyond the resolution limit of optical imaging through a point excitation mode similar to optical confocal imaging, so that the photoacoustic microscope is an important research point in current biophotonic medicine.
Photoacoustic microscopy generally scans an object by focusing light on one point, but the main problem at present is that the scanning mode is generally realized by controlling a photoacoustic component or a sample by a mechanical three-dimensional moving platform, and the common scanning precision is in the micron order. Although the mode can improve the detection sensitivity and the imaging contrast to the maximum extent, the time required for scanning a sample is too long, taking the scanning precision of 1 micron as an example, if a sample of 1 square centimeter is scanned, 10^8 times are required, a common high-precision three-dimensional moving platform can move more than 4 ten thousand times per minute, the imaging time consumption combined with the photoacoustic imaging is estimated to be 3 ten thousand times per minute, at the moment, the scanning time still needs more than 48h, and the imaging time is too long, which is one of the main problems of the current photoacoustic microscopy, so that the defect causes that the photoacoustic imaging has the defects of being insufficient for detecting the rapidly changing biological characteristics, and the problem of clinical tolerance also exists.
At present, scholars at home and abroad propose solutions for the problem of overlong photoacoustic microscopic imaging time, wherein a method for recovering a high sampling rate by using a low sampling rate image is one of main solutions, but a traditional image up-sampling method cannot achieve a good effect. In recent years, depth learning has been rapidly developed in medical imaging, and image restoration and image super-resolution technologies have been advanced sufficiently therein, and have been put to practical use in many fields, such as low-dose CT image enhancement, old photo restoration, and the like, so that a method of restoring a low-sampled photoacoustic image to a high-resolution image based on a depth learning method is more advantageous than a conventional method.
At present, two methods of image restoration and image super-resolution are not widely applied to photoacoustic images, because the main idea of image restoration is to restore missing parts in images by using the existing information in the images, but the photoacoustic images with low sampling rate cannot obtain better effect by directly using an image restoration method based on deep learning, and are particularly highlighted on the problems that the edges of the restored biological tissues are too smooth due to network overfitting and the like. The image super-resolution method based on deep learning is mainly highlighted in network model design, no network model can well learn the low sampling rate mode of the photoacoustic image at present, the basic idea of common image down-sampling methods such as bicubic linear interpolation, Gaussian blur and the like is to reduce the image volume in a mode of obtaining one pixel value by weighting and summing the pixel values in a certain range, and the method of the photoacoustic image low sampling rate is to only take one pixel value in the pixels in a certain range. From the aspect of frequency domain, the method is equivalent to the down-sampling mode in the existing image super-resolution method, namely compressing the whole signal, reserving most of low-frequency information therein, and mainly losing high-frequency information; the method for the low sampling rate of the photoacoustic image is to piecewise intercept partial frequency domain signals, namely most of low-frequency information and high-frequency information are lost. The difference of the down sampling causes the super-resolution of the image to have poor performance in restoring the photoacoustic low-sampling image, and the problems of poor network robustness, network overfitting and poor image quality are easy to occur.
Accordingly, the prior art is yet to be improved and developed.
Disclosure of Invention
In view of the defects of the prior art, the invention aims to provide a high-resolution photoacoustic image imaging method, a high-resolution photoacoustic image imaging device and an electronic device, and aims to solve the problem that the quality of a generated image is poor when the image of a low sampling rate is restored by an image super-resolution method in the prior art.
The technical scheme of the invention is as follows:
a first embodiment of the present invention provides a high-resolution photoacoustic image imaging method, including:
acquiring a photoacoustic image with a low sampling rate, preprocessing the photoacoustic image, and generating a preprocessed first photoacoustic image;
respectively inputting the first photoacoustic image into an image restoration network and an image super-resolution network, acquiring a first characteristic value output by the image restoration network, and acquiring a second characteristic value output by the image super-resolution network;
superposing the first characteristic value and the second characteristic value according to a preset weight to generate a target characteristic value;
and converting the target characteristic value into a second photoacoustic image according to the characteristic fusion network, wherein the second photoacoustic image is a high-resolution photoacoustic image.
Further, the image inpainting network is a generation countermeasure network, where the generation countermeasure network includes a generator and a discriminator, and before the first photoacoustic images are respectively input to the image inpainting network and the first feature values output by the image inpainting network are acquired, the method includes:
setting a network structure of a generator as a U-Net network, wherein the U-Net network comprises a down-sampling network and an up-sampling network corresponding to the down-sampling network, jump connections are correspondingly arranged between corresponding output characteristic values of the up-sampling network and the down-sampling network, the number of layers of the U-Net network is set to be N, the down-sampling network of a natural number with N being more than or equal to 5 comprises a convolution layer, a regularization layer and an activation function, and the up-sampling network comprises a transposition convolution layer, a regularization layer and an activation function;
the network structure of the discriminator is set as a 4-layer convolutional neural network, and the input of the discriminator is a first photoacoustic image generated by the generator, wherein the generated image corresponds to the generated image.
Further, before the first photoacoustic images are respectively input to an image repairing network and first feature values output by the image repairing network are acquired, the method further includes:
respectively inputting the first photoacoustic images into an image restoration network to obtain the target size of an output image;
and according to the target size of the output image, carrying out pixel zero padding on the first photoacoustic image to generate a first photoacoustic image with the target size.
Further, the inputting the first photoacoustic image into an image repairing network, and acquiring a first characteristic value output by the image repairing network, includes:
inputting the first photoacoustic image into an image restoration network, and acquiring an objective function of the image restoration network, wherein the formula of the objective function is as follows:
LGAN(G,N)＝minGmaxD(Ex,y[logD(x,y)]+Ex,z[log(1-D(x,G(x,z)))])
(formula 1)
Wherein G represents a generator network, D represents a discriminator network, G () represents the generator network output, and D () represents the discriminator networkOutput, E represents the L1 loss function, i.e.
yiIs the target value, f (x)i) Is an estimated value; x represents an input image, y represents a real image, and z represents random noise;
training the image restoration network according to the target function to obtain a trained target image restoration network;
and acquiring a characteristic value output by the target image restoration network and recording the characteristic value as a first characteristic value.
Further, the image super-resolution network adopts a depth residual error network structure,
inputting the first photoacoustic image into an image super-resolution network, and acquiring a second characteristic value output by the image super-resolution network, wherein the method comprises the following steps:
inputting the first photoacoustic image into an image super-resolution network, performing convolution calculation on the first photoacoustic image through a down-sampling convolution network, and outputting a first-stage feature;
inputting the first-stage features into W network residual blocks, outputting deep-layer image features, and recording the deep-layer image features as second-stage features;
outputting the second-stage characteristics through an up-sampling network;
performing network iteration on the image super-resolution network according to the third-stage characteristics and a random gradient descent algorithm, and judging that network training is finished when the loss function output is detected to reach an expected value, so as to generate a target super-resolution network;
and acquiring an output result of the target super-resolution network, and recording the output result as a second characteristic value.
Further, the first characteristic value and the second characteristic value are superposed according to a preset weight to generate a target characteristic value:
and splicing the first characteristic value and the second characteristic value in the dimension of the number of channels according to preset weight to generate a target characteristic value.
Further, the converting the target feature value into a second photoacoustic image according to the feature fusion network, where the second photoacoustic image is a high-resolution photoacoustic image, includes:
obtaining a loss function of the feature fusion network, and training a layer-by-layer transposition convolution algorithm according to the loss function; wherein the calculation formula of the loss function is as follows:
v () represents a certain layer of feature value of VGG16, W is a feature value of an input image obtained by VGG16iAn i-th layer feature value representing a feature value of the image. H is a characteristic value obtained by the VGG16 of the real image, HjA j-th layer feature value representing a feature value of the image;
acquiring a trained feature fusion network, and recording as a target fusion network;
and acquiring a second photoacoustic image output by the target fusion network, wherein the second photoacoustic image is a high-resolution photoacoustic image.
Another embodiment of the present invention provides a high-resolution photoacoustic image imaging apparatus, including:
the image preprocessing module is used for acquiring a photoacoustic image with a low sampling rate, preprocessing the photoacoustic image and generating a preprocessed first photoacoustic image;
the first characteristic value processing module is used for respectively inputting the first photoacoustic image into an image restoration network and an image super-resolution network, acquiring a first characteristic value output by the image restoration network and acquiring a second characteristic value output by the image super-resolution network;
the second characteristic value processing module is used for superposing the first characteristic value and the second characteristic value according to preset weight to generate a target characteristic value;
and the image output module is used for converting the target characteristic value into a second photoacoustic image according to the characteristic fusion network, wherein the second photoacoustic image is a high-resolution photoacoustic image.
Another embodiment of the present invention provides an electronic device comprising at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the high resolution photoacoustic image imaging method described above.
Another embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the above-described high resolution photoacoustic image imaging method.
Has the advantages that: the embodiment of the invention can restore the photoacoustic image with low sampling rate to generate the photoacoustic image with high resolution, and can acquire more image details and has good image restoration quality by respectively processing the image restoration network and the image super-resolution network.
Drawings
The invention will be further described with reference to the accompanying drawings and examples, in which:
FIG. 1 is a flow chart of a preferred embodiment of a method for imaging high resolution photoacoustic images of the present invention;
FIG. 2 is a schematic diagram of a generator structure of an image repairing network according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a discriminator structure of an image repairing network according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a network structure of an image super-resolution network according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a network structure of a feature fusion network according to an embodiment of the present invention;
fig. 6 is a functional block diagram of a high-resolution photoacoustic imaging apparatus according to a preferred embodiment of the present invention;
fig. 7 is a schematic diagram of a hardware structure of an electronic device according to a preferred embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and effects of the present invention clearer and clearer, the present invention is described in further detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Embodiments of the present invention will be described below with reference to the accompanying drawings.
Referring to fig. 1, fig. 1 is a flowchart illustrating a photoacoustic imaging method with high resolution according to a preferred embodiment of the photoacoustic imaging method with high resolution according to the present invention. As shown in fig. 1, it includes the steps of:
s100, acquiring a photoacoustic image with a low sampling rate, preprocessing the photoacoustic image, and generating a preprocessed first photoacoustic image;
step S200, inputting the first photoacoustic image into an image restoration network and an image super-resolution network respectively, acquiring a first characteristic value output by the image restoration network, and acquiring a second characteristic value output by the image super-resolution network;
step S300, superposing the first characteristic value and the second characteristic value according to preset weight to generate a target characteristic value;
and S400, converting the target characteristic value into a second photoacoustic image according to the characteristic fusion network, wherein the second photoacoustic image is a high-resolution photoacoustic image.
In specific implementation, the photoacoustic image imaging method with high resolution according to the embodiment of the invention is a hybrid network method combining image restoration and image super-resolution to solve the problem of restoring a photoacoustic image with high resolution.
The method comprises the steps of firstly obtaining a photoacoustic image with a low sampling rate, for example, reserving one pixel in every four adjacent pixels or reserving one pixel in every six adjacent pixels, then preprocessing the photoacoustic image with the low sampling rate, then respectively sending the preprocessed photoacoustic image into an image restoration network and an image super-resolution network to respectively obtain characteristic values output by the two networks, then adding the two characteristic values according to a certain weight to obtain a final characteristic value, and finally converting the characteristic value into an output image by using a characteristic fusion network.
In one embodiment, the image repairing network is a generation countermeasure network, the generation countermeasure network includes a generator and a discriminator, and before the first photoacoustic images are respectively input into the image repairing network, the method includes:
setting a network structure of a generator as a U-Net network, wherein the U-Net network comprises a down-sampling network and an up-sampling network corresponding to the down-sampling network, jump connections are correspondingly arranged between corresponding output characteristic values of the up-sampling network and the down-sampling network, the number of layers of the U-Net network is set to be N, the down-sampling network of a natural number with N being more than or equal to 5 comprises a convolution layer, a regularization layer and an activation function, and the up-sampling network comprises a transposition convolution layer, a regularization layer and an activation function;
the network structure of the discriminator is set as a 4-layer convolutional neural network, and the input of the discriminator is a first photoacoustic image generated by the generator, wherein the generated image corresponds to the generated image.
In specific implementation, as shown in fig. 2, the image inpainting network is a generative countermeasure structure, and includes two parts, namely a generator and an arbiter, which are required to be used in the network training process, but in the overall network, only a part of the network of the generator is effective.
In the training process, the network structure of the generator is a U-Net network, the generator comprises a down-sampling network and an up-sampling network corresponding to the down-sampling network, and meanwhile, in order to solve the problem of network convergence, jump connection is generated between corresponding output characteristic values of the up-sampling network and the down-sampling network, the number of layers of the U-Net network is required to be set according to the size of an actual input picture and the size of an output picture, but more than five layers are required to be ensured so as to ensure the acquisition effect of the U-Net network on deep features of the image; the network structure of the discriminator is a 4-layer convolution network, the input is the image of the generated image generated by the generator and the actual comparison sample, the output value is a similarity matrix, the basic idea is to divide the image into N parts, namely the output is also the N matrix, each part can be considered to be independent, each value represents the similarity of the corresponding part of the two images, and finally the average value of the N similarities can be taken to judge the similarity of the two images in a balanced way.
A generator part: the down sampling network is composed of convolution layer, regularization layer and activation function, and the up sampling network is composed of transposition convolution layer, regularization layer and activation function. The output of each up/down sampling is a matrix of eigenvalues.
The discriminator section shown in fig. 3: the input part is to generate an image and an actual image, and is used as a picture through a splicing operation, for example, two 64 x 64 images are spliced into a 128 x 64 image. And then through four convolution blocks, each convolution block network consists of a convolution layer, a pooling layer and an activation layer, the number of the convolution blocks can be increased or decreased according to the size of an input image, but the number of the convolution blocks cannot be less than 4. The output is a matrix of N x N values.
In one embodiment, before the first photoacoustic images are respectively input to an image inpainting network and first feature values output by the image inpainting network are acquired, the method further includes:
respectively inputting the first photoacoustic images into an image restoration network to obtain the target size of an output image;
and according to the target size of the output image, carrying out pixel zero padding on the first photoacoustic image to generate a first photoacoustic image with the target size.
In the specific implementation, the method is based on a pix2pix (Image-to-Image transformation) network. The low sampling rate image is input first, and then zero padding is performed on pixels according to the size of the image required by output, for example, 64 × 64 of the input image, the output image is required to be 128 × 128, that is, 3 zero-valued pixels need to be padded around each pixel. The general idea of taking the pixel points with zero values as the missing part of the image is the general idea of the part of the network.
In one embodiment, inputting the first photoacoustic image into an image inpainting network, and acquiring a first feature value output by the image inpainting network, comprises:
inputting the first photoacoustic image into an image restoration network, and acquiring an objective function of the image restoration network, wherein the formula of the objective function is as follows:
LGAN(G,N)＝minGmaxD(Ex,y[logD(x,y)]+Ex,z[log(1-D(x,G(x,z)))])
(formula 1)
Where G represents the generator network, D represents the arbiter network, G () represents the generator network output, D () represents the arbiter network output, and E represents the L1 loss function, i.e.
yiIs the target value, f (x)i) Is an estimated value; x represents an input image, y represents a real image, and z represents random noise;
training the image restoration network according to the target function to obtain a trained target image restoration network;
and acquiring a characteristic value output by the target image restoration network and recording the characteristic value as a first characteristic value.
In specific implementation, the generator continuously tries to minimize the objective function in the training process, and the arbiter continuously iterates to maximize the objective function. The objective function is specifically:
LGAN(G,N)＝minGmaxD(Ex,y[logD(x,y)]+Ex,z[log(1-D(x,G(x,z)))])
where G represents the generator network, D represents the arbiter network, G () represents the generator network output, D () represents the arbiter network output, and E represents the L1 loss function, i.e.
yiIs the target value, f (x)i) Are estimated values. x represents an input image, y represents a real image, and z represents random noise, and the function is to improve the robustness of the model. x and random noise z as input to the generator G to obtain a generated image G (x, z), then combining G (x, z) and x based on channel dimensions, and finallyObtaining a prediction probability value as an input of the discriminator D, wherein the prediction probability value represents whether the input is a pair of real images, and the closer the probability value is to 1, the more the discriminator D is sure that the input is a pair of real images; the real image y and the input image x are merged together based on the channel dimension and used as the input of the discriminator D to obtain D (x, y), namely a probability prediction value.
The training target of the discriminator D is to output a small probability value when the input is not a pair of real images (x and G (x, z)), and to output a large probability value when the input is a pair of real images (x and y). The training goal of the generator G is to make the probability value of the output of the discriminator D as large as possible when G (x, z) and x are generated as input to the discriminator D. When the network G generation image G (x, z) can cheat the discriminator D through iterative training by using a random gradient descent method, the network training is completed.
In one embodiment, the image super-resolution network employs a depth residual network structure,
inputting the first photoacoustic image into an image super-resolution network, and acquiring a second characteristic value output by the image super-resolution network, wherein the method comprises the following steps:
inputting the first photoacoustic image into an image super-resolution network, performing convolution calculation on the first photoacoustic image through a down-sampling convolution network, and outputting a first-stage feature;
inputting the first-stage features into W network residual blocks, outputting deep-layer image features, and recording the deep-layer image features as second-stage features;
outputting the second-stage characteristics through an up-sampling network;
performing network iteration on the image super-resolution network according to the third-stage characteristics and a random gradient descent algorithm, and judging that network training is finished when the loss function output is detected to reach an expected value, so as to generate a target super-resolution network;
and acquiring an output result of the target super-resolution network, and recording the output result as a second characteristic value.
In specific implementation, as shown in fig. 4, the image super-resolution network is based on a depth residual error network, and a channel attention mechanism is introduced. The network consists of N residual block networksThe value of N needs to be designed according to the pixel size of an input image, the larger the pixel size is, the larger the required value of N is, and the value of N cannot be smaller than 4; each residual block is composed of three residual channel attention mechanism convolution blocks, jump connection is carried out between the input and the output of the residual block network, and the loss function uses L2 loss function, namely
The attention mechanism is that the feature matrix in each channel and all feature matrices of the previous N channels use a correlation function to calculate the correlation, then the weight w of the feature and the feature matrix of the previous N channels is calculated, and w is multiplied by the feature matrix of the previous N channels and added to the feature value of the feature matrix.
The method comprises the steps of preprocessing an input low-sampling-rate image, reducing network input and output through a down-sampling convolution network, obtaining deep image characteristics through N network residual blocks, and finally obtaining an output image through an up-sampling network (transposition convolution). The network training uses a random gradient descent method to carry out network iteration, and when the output of the L2 loss function reaches an expected value, the network training is finished.
In one embodiment, the first eigenvalue and the second eigenvalue are superimposed according to a preset weight to generate a target eigenvalue:
and splicing the first characteristic value and the second characteristic value in the dimension of the number of channels according to preset weight to generate a target characteristic value.
In specific implementation, as shown in fig. 5, after the image inpainting network and the image super-resolution network are trained, the last step of the method is performed. The input image is a photoacoustic image with a low sampling rate, and in an image restoration network, the input characteristic of a first up-sampling network in a generator network is taken as an input characteristic A; in the image super-resolution network, the input characteristic of the up-sampling network is taken as an input characteristic B. In the stage of network design, the sizes of convolution kernels and pooling need to be adjusted to ensure that the dimension of the input feature A is consistent with that of the input feature B. And splicing the input features A and the input features B in the dimension of the number of channels to be used as the input features of the feature fusion network.
In one embodiment, converting the target feature value into a second photoacoustic image according to a feature fusion network, the second photoacoustic image being a high-resolution photoacoustic image, includes:
obtaining a loss function of the feature fusion network, and training a layer-by-layer transposition convolution algorithm according to the loss function; wherein the calculation formula of the loss function is as follows:
v () represents a certain layer of feature value of VGG16, W is a feature value of an input image obtained by VGG16iAn i-th layer feature value representing a feature value of the image. H is a characteristic value obtained by the VGG16 of the real image, HjA j-th layer feature value representing a feature value of the image; the VGG16 network is a convolutional neural network composed of 13 convolutional layers and 3 fully-connected layers.
Acquiring a trained feature fusion network, and recording as a target fusion network;
and acquiring a second photoacoustic image output by the target fusion network, wherein the second photoacoustic image is a high-resolution photoacoustic image.
In specific implementation, as shown in fig. 5, in order to solve the problem that the edge of the output image is too smooth, the loss function of the partial network adopts a perceptual loss function, that is, a feature matrix obtained by convolution of a real image is compared with a feature matrix obtained by convolution of a generated image, so that more image details can be obtained. In the present method vgg16 is used as a feature matrix for the perceptual loss function. I.e. the loss function is:
v () represents a certain layer of feature value of VGG16, W is a feature value of an input image obtained by VGG16iAn i-th layer feature value representing a feature value of the image. H is true image passing VGCharacteristic value, H, from G16jA j-th layer feature value representing a feature value of the image. And (4) iterating the loss function by using a random gradient descent method, and finishing the training of the whole network when the output value of the loss function reaches a preset value.
It should be noted that, a certain order does not necessarily exist between the above steps, and those skilled in the art can understand, according to the description of the embodiments of the present invention, that in different embodiments, the above steps may have different execution orders, that is, may be executed in parallel, may also be executed interchangeably, and the like.
Another embodiment of the present invention provides a high-resolution photoacoustic image imaging apparatus, as shown in fig. 6, the apparatus 1 including:
the image preprocessing module 11 is configured to acquire a photoacoustic image with a low sampling rate, preprocess the photoacoustic image, and generate a preprocessed first photoacoustic image;
the first feature value processing module 12 is configured to input the first photoacoustic image into an image inpainting network and an image super-resolution network, respectively, acquire a first feature value output by the image inpainting network, and acquire a second feature value output by the image super-resolution network;
the second eigenvalue processing module 13 is configured to superimpose the first eigenvalue and the second eigenvalue according to a preset weight to generate a target eigenvalue;
and the image output module 14 is configured to convert the target feature value into a second photoacoustic image according to the feature fusion network, where the second photoacoustic image is a high-resolution photoacoustic image.
The specific implementation is shown in the method embodiment, and is not described herein again.
Another embodiment of the present invention provides an electronic device, as shown in fig. 7, an electronic device 10 includes:
one or more processors 110 and a memory 120, where one processor 110 is illustrated in fig. 7, the processor 110 and the memory 120 may be connected by a bus or other means, and where fig. 7 illustrates a bus connection.
The processor 110 is used to implement various control logic for the electronic device 10, which may be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a single chip, an ARM (Acorn RISC machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware controls, or any combination of these components. Also, the processor 110 may be any conventional processor, microprocessor, or state machine. Processor 110 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
The memory 120, which is a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions corresponding to the high-resolution photoacoustic image imaging method in the embodiments of the present invention. The processor 110 executes various functional applications and data processing of the device 10, i.e., implements the high-resolution photoacoustic image imaging method in the above-described method embodiments, by executing the nonvolatile software programs, instructions, and units stored in the memory 120.
The memory 120 may include a storage program area and a storage data area, wherein the storage program area may store an application program required for operating the device, at least one function; the storage data area may store data created according to the use of the device 10, and the like. Further, the memory 120 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 120 optionally includes memory located remotely from processor 110, which may be connected to device 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
One or more cells are stored in the memory 120, which when executed by the one or more processors 110, perform the high resolution photoacoustic image imaging method of any of the method embodiments described above, e.g., performing the method steps S100-S400 of fig. 1 described above.
Embodiments of the present invention provide a non-transitory computer-readable storage medium storing computer-executable instructions for execution by one or more processors, e.g., to perform method steps S100-S400 of fig. 1 described above.
By way of example, non-volatile storage media can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in many forms such as Synchronous RAM (SRAM), dynamic RAM, (DRAM), Synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The disclosed memory controls or memories of the operating environments described herein are intended to comprise one or more of these and/or any other suitable types of memory.
Another embodiment of the present invention provides a computer program product comprising a computer program stored on a non-volatile computer readable storage medium, the computer program comprising program instructions which, when executed by a processor, cause the processor to perform the high resolution photoacoustic image imaging method of the above-described method embodiments. For example, the method steps S100 to S400 in fig. 1 described above are performed.
The above-described embodiments are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the embodiment.
Through the above description of the embodiments, those skilled in the art will clearly understand that the embodiments may be implemented by software plus a general hardware platform, and may also be implemented by hardware. Based on such understanding, the above technical solutions essentially or contributing to the related art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of the various embodiments or some parts of the embodiments.
Conditional language such as "can," "might," or "may" is generally intended to convey that a particular embodiment can include (yet other embodiments do not include) particular features, elements, and/or operations, among others, unless specifically stated otherwise or otherwise understood within the context as used. Thus, such conditional language is also generally intended to imply that features, elements, and/or operations are in any way required for one or more embodiments or that one or more embodiments must include logic for deciding, with or without input or prompting, whether such features, elements, and/or operations are included or are to be performed in any particular embodiment.
What has been described herein in the specification and drawings includes examples of photoacoustic image imaging methods and apparatuses capable of providing high resolution. It will, of course, not be possible to describe every conceivable combination of components and/or methodologies for purposes of describing the various features of the disclosure, but it can be appreciated that many further combinations and permutations of the disclosed features are possible. It is therefore evident that various modifications can be made to the disclosure without departing from the scope or spirit thereof. In addition, or in the alternative, other embodiments of the disclosure may be apparent from consideration of the specification and drawings and from practice of the disclosure as presented herein. It is intended that the examples set forth in this specification and the drawings be considered in all respects as illustrative and not restrictive. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.
Claims (10)
1. A method of high resolution photoacoustic image imaging, the method comprising:
acquiring a photoacoustic image with a low sampling rate, preprocessing the photoacoustic image, and generating a preprocessed first photoacoustic image;
respectively inputting the first photoacoustic image into an image restoration network and an image super-resolution network, acquiring a first characteristic value output by the image restoration network, and acquiring a second characteristic value output by the image super-resolution network;
superposing the first characteristic value and the second characteristic value according to a preset weight to generate a target characteristic value;
and converting the target characteristic value into a second photoacoustic image according to the characteristic fusion network, wherein the second photoacoustic image is a high-resolution photoacoustic image.
2. The method according to claim 1, wherein the image repairing network is a generation countermeasure network, the generation countermeasure network comprises a generator and a discriminator, and before the first photoacoustic images are respectively input into the image repairing network and the first characteristic values output by the image repairing network are obtained, the method comprises:
setting a network structure of a generator as a U-Net network, wherein the U-Net network comprises a down-sampling network and an up-sampling network corresponding to the down-sampling network, jump connections are correspondingly arranged between corresponding output characteristic values of the up-sampling network and the down-sampling network, the number of layers of the U-Net network is set to be N, the down-sampling network of a natural number with N being more than or equal to 5 comprises a convolution layer, a regularization layer and an activation function, and the up-sampling network comprises a transposition convolution layer, a regularization layer and an activation function;
the network structure of the discriminator is set as a 4-layer convolutional neural network, and the input of the discriminator is a first photoacoustic image generated by the generator, wherein the generated image corresponds to the generated image.
3. The method according to claim 2, wherein before inputting the first photoacoustic images into the image inpainting network respectively and acquiring the first characteristic values output by the image inpainting network, the method further comprises:
respectively inputting the first photoacoustic images into an image restoration network to obtain the target size of an output image;
and according to the target size of the output image, carrying out pixel zero padding on the first photoacoustic image to generate a first photoacoustic image with the target size.
4. The method according to claim 3, wherein the inputting the first photoacoustic image into an image inpainting network, and acquiring the first feature value output by the image inpainting network comprises:
inputting the first photoacoustic image into an image restoration network, and acquiring an objective function of the image restoration network, wherein the formula of the objective function is as follows:
LGAN(G,N)＝minGmaxD(Ex,y[logD(x,y)]+Ex,z[log(1-D(x,G(x,z)))]) (formula 1)
Where G represents the generator network, D represents the arbiter network, G () represents the generator network output, D () represents the arbiter network output, and E represents the L1 loss function, i.e.
yiIs the target value, f (x)i) Is an estimated value; x represents an input image, y represents a real image, and z represents random noise;
training the image restoration network according to the target function to obtain a trained target image restoration network;
and acquiring a characteristic value output by the target image restoration network and recording the characteristic value as a first characteristic value.
5. The method of claim 4, wherein the image super-resolution network employs a depth residual network structure,
inputting the first photoacoustic image into an image super-resolution network, and acquiring a second characteristic value output by the image super-resolution network, wherein the method comprises the following steps:
inputting the first photoacoustic image into an image super-resolution network, performing convolution calculation on the first photoacoustic image through a down-sampling convolution network, and outputting a first-stage feature;
inputting the first-stage features into W network residual blocks, outputting deep-layer image features, and recording the deep-layer image features as second-stage features;
outputting the second-stage characteristics through an up-sampling network;
performing network iteration on the image super-resolution network according to the third-stage characteristics and a random gradient descent algorithm, and judging that network training is finished when the loss function output is detected to reach an expected value, so as to generate a target super-resolution network;
and acquiring an output result of the target super-resolution network, and recording the output result as a second characteristic value.
6. The method according to claim 5, wherein the first eigenvalue and the second eigenvalue are superimposed according to a preset weight to generate a target eigenvalue:
and splicing the first characteristic value and the second characteristic value in the dimension of the number of channels according to preset weight to generate a target characteristic value.
7. The method of claim 6, wherein the converting the target feature value into a second photoacoustic image according to the feature fusion network, the second photoacoustic image being a high resolution photoacoustic image, comprises:
obtaining a loss function of the feature fusion network, and training a layer-by-layer transposition convolution algorithm according to the loss function; wherein the calculation formula of the loss function is as follows:
v () represents taking a certain layer of characteristic value of VGG16, W is outputThe characteristic value, W, of the input image obtained by VGG16iAn i-th layer feature value representing a feature value of the image. H is a characteristic value obtained by the VGG16 of the real image, HjA j-th layer feature value representing a feature value of the image;
acquiring a trained feature fusion network, and recording as a target fusion network;
and acquiring a second photoacoustic image output by the target fusion network, wherein the second photoacoustic image is a high-resolution photoacoustic image.
8. A high resolution photoacoustic image imaging apparatus, the apparatus comprising:
the image preprocessing module is used for acquiring a photoacoustic image with a low sampling rate, preprocessing the photoacoustic image and generating a preprocessed first photoacoustic image;
the first characteristic value processing module is used for respectively inputting the first photoacoustic image into an image restoration network and an image super-resolution network, acquiring a first characteristic value output by the image restoration network and acquiring a second characteristic value output by the image super-resolution network;
the second characteristic value processing module is used for superposing the first characteristic value and the second characteristic value according to preset weight to generate a target characteristic value;
and the image output module is used for converting the target characteristic value into a second photoacoustic image according to the characteristic fusion network, wherein the second photoacoustic image is a high-resolution photoacoustic image.
9. An electronic device, characterized in that the electronic device comprises at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of imaging a high resolution photoacoustic image of any one of claims 1-7.
10. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the method of imaging a high resolution photoacoustic image of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110779167.2A CN113538238B (en) | 2021-07-09 | 2021-07-09 | High-resolution photoacoustic image imaging method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110779167.2A CN113538238B (en) | 2021-07-09 | 2021-07-09 | High-resolution photoacoustic image imaging method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113538238A true CN113538238A (en) | 2021-10-22 |
| CN113538238B CN113538238B (en) | 2024-06-28 |
Family
ID=78098277
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110779167.2A Active CN113538238B (en) | 2021-07-09 | 2021-07-09 | High-resolution photoacoustic image imaging method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113538238B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103393409A (en) * | 2013-08-22 | 2013-11-20 | 南京大学 | Comprehensive analysis method of multi-dimensional biology opto-acoustic image spectra |
| CN106709875A (en) * | 2016-12-30 | 2017-05-24 | 北京工业大学 | Compressed low-resolution image restoration method based on combined deep network |
| US20190130217A1 (en) * | 2018-11-15 | 2019-05-02 | Chyuan-Tyng Wu | Trainable vision scaler |
| CN110827213A (en) * | 2019-10-11 | 2020-02-21 | 西安工程大学 | Super-resolution image restoration method based on generation type countermeasure network |
| CN111080527A (en) * | 2019-12-20 | 2020-04-28 | 北京金山云网络技术有限公司 | Image super-resolution method and device, electronic equipment and storage medium |
| CN111754403A (en) * | 2020-06-15 | 2020-10-09 | 南京邮电大学 | Image super-resolution reconstruction method based on residual learning |
| WO2021056969A1 (en) * | 2019-09-29 | 2021-04-01 | 中国科学院长春光学精密机械与物理研究所 | Super-resolution image reconstruction method and device |
| US20210125313A1 (en) * | 2019-10-25 | 2021-04-29 | Samsung Electronics Co., Ltd. | Image processing method, apparatus, electronic device and computer readable storage medium |
-
2021
- 2021-07-09 CN CN202110779167.2A patent/CN113538238B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103393409A (en) * | 2013-08-22 | 2013-11-20 | 南京大学 | Comprehensive analysis method of multi-dimensional biology opto-acoustic image spectra |
| CN106709875A (en) * | 2016-12-30 | 2017-05-24 | 北京工业大学 | Compressed low-resolution image restoration method based on combined deep network |
| US20190130217A1 (en) * | 2018-11-15 | 2019-05-02 | Chyuan-Tyng Wu | Trainable vision scaler |
| WO2021056969A1 (en) * | 2019-09-29 | 2021-04-01 | 中国科学院长春光学精密机械与物理研究所 | Super-resolution image reconstruction method and device |
| CN110827213A (en) * | 2019-10-11 | 2020-02-21 | 西安工程大学 | Super-resolution image restoration method based on generation type countermeasure network |
| US20210125313A1 (en) * | 2019-10-25 | 2021-04-29 | Samsung Electronics Co., Ltd. | Image processing method, apparatus, electronic device and computer readable storage medium |
| CN111080527A (en) * | 2019-12-20 | 2020-04-28 | 北京金山云网络技术有限公司 | Image super-resolution method and device, electronic equipment and storage medium |
| CN111754403A (en) * | 2020-06-15 | 2020-10-09 | 南京邮电大学 | Image super-resolution reconstruction method based on residual learning |
Non-Patent Citations (2)
| Title |
|---|
| WENMING YANG ET AL.: "Lightweight Feature Fusion Network for Single Image Super-Resolution", IEEE SIGNAL PROCESSING LETTERS, vol. 26, no. 4, pages 538 - 542, XP011711789, DOI: 10.1109/LSP.2018.2890770 * |
| 丁明航等: "基于注意力生成对抗网络的图像超分辨率重建方法", 计算机系统应用, vol. 29, no. 2, pages 205 - 211 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113538238B (en) | 2024-06-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Qiu et al. | Multiple improved residual networks for medical image super-resolution | |
| CN115953494B (en) | Multi-task high-quality CT image reconstruction method based on low dose and super resolution | |
| Cheng et al. | High-resolution photoacoustic microscopy with deep penetration through learning | |
| CN113989129A (en) | Image restoration method based on gating and context attention mechanism | |
| CN113538616B (en) | Magnetic resonance image reconstruction method combining PUGAN with improved U-net | |
| He et al. | Remote sensing image super-resolution using deep–shallow cascaded convolutional neural networks | |
| Du et al. | Blind image denoising via dynamic dual learning | |
| CN116612174A (en) | Three-dimensional reconstruction method and system for soft tissue and computer storage medium | |
| CN116823625A (en) | Cross-contrast magnetic resonance super-resolution method and system based on variational self-encoder | |
| CN116309278A (en) | Medical image segmentation model and method based on multi-scale context awareness | |
| Kim et al. | Deep learning alignment of bidirectional raster scanning in high speed photoacoustic microscopy | |
| Cheng et al. | Prior guided fundus image quality enhancement via contrastive learning | |
| CN114066729A (en) | Face super-resolution reconstruction method capable of recovering identity information | |
| CN116071239B (en) | CT image super-resolution method and device based on mixed attention model | |
| CN112767403A (en) | Medical image segmentation model training method, medical image segmentation method and device | |
| Tang et al. | Learning spatially variant degradation for unsupervised blind photoacoustic tomography image restoration | |
| CN116128854B (en) | Hip joint ultrasonic image quality assessment method based on convolutional neural network | |
| CN113538238B (en) | High-resolution photoacoustic image imaging method and device and electronic equipment | |
| CN116563554A (en) | Low-dose CT image denoising method based on hybrid characterization learning | |
| Qiu et al. | Cardiac Magnetic Resonance Images Superresolution via Multichannel Residual Attention Networks | |
| CN116757982A (en) | Multi-mode medical image fusion method based on multi-scale codec | |
| Zhang et al. | Hformer: highly efficient vision transformer for low-dose CT denoising | |
| Hakimnejad et al. | Unsupervised photoacoustic tomography image reconstruction from limited-view unpaired data using an improved CycleGAN | |
| CN112529949A (en) | Method and system for generating DWI image based on T2 image | |
| Liu et al. | A multi-view assisted registration network for MRI registration pre-and post-therapy |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |