CN115131452A

CN115131452A - Image processing method and device for artifact removal

Info

Publication number: CN115131452A
Application number: CN202210408968.2A
Authority: CN
Inventors: 王红; 李悦翔; 郑冶枫; 孟德宇
Original assignee: Tencent Healthcare Shenzhen Co Ltd
Current assignee: Tencent Healthcare Shenzhen Co Ltd
Priority date: 2022-04-19
Filing date: 2022-04-19
Publication date: 2022-09-30
Also published as: WO2023202265A1

Abstract

Embodiments of the present disclosure provide an image processing method, apparatus, computer program product, and storage medium for artifact removal. The image processing method of the present disclosure includes: establishing a training data set for training a neural network, utilizing an adaptive convolution dictionary network to perform artifact removal processing on the image with the artifact to obtain a processed image, and performing iterative training on the adaptive convolution dictionary network based on the artifact-free image, the processed image and an objective function processed by the image mask to optimize network parameters of the adaptive convolution dictionary network. The image processing method can simply and effectively remove the artifacts in the image so as to obtain a clearer image without being interfered by the artifacts.

Description

Image processing method and device for artifact removal

Technical Field

The present disclosure relates to the field of artificial intelligence, and more particularly, to an image processing method, apparatus, computer program product, and storage medium for artifact removal.

Background

Images have a great weight in various information bases as one of the most important sources for human to acquire information. With the rapid development of computer technology, image processing has been widely applied to various aspects of human social life, such as: industrial inspection, medicine, intelligent robots, etc. The image is often applied to various fields to describe and express the characteristics and logical relations of objects with vividness and intuition, and the application range is wide, so the development of the image processing technology and the information processing of various fields are very important.

The image processing technology is a technology for processing image information by a computer. The method mainly comprises the steps of image digitization, image enhancement and restoration, image data coding, image segmentation, image identification and the like. Among them, the image restoration technique aims to restore a degraded image to the original true face as much as possible. For example, the method is used in the field of removing artifacts in images, such as removing noise in images, removing raindrops in images with rain taken in rainy days, removing metal artifacts in CT images, and the like. However, the current image processing method faces technical bottlenecks such as complex mathematical model, limited application range, and difficulty in acquiring partial data when removing artifacts.

Therefore, there is a need for an image processing method that can be widely and effectively applied to identify artifacts in an image and remove the artifacts from the image with the artifacts, so as to help people obtain a clearer image without being disturbed by the artifacts.

Disclosure of Invention

In order to solve the above problems, the present disclosure provides an image processing method, apparatus, computer program product, and storage medium for artifact removal. The image processing method for artifact removal provided by the present disclosure includes: establishing a training data set for training a neural network, wherein the training data set comprises a plurality of groups of image samples, and each group of image samples comprises an image (Y) with an artifact, a non-artifact image (X) corresponding to the image (Y) with the artifact and an image mask (I); for at least one of the sets of image samples, performing an artifact removal process on the artifact-bearing image (Y) using an adaptive convolution dictionary network to obtain a processed image, iteratively training the adaptive convolution dictionary network based on the artifact-free image (X) and the processed image and an objective function processed by the image mask (I) to optimize network parameters of the adaptive convolution dictionary network, wherein the adaptive convolution dictionary network comprises a basic artifact dictionary that is a sample-invariant convolution dictionary and comprises a first number of convolution kernels, and determining a second number of adaptive convolution kernels for the image samples by a plurality of artifact convolution kernels in the basic artifact dictionary and sample-variant weighting coefficients, wherein, determining an artifact image in the artifact image by convolution of the second number of adaptive convolution kernels with image features of the artifact image, and removing the artifact image from the artifact image to obtain the processed image.

The image processing method can simply and effectively remove the artifacts in the image so as to obtain a clearer image without artifact interference.

According to the embodiment of the disclosure, the first number of artifact convolution kernels indicates an artifact pattern, and the image feature indicates a position of the artifact pattern, wherein the adaptive convolution dictionary network includes a T-level network, wherein in the T-level network, a weighting coefficient and an image feature output by the T-1-level network are updated by using an iterative update rule based on a near-end gradient descent to obtain a weighting coefficient and an image feature of the T-level network, where T is an integer greater than 1 and less than or equal to T.

According to the embodiment of the disclosure, each stage of network comprises a weighting coefficient updating network, an image characteristic updating network and an artifact removing image updating network, wherein the weighting coefficient updating network, the image characteristic updating network and the artifact removing image updating network comprise a residual error network structure and a normalization processing layer.

According to an embodiment of the present disclosure, the establishing a training data set for training a neural network further comprises at least one of: -normalizing the pixel values of said artifact-bearing image (Y); and randomly cutting the artifact image (Y) to obtain an image block, and randomly turning the image block according to a preset probability.

According to an embodiment of the present disclosure, the objective function is a loss objective function constructed based on the artifact-free image (X) and the processed image, wherein iteratively training the adaptive convolutional dictionary network based on the artifact-free image (X), the processed image and the objective function to optimize network parameters of the adaptive convolutional dictionary network further comprises: calculating a loss objective function, reversely transmitting the result to the Adaptive convolution dictionary network, and optimizing network parameters of the Adaptive convolution dictionary network based on an Adaptive moment estimation (Adam) algorithm.

According to an embodiment of the present disclosure, an image processing method further includes: after training is completed, testing the adaptive convolution dictionary network, wherein the testing the adaptive convolution dictionary network comprises: preprocessing an image with an artifact to be tested and inputting the preprocessed image into the self-adaptive convolution dictionary network; and processing the image with the artifact to be tested by utilizing an adaptive convolution dictionary network to obtain a processed image with the artifact removed.

According to the embodiment of the disclosure, the artifacts are metal artifacts, the artifact-carrying images are CT images with metal artifacts, the training data set includes a plurality of sets of CT image samples, and each set of image samples includes a CT image with metal artifacts, and a CT image without metal artifacts and a non-metal area mask corresponding thereto.

According to the embodiment of the disclosure, each stage of network comprises a weighting coefficient updating network, a metal artifact image feature updating network and a metal artifact removing image updating network, wherein the weighting coefficient updating network, the metal artifact image feature updating network and the metal artifact removing image updating network comprise a residual error network structure; and the weighting coefficient update network includes: a Linear layer, a modified Linear Unit (ReLU) layer, a cross-link layer, and a Batch Normalization (BN) layer; the metal artifact image feature updating network comprises: convolutional layer, BN layer, ReLU layer, and cross-link layer; the metal artifact removal image update network comprises: convolutional layers, BN layers, ReLU layers, and cross-link layers.

An embodiment of the present disclosure further provides an image processing method for artifact removal, including: acquiring an input image to be processed; processing the input image by using an adaptive convolutional dictionary network to obtain a processed image with artifact removed, wherein the adaptive convolutional dictionary network is trained on the basis of an artifact database (D) and comprises a T-level network, and the first-level network obtains a 1 st-level image feature and a first-level artifact removed image which are output by the first-level network on the basis of the input image; the T-level network obtains and outputs the T-level image characteristic and the T-level artifact removed image output by the T-level network at least partially based on the T-1-level image characteristic and the T-1-level artifact removed image output by the T-level network, wherein T is greater than 1 and less than or equal to T; and the T-level network obtains and outputs the T-level artifact removed image output by the T-level network as the processed image for removing the artifact based on the T-1 level image feature and the T-1 level artifact removed image output by the T-level network.

According to an embodiment of the present disclosure, the adaptive convolution dictionary network includes a basic artifact dictionary that is a convolution dictionary that does not vary with an input image and includes a first number of artifact convolution kernels, a t-th order weighting coefficient for a t-th order network is determined, a second number of adaptive convolution kernels for the t-th order network is determined by a plurality of artifact convolution kernels in the basic artifact dictionary and the t-th order weighting coefficient; and determining a t-th level artifact-removed image based on the second number of adaptive convolution kernels and image characteristics of the t-th level network.

An embodiment of the present disclosure provides an image processing apparatus for artifact removal, including: a training data set establishment module configured to: establishing a training data set for training a neural network, wherein the training data set comprises a plurality of groups of image samples, and each group of image samples comprises an image (Y) with an artifact, a non-artifact image (X) corresponding to the image (Y) with the artifact and an image mask (I); an adaptive convolutional dictionary network configured to: for at least one of the groups of image samples, performing artifact removal processing on the artifact-bearing image (Y) to obtain a processed image; a training module configured to: iteratively training the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and an objective function processed by the image mask (I), to optimize network parameters of the adaptive convolutional dictionary network; wherein the adaptive convolution dictionary network includes a base artifact dictionary that is a sample-invariant convolution dictionary and includes a first number of artifact convolution kernels, and a second number of adaptive convolution kernels for the image samples are determined by a plurality of artifact convolution kernels in the base artifact dictionary and sample-variant weighting coefficients, wherein an artifact image in the artifact-bearing image is determined by convolution of the second number of adaptive convolution kernels with image features of the artifact-bearing image, and the artifact image is removed from the artifact-bearing image to yield the processed image.

An embodiment of the present disclosure provides an image processing apparatus for artifact removal, including: an image acquisition module configured to: acquiring an input image to be processed; an image processing module configured to: processing the input image by using an adaptive convolution dictionary network to obtain a processed image with artifacts removed; wherein the adaptive convolutional dictionary network is trained based on an artifact database (D) and comprises a T-level network, wherein the first-level network obtains a level 1 image feature and a first-level artifact-removed image output by the first-level network based on the input image; the T-level network obtains and outputs the T-level image characteristic and the T-level artifact removed image output by the T-level network at least partially based on the T-1-level image characteristic and the T-1-level artifact removed image output by the T-level network, wherein T is greater than 1 and less than or equal to T; and the T-level network obtains and outputs the T-level artifact removed image output by the T-level network as the processed image for removing the artifact based on the T-1 level image feature and the T-1 level artifact removed image output by the T-level network.

Embodiments of the present disclosure provide a computer program product comprising computer software code which, when executed by a processor, provides the above method.

Embodiments of the present disclosure provide a computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, provide the above-described method.

The image processing method can perform artifact removal processing only based on the image to be processed without additionally acquiring the chord graph of the image. The self-adaptive convolution dictionary network fully utilizes the prior structure of the artifact image, the artifact removing effect is better, and the model generalization performance is stronger. In addition, the mathematical model adopted by the image processing method disclosed by the invention has clear physical meaning and strong interpretability, and the physical meaning of each network module is more definite, so that the image processing method is convenient for the understanding and application of technicians in the field. The image processing method can simply and effectively remove the artifacts in the image so as to obtain a clearer image without being interfered by the artifacts.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings that are required to be used in the description of the embodiments will be briefly described below. It is apparent that the drawings in the following description are only exemplary embodiments of the disclosure, and that other drawings may be derived from those drawings by a person of ordinary skill in the art without inventive effort.

Herein, in the drawings:

1A-1B are schematic diagrams illustrating an artifact-bearing image and an artifact-free image, respectively, according to embodiments of the present disclosure;

2A-2B are schematic diagrams illustrating a chordal graph-based image processing method according to an embodiment of the disclosure;

3A-3B are schematic flow diagrams illustrating an image processing method for artifact removal according to an embodiment of the present disclosure;

FIG. 4 is an exemplary diagram illustrating a weighted adaptive convolution dictionary based image processing model according to an embodiment of the present disclosure;

5A-5B are schematic diagrams illustrating network iterative update processes at various levels in an adaptive convolutional dictionary network according to embodiments of the present disclosure;

FIG. 6 is a schematic flow chart diagram illustrating an image processing procedure for artifact removal according to an embodiment of the present disclosure;

7A-7B are component schematic diagrams illustrating an image processing apparatus for artifact removal according to an embodiment of the present disclosure;

FIG. 8 is an architecture illustrating a computing device according to an embodiment of the present disclosure; and

fig. 9 is a schematic diagram illustrating a storage medium according to an embodiment of the present disclosure.

Detailed Description

In order to make the objects, technical solutions and advantages of the present disclosure more apparent, example embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. It is to be understood that the described embodiments are merely a subset of the embodiments of the present disclosure and not all embodiments of the present disclosure, with the understanding that the present disclosure is not limited to the example embodiments described herein.

Further, in the present specification and the drawings, steps and elements having substantially the same or similar characteristics are denoted by the same or similar reference numerals, and repeated description of the steps and elements will be omitted.

Furthermore, in the specification and drawings, elements are described in the singular or plural according to the embodiments. However, the singular and plural forms are appropriately selected for the proposed cases only for convenience of explanation and are not intended to limit the present disclosure thereto. Thus, the singular may include the plural and the plural may also include the singular, unless the context clearly dictates otherwise.

In the present specification and the drawings, steps and elements having substantially the same or similar characteristics are denoted by the same or similar reference numerals, and repeated description of the steps and elements will be omitted. Also, in the description of the present disclosure, the terms "first," "second," and the like are used solely to distinguish one from another, and are not to be construed as indicating or implying a relative importance or order.

For the purpose of describing the present disclosure, concepts related to the present disclosure are introduced below.

The methods of the present disclosure may be Artificial Intelligence (AI) based. Artificial intelligence is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. For example, for artificial intelligence based methods, it is possible to perform machine learning in a manner similar to human perception, such as by training neural networks to extract image information, perform image analysis and processing.

The image processing technology is a technology for processing image information by a computer. The method mainly comprises the following steps: image digitization, image enhancement and restoration, image data encoding, image segmentation, image recognition, and the like. Image processing technology is currently implemented mainly on a computer basis. An Artificial Neural Network (ANN) is an algorithmic mathematical model that simulates the structure and behavior of a biological nervous system and performs distributed parallel information processing. The ANN achieves the purpose of processing information by adjusting the weight relation between the internal neurons and the neurons. When processing images, the computer takes the original image or the image which is processed properly as the input signal of the neural network, and the processed image signal or the classification result is obtained at the output end of the neural network.

In summary, the embodiments of the present disclosure provide solutions related to computer technologies such as artificial intelligence, image processing, neural network, and the like, and will be further described with reference to the accompanying drawings.

Taking a CT image as an example, fig. 1A is a schematic diagram illustrating an artifact-bearing image according to an embodiment of the present disclosure.

X-ray Computed Tomography (CT) has been widely used for clinical diagnosis. However, during the imaging process, the loss of projection data is usually caused by the presence of metal implants in the patient, such as dental fillings and hip prostheses, thereby causing severe metal artifacts to appear in the reconstructed CT images. As shown in fig. 1A, the structure of the metal artifact is a set of bar-shaped shadows with similar morphology. The thickness and brightness of the strip-shaped shadows can be different for different tissue structures and/or metal implants. As can be seen from fig. 1A, the existence of the metal artifact makes the CT image blurred, which seriously interferes with the clear presentation of the detail image of the patient body in the CT image, and is not easy for the doctor to make a correct judgment according to the CT image.

Therefore, there is a need for an image processing method capable of effectively identifying artifacts in an image and removing the artifacts from an artifact-containing image, so that the processed artifact-free image is as shown in fig. 1B. A more accurate diagnosis can be made by the physician, for example, by a sharp, undisturbed CT image as shown in fig. 1B.

FIG. 2A shows a schematic diagram of a chordal graph-based image processing method according to an embodiment of the disclosure.

As shown in fig. 2A, DuDoNet + + is a joint learning scheme based on CT images and chord graphs, and uses two network modules SE-Net and IE-Net to process CT images with metal artifacts together, where SE-Net is the network module for chord graphs and IE-Net is the network module for CT images. In FIG. 2A, S _ma Chord graph, S, representing contamination by metal _se Shows the chord graph after the repair enhancement, X _se Is represented by S _se A CT image obtained after back projection transformation (RIL), wherein M represents the shape characteristics (Mask) of metal in the CT image domain _p Represents the shape characteristics of the metal of the chord graph domain obtained by radon transform (FP) of M, X _ma Representing CT images with metal artefacts, X _out Representing the output reconstructed image. As can be seen from fig. 2A, the CT image with metal artifacts needs to be processed by the chord domain (using SE-Net) and the CT image domain (using IE-Net) together to achieve the purpose of image deghost, where the transformation between the chord domain and the CT image domain is converted by the differentiable radon transform layer.

Similarly, fig. 2B shows a schematic diagram of another chord graph-based image processing method according to an embodiment of the present disclosure.

As shown in fig. 2B, the DSCMAR is also a joint processing scheme based on a CT image and a chord graph, and is different from DuDoNet + +, in the scheme, a relatively clean repaired image is obtained by using PriorNet, then the chord graph is further modified and enhanced by using SinoNet, and finally, a reconstructed CT image is obtained by filtering a post-projection layer (FBP) conversion. In FIG. 2B, S _ma Chord graph, T, showing contamination by metal _r Chord graphs representing features of metals, S _LI Representing a chord graph, X, after linear interpolation _LI Is represented by S _LI CT image, X, obtained after back-projection transformation _ma Representing CT images with metal artefacts, X _prior Represents the initial repair CT image after PriorNet treatment, S _prior Is represented by X _prior Initial restored chord chart obtained after forward projection transformation, S _res Represents passing through the pair S _LI And S _prior Chord graph obtained by difference, S _corr Represents a further modified enhanced chord graph after SinoNet treatment by adding S _corr And performing filtering back projection transformation to obtain an image without metal artifacts.

The chord graph-based image processing method shown in fig. 2A and 2B can realize image processing for artifact removal as illustrated in fig. 1A and 1B. However, the limitations shared by DuDoNet + + and DSCMAR are: in both of these solutions, the chord chart information used is difficult to obtain in practice and usually needs to be provided by the equipment manufacturer. Moreover, the designed network does not embed well the a priori information specific to the Metal Artifacts Reduction (MAR) task, and the generalization capability of the network model is limited. In addition, each network module included in these two schemes is physically less interpretable and not readily understood and used by those skilled in the art.

In view of these problems, the present disclosure provides an image processing method for artifact removal, which can perform reconstruction processing based on an artifact-containing image only, and overcome the problem of difficulty in obtaining chord graph data. In addition, the image processing method utilizes a specific weighted convolution dictionary model to encode the prior structure of the artifact, so that the network is more reliable and the generalization capability is stronger. Furthermore, the adaptive convolutional dictionary network model of the present disclosure has clear physical interpretability, which is easy for those skilled in the art to understand and use.

FIG. 4 is an exemplary diagram of an image processing model based on a weighted adaptive convolution dictionary in accordance with an embodiment of the present disclosure.

For a CT image with metal artifacts, its non-metal regions can be decomposed into the following models:

I⊙Y＝I⊙X+I⊙A， (1)

wherein, the first and the second end of the pipe are connected with each other,

is a CT image with metal artifacts; h and W are the height and width of the image, respectively; x is a clean CT image to be restored; i is a non-metal region mask, the elements of which are {0, 1}, wherein 1 represents a non-metal region; a is a metal artifact. According to an embodiment of the present disclosure, the non-metal region mask I may be known, which is used to focus the mathematical model of the solution on the non-metal domain without paying attention to the solution of the metal domain.

It should be appreciated that for different CT images with metal artifacts, the metal artifacts a contained therein substantially exhibit a common or similar pattern, i.e., a morphologically similar stripe structure. Meanwhile, due to the mutual influence of the normal tissues of the human body and the metal artifacts, the artifact patterns contained in different CT images with the metal artifacts have some specific properties, such as pixel intensity.

In view of the above-mentioned characteristics of CT images with metal artifacts, according to the embodiment of the present disclosure, for each artifact-bearing image (Y) to be processed, by using a basic artifact dictionary learned in advance from existing samples

To construct an adaptive convolution kernel for the image

And extracting artifact characteristics of the image

An adaptive convolution kernel that can then be based on the image

And artifact characteristics

And obtaining the metal artifact A of the image, thereby realizing the extraction of the metal artifact. It should be appreciated that the basic artifact dictionary

The method may be an existing known basic artifact dictionary, or may be a basic artifact dictionary obtained by training and synthesizing existing samples.

According to the embodiment of the present disclosure, a weighted convolutional dictionary model may be adopted to code and model the metal artifact a:

wherein the content of the first and second substances,

is a sample-invariant dictionary, which comprises d convolution kernels and generationsA common pattern of different metal artifacts is indicated, in short,

a common database representing different metal artifact types in all CT images with metal artifacts;

is a weighting coefficient that varies with the sample;

representing a particular convolution kernel that represents a pattern of repeated occurrences of a metal artifact, p × p being the size of the convolution kernel;

is a feature layer, representing the position where the local pattern repeats; n is the number of true specific convolution kernels used to encode A;

is a two-dimensional plane convolution operation, and p and d are both positive integers.

In addition, in the case of the present invention,

and is

That is, the base artifact dictionary is a convolution dictionary that does not vary with sample

And includes a first number (d) of artifact convolution kernels. A second number of adaptive convolution kernels for the image samples can be determined from a plurality of artifact convolution kernels in a basic artifact dictionary and a sample-dependent weighting factor (K)

Substituting equation (2) into equation (1) can obtain a model of the non-metal region corresponding to the final CT image with metal artifacts as follows:

fig. 5A shows a schematic structure of an adaptive convolutional dictionary network, including a T-level network, according to an embodiment of the present disclosure. In each level of network, weighting coefficient K and characteristic layer are respectively matched

And the clean CT image X. Fig. 5B shows a schematic structure of a per-level network according to an embodiment of the present disclosure.

An image processing method according to an embodiment of the present disclosure will be described first with reference to fig. 4, 5A, and 5B, and then a mathematical model of the adaptive convolution dictionary network of fig. 5A and 5B will be described.

Fig. 3B is a schematic flow chart diagram 320 illustrating a model usage process of an image processing method for artifact removal according to an embodiment of the present disclosure.

In step S321, an input image to be processed is acquired.

According to an embodiment of the present disclosure, the input image to be processed may be an image obtained after applying its corresponding image mask (I) to the original CT image, i.e., I ≧ Y. Alternatively, the input image to be processed may be an image with an artifact, or may be an image with an artifact that has been subjected to preprocessing (e.g., denoising processing, normalization processing). It is noted that the image mask (I) in the present disclosure is a mask of the region to be studied or the region of interest.

In step S322, the input image is processed by using an adaptive convolutional dictionary network to obtain a processed image with artifacts removed.

Wherein the adaptive convolutional dictionary network is trained based on an artifact database (D) and comprises a T-level network (as shown in fig. 5A), wherein the first-level network obtains a level 1 image feature and a first-level artifact-removed image output by the first-level network based on the input image; the T-level network obtains and outputs the T-level image characteristic and the T-level artifact removed image output by the T-level network at least partially based on the T-1-level image characteristic and the T-1-level artifact removed image output by the T-level network, wherein T is greater than 1 and less than or equal to T; and the T-level network obtains and outputs the T-level artifact removed image output by the T-level network as the processed image for removing the artifact based on the T-1 level image feature and the T-1 level artifact removed image output by the T-level network. The processed image with the artifact removed can be output in a display mode or can be output to a user in a film mode, a drawing mode and the like.

According to an embodiment of the present disclosure, the adaptive convolution dictionary network includes a basic artifact dictionary which is a convolution dictionary that does not vary with an input image and includes a first number of artifact convolution kernels, in a t-th-order network, a t-th-order weighting coefficient of the t-th-order network is first determined, and then a second number of adaptive convolution kernels for the t-th-order network are determined by the first number of artifact convolution kernels in the basic artifact dictionary and the t-th-order weighting coefficient; a t-th level artifact-removed image is then determined based on the second number of adaptive convolution kernels and image features of the t-th level network.

According to the embodiment of the disclosure, each stage of network may include a weighting coefficient updating network, an image feature updating network, and an artifact-removed image updating network, wherein the weighting coefficient updating network, the image feature updating network, and the artifact-removed image updating network include a residual error network structure and a normalization processing layer. It should be understood that the network structure of the weighting coefficient update network, the image feature update network, and the artifact removal image update network may be varied, and the network structure is generally related to the dimensional features of the solution variables corresponding to the network. For example, the weighting coefficient update network solution variables are coefficients and therefore typically include linear layers, while the image feature update network and artifact removal image update network solution variables are two-dimensional images and therefore typically include convolutional layers.

According to the embodiment of the present disclosure, the artifact may be a metal artifact, the input image to be processed may be a CT image with a metal artifact, and the image mask (I) may be a non-metal region mask corresponding to the CT image with the metal artifact. Under this embodiment, each stage of network may include a weighting coefficient update network, a metal artifact image feature update network, and a metal artifact removed image update network, where the weighting coefficient update network, the metal artifact image feature update network, and the metal artifact removed image update network include a residual network structure; and the weighting coefficient update network includes: a Linear layer, a modified Linear Unit (ReLU) layer, a cross-link layer, and a Batch Normalization (BN) layer; the metal artifact image feature updating network comprises: convolutional layer, BN layer, ReLU layer, and cross-link layer; the metal artifact removal image update network includes: convolutional layers, BN layers, ReLU layers, and cross-link layers.

Fig. 3A is a schematic flow diagram 310 illustrating a neural network training process of an image processing method for artifact removal according to an embodiment of the present disclosure.

In step S311, a training data set for training a neural network is established, wherein the training data set includes a plurality of sets of image samples, and each set of image samples includes an artifact-containing image (Y) and a non-artifact image (X) and an image mask (I) corresponding thereto.

According to the embodiment of the disclosure, the disclosed image library and different types of metal masks (masks) can be used for synthesizing the artifacts according to the data simulation flow and enabling the images with the artifacts and the metal masks (M) corresponding to the images with the artifacts _e ) As training data. For example, in an application scenario of removing artifacts for a CT image with metal artifacts, a published deep image library and different types of metal masks may be used to synthesize the metal artifacts according to a data simulation procedure, and use the CT image with metal artifacts and the metal masks as training data. It should be understood that the different types of masks (masks) or different types of metal masks used to set the size and shape of the implant in the CT image can be simulated to obtain their corresponding pseudo-masks based on the size and shape of the implantThe shape of the shadow. Metal mask (M) _e ) Corresponding to the non-metal region mask (I), can be deduced from each other, i.e. M _e +I＝1。

According to an embodiment of the present disclosure, in order to process sample data of a larger data range, the establishing a training data set for training a neural network may further include: -normalizing the pixel values of said artifact-bearing image (Y).

Furthermore, in order to obtain diversity of sample data, the establishing a training data set for training a neural network may further comprise: and randomly cutting the image (Y) with the artifact to obtain an image block, and randomly turning the image block according to a preset probability.

For example, the range of values of the artifact-bearing image in the training data may be clipped (e.g., to remove values or negative values that are too large such that the range of values that are not needed is no longer preserved, but the needed tissue information is not lost), and then normalized such that the pixel values of the image are stuck within the threshold [0,1 ]. Optionally, the normalized data may be reconverted to the [0,255] range to facilitate computer processing.

According to the embodiment of the present disclosure, each training image and the corresponding mask may also be randomly cropped to form smaller image blocks (for example, image blocks that may be 64 × 64 pixels in size), and then random horizontal mirror flipping and random vertical mirror flipping may be performed with a predetermined probability (for example, may be 0.5), respectively, to obtain more diversified training sample data.

In step S312, for at least one of the image samples in the multiple groups of image samples, an adaptive convolutional dictionary network is used to perform artifact removal processing on the artifact-carrying image (Y) to obtain a processed image.

According to an embodiment of the present disclosure, as shown in fig. 4, the adaptive convolution dictionary network may include a base artifact dictionary that is a convolution dictionary that does not vary with samples and includes a first number of artifact convolution kernels, and a second number of adaptive convolution kernels for the image samples may be determined by a plurality of artifact convolution kernels in the base artifact dictionary and weighting coefficients that vary with samples. Moreover, an artifact image of the artifact images may be determined by convolution of the second number of adaptive convolution kernels with an image feature of the artifact image, and the artifact image may be removed from the artifact image to obtain the processed image.

In step S313, iteratively training the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and the objective function processed by the image mask (I), so as to optimize network parameters of the adaptive convolutional dictionary network.

According to an embodiment of the present disclosure, the objective function may be a loss objective function constructed based on the artifact-free image (X) and the processed image, wherein iteratively training the adaptive convolutional dictionary network based on the artifact-free image (X), the processed image and the objective function to optimize network parameters of the adaptive convolutional dictionary network further includes: calculating a loss objective function, reversely transmitting the result to the Adaptive convolution dictionary network, and optimizing network parameters of the Adaptive convolution dictionary network based on an Adaptive moment estimation (Adam) algorithm.

According to an embodiment of the present disclosure, the objective function may also be a loss objective function constructed based on the artifact-free image (X) and the processed image and processed by an image mask (I).

According to the embodiment of the disclosure, the first number of artifact convolution kernels indicates an artifact pattern, and the image feature indicates a position of the artifact pattern, wherein the adaptive convolution dictionary network includes a T-level network, wherein in the T-level network, a weighting coefficient and an image feature output by the T-1-level network are updated by using an iterative update rule based on near-end gradient descent to obtain a weighting coefficient and an image feature of the T-level network, where T is an integer greater than 1 and less than or equal to T

According to an embodiment of the present disclosure, the image processing method for artifact removal may further include: and after the training is finished, testing the self-adaptive convolution dictionary network to evaluate the image processing effect. Wherein the testing the adaptive convolutional dictionary network comprises: preprocessing an image with an artifact to be tested and inputting the preprocessed image into the self-adaptive convolution dictionary network; and processing the image with the artifact to be tested by utilizing an adaptive convolution dictionary network to obtain a processed image with the artifact removed.

According to an embodiment of the present disclosure, the artifact may be a metal artifact, and the artifact-bearing image may be a CT image with a metal artifact. Therefore, the training data set comprises a plurality of sets of CT image samples, each set of image samples comprising a CT image with metal artifacts and a CT image without metal artifacts corresponding thereto; the adaptive convolution dictionary network includes a base metal artifact dictionary that is a metal artifact convolution dictionary that does not vary with CT image samples and includes a first number of metal artifact convolution kernels, and a second number of adaptive convolution kernels for the CT image samples are determined by a plurality of metal artifact convolution kernels in the base metal artifact dictionary and weighting coefficients that vary with CT image samples, wherein the first number of metal artifact convolution kernels indicates a metal artifact pattern and the metal artifact image features indicate where the metal artifact pattern is located.

According to the embodiment of the disclosure, in an application scenario for removing an artifact of a CT image, each stage of network may include a weighting coefficient update network, a metal artifact image feature update network, and a metal artifact removed image update network, wherein the weighting coefficient update network, the metal artifact image feature update network, and the metal artifact removed image update network include a residual error network structure; and the weighting coefficient update network includes: a Linear layer, a modified Linear Unit (ReLU) layer, a cross-link layer, and a Batch Normalization (BN) layer; the metal artifact image feature updating network comprises: convolutional layer, BN layer, ReLU layer, and cross-link layer; the metal artifact removal image update network comprises: convolutional layers, BN layers, ReLU layers, and cross-link layers.

Continuing with the mathematical model for removing metal artifacts of the CT image with metal artifacts in fig. 4 as an example, the structures of the mathematical model and the adaptive convolutional dictionary network specifically illustrating the image processing method for artifact removal described in the present disclosure are described with reference to fig. 4, 5A and 5B.

According to the embodiment of the disclosure, the common database representing different metal artifact types in all CT images with metal artifacts can be used

And constructing a convolutional layer, constructing the self-adaptive convolutional dictionary network based on the convolutional layer, and obtaining the optimization parameters of the self-adaptive convolutional dictionary network through end-to-end training of a training data set.

For the mathematical model shown in equation (4) above:

the solution objective is to estimate K from Y,

and X, wherein the corresponding optimization problems are as follows:

subject to‖K _n ‖ ₂ ＝1,n＝1,2,…,N (5)

where α, β and γ are trade-off parameters, f ₁ (·)、f ₂ (. and f) ₃ (. cndot.) are all regular terms, respectively representing weighting coefficient K and characteristic layer

And a priori structure of a clean CT image X, which can be designed as a neural network module to solve.

To solve the optimization problem in (5), a near-end gradient technique can be adopted to alternately update the weighting coefficient K and the feature layer

And a clean CT image X. The method comprises the following specific steps:

updating K:

in the (t +1) th iteration, K may be updated as:

subject to‖K _n ‖ ₂ ＝1,n＝1,2,…,N (6)

the corresponding quadratic approximation form is:

wherein, Ω ═ K | K _n ‖ ₂ ＝1,n＝1,2,…,N}；

η ₁ To update the step size, one can derive:

wherein the content of the first and second substances,

is a deep convolution operation;

representing the expansion of the tensor in the 3 rd dimension; vec (-) represents a vectorization operation.

Equation (7) can be equivalently written as:

for general a priori term f ₁ (. cndot.), equation (9) can be written as:

wherein the content of the first and second substances,

is a near-end operator, and a regularization term f ₁ (. o); omega can pass through the pair

A normalization operation is introduced to achieve this.

Updating

Similar to the update of K, in the (t +1) th iteration,

can be updated as:

wherein eta is ₂ In order to update the step size,

for general prior term f ₂ (. cndot.), equation (11) can be written as:

wherein the content of the first and second substances,

is a near-end operator, with a regularization term f ₂ (. about);

is a transposed convolution operation.

Updating X:

given K ^(t+1) And

can be updated as:

wherein the content of the first and second substances,

further, the update rule of X can be obtained as:

wherein the content of the first and second substances,

is a near-end operator, and a regularization term f ₃ (. cndot.) about.

By expanding the update formulas (10), (12) and (14), a complete Adaptive Convolutional Dictionary Network (acdinet) can be finally constructed, wherein each Network has good physical interpretability.

A schematic structure of an adaptive convolutional dictionary network, including a T-level network, according to an embodiment of the present disclosure is shown in fig. 5A. In each level of network, weighting coefficient K and characteristic layer are respectively matched

And the processed CT image X is updated. A schematic structure of each level of the network according to an embodiment of the present disclosure is shown in fig. 5B.

According to the embodiment of the disclosure, the ACDNET shown in FIG. 5A is composed of T stages (i.e., T-level networks), and in each stage, the corresponding network structure is composed of K-net,

and X-net, respectively for realizing K,

And an iterative update of X.

Next, the correspondence between the network structure and the above mathematical model in fig. 5A and 5B is explained.

In fig. 5A and 5B, specifically,

wherein

Is a residual structure, which specifically comprises: a linear layer, a ReLU layer, a linear layer, a cross-link layer, and a normalized operational layer at dimension d;

wherein

Is composed of 3 residual blocks, each of which comprises in sequence: convolutional layer, BN layer, ReLU layer, convolutional layer, BN layer, and cross-link layer;

wherein

Constructed from 3 residual blocksEach residual block comprises in sequence: convolutional layer, BN layer, ReLU layer, convolutional layer, BN layer, and cross-link layer.

As can be seen from FIG. 5A, the first-level network obtains the level 1 image feature M output by the first-level network based on the input image ⁽¹⁾ And first stage artifact removal image X ⁽¹⁾ (ii) a T-level image features M for a t-level network based at least in part on the output of the t-1 level network ^(t-1) And t-1 stage artifact removed image X ^(t-1) Obtaining and outputting the t-level image characteristics M output by the t-level network ^(t) And t-th order artifact removed image X ^(t) Wherein T is greater than 1 and less than or equal to T; t-1 level image feature M output by T-1 level network based on T-1 level network ^(T-1) And a T-1 th order artifact removed image X ^(T-1) Obtaining and outputting a Tth-level artifact removed image X output by the Tth-level network ^(T) As the artifact-removed processed image.

For each stage of the adaptive convolutional dictionary network, the iterative solution process is shown in fig. 5B. In the T-level network, updating the weighting coefficient and the image feature output by the T-1-level network by using an iterative updating rule based on the gradient descent of the near end to obtain the weighting coefficient and the image feature of the T-level network, wherein T is an integer which is greater than 1 and less than or equal to T. And sequentially and iteratively solving the K-net, the M-net and the X-net in a serial mode.

Iterative training can be performed on the basis of the artifact-free image (X) and the processed image and the target function processed by the image mask (I) for the adaptive convolutional dictionary network so as to optimize the network parameters of the adaptive convolutional dictionary network. Wherein the objective function may be a loss objective function constructed based on the artifact-free image (X) and the processed image, i.e. a

Wherein, mu _t To compromise the parameters, ω ₁ And ω ₂ Are weights used to balance the losses of the terms. For example, in a simulation experiment, the method canTo set up mu _t ＝0.1(t＝0,1,…,T-1)，μ _T ＝1，ω ₁ ＝ω ₂ ＝5×10 ^-4 ，T＝10。

According to embodiments of the present disclosure, Adam-based algorithms (Adaptive motion) may be employed to update solution optimization parameters, including,

convolution kernel

Step length eta ₁ 、η ₂ And η ₃ . In each iteration process, the error of the prediction result is calculated and reversely propagated to the convolutional neural network model, and the gradient is calculated and the parameters of the convolutional neural network model are updated.

Fig. 6 is a schematic flow chart diagram illustrating an image processing procedure for artifact removal according to an embodiment of the present disclosure.

As shown in fig. 6, an image processing procedure for artifact removal according to an embodiment of the present disclosure includes a neural network training phase and a testing phase.

In the neural network training phase, the artifact-bearing image may be first preprocessed to create a training data set for training the neural network, where the training data set includes a plurality of sets of image samples, and each set of image samples includes an artifact-bearing image (Y) and an artifact-free image (X) and an image mask (I) corresponding to the artifact-bearing image (Y). The computer then iteratively trains the acdtet according to the neural network training settings based on the preprocessed image samples, wherein the parameters of the acdtet are updated based on a predetermined objective function and Adam optimization algorithm during the training process. If the preset iteration times are reached in the training process, the trained model is saved, and if the preset iteration times are not reached, the ACDNet is continuously trained.

It should be appreciated that the use of up to a predetermined number of iterations as a criterion for determining completion of acdinet training is to prevent overfitting of the network. Optionally, the optimized image may be output in a visual manner, and the training of the acdinet is stopped after the optimized image is confirmed to meet the requirement.

In the testing stage, an input image to be processed and an image mask (I) corresponding to the input image are input to a computer, the computer loads a trained model, an image with artifacts removed is obtained through ACDNet forward calculation, and the computer can output the image with the artifacts removed for a user to refer.

Similarly, in the actual use process, the image processing process is similar to the test stage, and therefore, the description is omitted.

Fig. 7A is a block diagram 710 illustrating an image processing apparatus for artifact removal according to an embodiment of the present disclosure, where the apparatus 710 is used for a neural network training process in image processing.

According to an embodiment of the present disclosure, the image processing apparatus 710 for artifact removal may include: a training data set building module 711, an adaptive convolutional dictionary network 712, and a training module 713.

Wherein the training data set establishing module 711 may be configured to: establishing a training data set for training a neural network, wherein the training data set comprises a plurality of groups of image samples, and each group of image samples comprises an artifact-containing image (Y) and a non-artifact image (X) and an image mask (I) corresponding to the artifact-containing image (Y).

Optionally, the establishing a training data set for training the neural network may further include at least one of: -normalizing the pixel values of said artifact-bearing image (Y); and randomly cutting the artifact image (Y) to obtain an image block, and randomly turning the image block according to a preset probability.

Adaptive convolutional dictionary network 712 may be configured to: for at least one of the sets of image samples, performing artifact removal processing on the artifact-bearing image (Y) to obtain a processed image.

The adaptive convolution dictionary network 712 may include a base artifact dictionary that is a sample-invariant convolution dictionary and includes a first number of artifact convolution kernels, and a second number of adaptive convolution kernels for the image samples may be determined by a plurality of artifact convolution kernels in the base artifact dictionary and sample-variant weighting coefficients. Moreover, an artifact image of the artifact images may be determined by convolution of the second number of adaptive convolution kernels with an image feature of the artifact image, and the artifact image may be removed from the artifact image to obtain the processed image.

Training module 713 may be configured to: iteratively training the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and the objective function processed by the image mask (I) to optimize network parameters of the adaptive convolutional dictionary network.

According to the embodiment of the disclosure, each stage of network comprises a weighting coefficient updating network, an image feature updating network and an artifact removing image updating network, wherein the weighting coefficient updating network, the image feature updating network and the artifact removing image updating network comprise a residual error network structure and a normalization processing layer.

According to an embodiment of the present disclosure, the image processing apparatus 710 for artifact removal may further include: a testing module 714 configured to: after training is completed, testing the adaptive convolutional dictionary network, wherein the testing the adaptive convolutional dictionary network comprises: preprocessing an image with an artifact to be tested and inputting the preprocessed image into the self-adaptive convolution dictionary network; and processing the image with the artifact to be tested by utilizing an adaptive convolution dictionary network to obtain a processed image with the artifact removed.

Fig. 7B is a schematic diagram 720 showing the composition of an image processing apparatus for artifact removal according to an embodiment of the present disclosure, the apparatus 720 being used for a model using process in image processing.

According to an embodiment of the present disclosure, the image processing apparatus 720 for artifact removal may include: an image acquisition module 721, an image processing module 722.

Wherein the image acquisition module 721 may be configured to: an input image to be processed is acquired.

According to an embodiment of the present disclosure, the input image to be processed may be an image obtained after the original CT image is applied to its corresponding image mask. For example, the image acquisition module 721 may receive an original CT image and an image mask corresponding thereto, and apply the original CT image to the image mask corresponding thereto, thereby acquiring an input image to be processed.

According to the embodiment of the present disclosure, the input image to be processed may be an image with an artifact, or may be an image with an artifact that is subjected to preprocessing (e.g., denoising processing, normalization processing).

The image processing module 722 may be configured to: and processing the input image by using an adaptive convolution dictionary network to obtain a processed image with the artifact removed.

Wherein the adaptive convolutional dictionary network is trained based on an artifact database (D) and comprises a T-level network, wherein the first-level network obtains a level 1 image feature and a first-level artifact-removed image output by the first-level network based on the input image; the T-level network obtains and outputs the T-level image characteristic and the T-level artifact removed image output by the T-level network at least partially based on the T-1-level image characteristic and the T-1-level artifact removed image output by the T-level network, wherein T is greater than 1 and less than or equal to T; and the T-level network obtains and outputs the T-level artifact removed image output by the T-level network as the processed image for removing the artifact based on the T-1 level image feature and the T-1 level artifact removed image output by the T-level network.

According to an embodiment of the present disclosure, the adaptive convolution dictionary network includes a basic artifact dictionary that is a convolution dictionary that does not vary with an input image and includes a first number of artifact convolution kernels, a t-th order weighting coefficient of a t-th order network is determined, a second number of adaptive convolution kernels for the t-th order network is determined by a plurality of artifact convolution kernels in the basic artifact dictionary and the t-th order weighting coefficient; and determining a t-th level artifact-removed image based on the second number of adaptive convolution kernels and image characteristics of the t-th level network.

In general, the various example embodiments of this disclosure may be implemented in hardware or special purpose circuits, software, firmware, logic or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device. While aspects of embodiments of the disclosure have been illustrated or described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that the blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

For example, a method or apparatus according to embodiments of the disclosure may also be implemented by means of the architecture of computing device 3000 shown in fig. 8. As shown in fig. 8, computing device 3000 may include a bus 3010, one or more CPUs 3020, a Read Only Memory (ROM)3030, a Random Access Memory (RAM)3040, a communication port 3050 to connect to a network, input/output components 3060, a hard disk 3070, and the like. A storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used in the processing and/or communication of the methods provided by the present disclosure, as well as program instructions executed by the CPU. Computing device 3000 can also include user interface 3080. Of course, the architecture shown in FIG. 8 is merely exemplary, and one or more components of the computing device shown in FIG. 8 may be omitted as needed in implementing different devices.

According to yet another aspect of the present disclosure, there is also provided a computer-readable storage medium. Fig. 9 shows a schematic diagram 4000 of a storage medium according to the present disclosure.

As shown in fig. 9, computer storage media 4020 has stored thereon computer readable instructions 4010. The computer readable instructions 4010, when executed by a processor, can perform methods according to embodiments of the present disclosure described with reference to the above figures. The computer readable storage medium in embodiments of the present disclosure may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be Random Access Memory (RAM), which acts as external cache memory. By way of example and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchronous Link Dynamic Random Access Memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memories of the methods described herein are intended to comprise, without being limited to, these and any other suitable types of memory. It should be noted that the memories of the methods described herein are intended to comprise, without being limited to, these and any other suitable types of memory.

Embodiments of the present disclosure also provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions are read by a processor of a computer device from a computer-readable storage medium, and the computer instructions are executed by the processor to cause the computer device to perform a method according to an embodiment of the present disclosure.

In summary, embodiments of the present disclosure provide an image processing method, an apparatus, a computer program product, and a storage medium for artifact removal. The image processing method for artifact removal provided by the present disclosure includes: establishing a training data set for training a neural network, wherein the training data set comprises a plurality of groups of image samples, and each group of image samples comprises an image (Y) with an artifact, a non-artifact image (X) corresponding to the image sample and an image mask (I); for at least one of the sets of image samples, performing an artifact-removal process on the artifact-bearing image (Y) using an adaptive convolutional dictionary network to obtain a processed image, iteratively training the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image and an objective function processed by the image mask (I) to optimize network parameters of the adaptive convolutional dictionary network, wherein the adaptive convolutional dictionary network comprises a base artifact dictionary that is a sample-invariant convolutional dictionary and comprises a first number of artifact convolution kernels, and a second number of adaptive convolution kernels for the image samples is determined by a plurality of artifact convolution kernels in the base artifact dictionary and sample-variant weighting coefficients, wherein, determining an artifact image in the artifact image by convolution of the second number of adaptive convolution kernels with image features of the artifact image, and removing the artifact image from the artifact image to obtain the processed image.

It is to be noted that the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises at least one executable instruction for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The present disclosure uses specific words to describe embodiments of the disclosure. Reference to "a first/second embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic described in connection with at least one embodiment of the disclosure is included. Therefore, it is emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, certain features, structures, or characteristics of one or more embodiments of the disclosure may be combined as appropriate.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

The foregoing is illustrative of the present invention and is not to be construed as limiting thereof. Although a few exemplary embodiments of this invention have been described, those skilled in the art will readily appreciate that many modifications are possible in the exemplary embodiments without materially departing from the novel teachings and advantages of this invention. Accordingly, all such modifications are intended to be included within the scope of this invention as defined in the claims. It is to be understood that the foregoing is illustrative of the present invention and is not to be construed as limited to the specific embodiments disclosed, and that modifications to the disclosed embodiments, as well as other embodiments, are intended to be included within the scope of the appended claims. The invention is defined by the claims and their equivalents.

Claims

1. An image processing method for artifact removal, comprising:

establishing a training data set for training a neural network, wherein the training data set comprises a plurality of groups of image samples, and each group of image samples comprises an image (Y) with an artifact, a non-artifact image (X) corresponding to the image sample and an image mask (I);

for at least one of the sets of image samples,

performing artifact removal processing on the artifact-bearing image (Y) by using an adaptive convolutional dictionary network to obtain a processed image,

iteratively training the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and an objective function processed by the image mask (I), to optimize network parameters of the adaptive convolutional dictionary network,

wherein the adaptive convolutional dictionary network comprises a base artifact dictionary that is a sample-invariant convolutional dictionary and includes a first number of artifact convolution kernels, and a second number of adaptive convolution kernels for the image samples are determined by a plurality of artifact convolution kernels in the base artifact dictionary and sample-variant weighting coefficients,

wherein an artifact image of the artifact images is determined by convolution of the second number of adaptive convolution kernels with image features of the artifact images, and the artifact image is removed from the artifact images to obtain the processed image.

2. The image processing method of claim 1, wherein the first number of artifact convolution kernels indicates an artifact pattern, the image feature indicates a location where the artifact pattern is located,

the adaptive convolutional dictionary network comprises a T-level network, wherein in the T-level network, a weighting coefficient and image features output by the T-1-level network are updated by using an iterative update rule based on near-end gradient descent to obtain the weighting coefficient and the image features of the T-level network, wherein T is an integer which is greater than 1 and less than or equal to T.

3. The image processing method of claim 2, wherein each stage network comprises a weighting coefficient update network, an image feature update network, and an artifact-removed image update network, wherein,

the weighting coefficient updating network, the image characteristic updating network and the artifact removing image updating network comprise a residual error network structure and a normalization processing layer.

4. The image processing method of claim 1, wherein the establishing a training data set for training a neural network further comprises at least one of:

-normalizing the pixel values of said artifact-bearing image (Y); and

and randomly cutting the image (Y) with the artifact to obtain an image block, and randomly turning the image block according to a preset probability.

5. Image processing method of claim 1, wherein the objective function is a loss objective function constructed on the basis of the artifact-free image (X) and the processed image,

wherein the iteratively training the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and an objective function to optimize network parameters of the adaptive convolutional dictionary network further comprises:

calculating a loss objective function, reversely transmitting the result to the Adaptive convolution dictionary network, and optimizing network parameters of the Adaptive convolution dictionary network based on an Adaptive moment estimation (Adam) algorithm.

6. The image processing method of claim 1, further comprising: after training is completed, testing the self-adaptive convolution dictionary network,

wherein the testing the adaptive convolutional dictionary network comprises:

preprocessing an image with an artifact to be tested and inputting the preprocessed image into the self-adaptive convolution dictionary network;

and processing the image with the artifact to be tested by utilizing an adaptive convolution dictionary network to obtain a processed image with the artifact removed.

7. The image processing method of claim 1, wherein the artifact is a metal artifact and the artifact-bearing image is a CT image with a metal artifact,

the training data set comprises a plurality of groups of CT image samples, and each group of image samples comprises a CT image with a metal artifact, a CT image without the metal artifact corresponding to the CT image and a non-metal area mask;

the adaptive convolution dictionary network includes a base metal artifact dictionary that is a metal artifact convolution dictionary that does not vary with CT image samples and includes a first number of metal artifact convolution kernels, and a second number of adaptive convolution kernels for the CT image samples are determined by a plurality of metal artifact convolution kernels in the base metal artifact dictionary and weighting coefficients that vary with CT image samples, wherein the first number of metal artifact convolution kernels indicates a metal artifact pattern and the metal artifact image features indicate where the metal artifact pattern is located.

8. The image processing method of claim 7, wherein each level of network comprises a weighting coefficient update network, a metal artifact image feature update network, and a metal artifact removed image update network, wherein,

the weighting coefficient updating network, the metal artifact image characteristic updating network and the metal artifact removing image updating network comprise a residual error network structure; and is

The weighting coefficient update network includes: a Linear layer, a modified Linear Unit (ReLU) layer, a cross-link layer, and a Batch Normalization (BN) layer;

the metal artifact image feature updating network comprises: convolutional layer, BN layer, ReLU layer, and cross-link layer;

the metal artifact removal image update network comprises: convolutional layers, BN layers, ReLU layers, and cross-link layers.

9. An image processing method for artifact removal, comprising:

acquiring an input image to be processed;

processing the input image using an adaptive convolutional dictionary network to obtain a processed image with artifacts removed,

wherein the adaptive convolutional dictionary network is trained based on an artifact database (D) and comprises a T-level network, wherein the first-level network obtains a 1 st-level image feature and a first-level artifact-removed image output by the first-level network based on the input image; the T-level network obtains and outputs the T-level image characteristic and the T-level artifact removed image output by the T-level network at least partially based on the T-1-level image characteristic and the T-1-level artifact removed image output by the T-level network, wherein T is greater than 1 and less than or equal to T; and the T-level network obtains and outputs the T-level artifact removed image output by the T-level network as the processed image for removing the artifact based on the T-1 level image feature and the T-1 level artifact removed image output by the T-level network.

10. The image processing method of claim 9, wherein the adaptive convolutional dictionary network comprises a base artifact dictionary that is a convolutional dictionary that does not vary with the input image and comprises a first number of artifact convolution kernels,

determining a t-th order weighting coefficient for a t-th order network, determining a second number of adaptive convolution kernels for the t-th order network from a plurality of artifact convolution kernels in the base artifact dictionary and the t-th order weighting coefficient; and determining a t-th level artifact-removed image based on the second number of adaptive convolution kernels and image characteristics of the t-th level network.

11. The image processing method of claim 9, wherein each stage network comprises a weighting coefficient update network, an image feature update network, and an artifact-removed image update network, wherein,

12. The image processing method as claimed in claim 9, wherein the input image to be processed is an image to which an image mask (I) is applied.

13. The image processing method according to claim 12, wherein the artifact is a metal artifact, the input image to be processed is a CT image with a metal artifact, the image mask (I) is a non-metal region mask corresponding to the CT image with a metal artifact,

wherein each stage of network comprises a weighting coefficient updating network, a metal artifact image characteristic updating network and a metal artifact removing image updating network, wherein,

14. An image processing apparatus for artifact removal, comprising:

a training data set establishment module configured to: establishing a training data set for training a neural network, wherein the training data set comprises a plurality of groups of image samples, and each group of image samples comprises an image (Y) with an artifact, a non-artifact image (X) corresponding to the image sample and an image mask (I);

an adaptive convolutional dictionary network configured to: for at least one of the groups of image samples, performing artifact removal processing on the artifact-bearing image (Y) to obtain a processed image;

a training module configured to: iteratively training the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and an objective function processed by the image mask (I), to optimize network parameters of the adaptive convolutional dictionary network;

15. An image processing apparatus for artifact removal, comprising:

an image acquisition module configured to: acquiring an input image to be processed;

an image processing module configured to: processing the input image by using an adaptive convolution dictionary network to obtain a processed image with artifacts removed;

wherein the adaptive convolutional dictionary network is trained based on an artifact database (D) and comprises a T-level network, wherein the first-level network obtains a level 1 image feature and a first-level artifact-removed image output by the first-level network based on the input image; the T-level network obtains and outputs the T-level image characteristic and the T-level artifact removed image output by the T-level network at least partially based on the T-1-level image characteristic and the T-1-level artifact removed image output by the T-level network, wherein T is greater than 1 and less than or equal to T; and the T-level network obtains and outputs the T-level artifact removed image output by the T-level network as the processed image for removing the artifact based on the T-1 level image characteristic output by the T-1 level network and the T-1 level artifact removed image.

16. A computer program product comprising computer software code for implementing a method according to any one of claims 1-13 when executed by a processor.

17. A computer-readable storage medium having stored thereon computer-executable instructions, which when executed by a processor, are for implementing the method of any one of claims 1-13.