CN115965837A - Image reconstruction model training method, image reconstruction method and related equipment - Google Patents
Image reconstruction model training method, image reconstruction method and related equipment Download PDFInfo
- Publication number
- CN115965837A CN115965837A CN202310162463.7A CN202310162463A CN115965837A CN 115965837 A CN115965837 A CN 115965837A CN 202310162463 A CN202310162463 A CN 202310162463A CN 115965837 A CN115965837 A CN 115965837A
- Authority
- CN
- China
- Prior art keywords
- image
- dimensional
- reconstruction model
- slice
- reconstructed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The disclosure provides an image reconstruction model training method, an image reconstruction method and related equipment, and relates to the technical field of image analysis. The method comprises the steps of obtaining a three-dimensional training image and a first slice image sequence corresponding to the three-dimensional image in different directions; respectively inputting the first slice image sequences in different directions into a preset image reconstruction model to generate a first reconstructed image; training a preset image reconstruction model according to first loss between each generated first reconstruction image and the three-dimensional training image to obtain an initial image reconstruction model; acquiring a two-dimensional training image; respectively inputting each second slice image sequence in the two-dimensional training image into the initial image reconstruction model to generate a second reconstruction image; and respectively constraining each generated second reconstructed image according to the two-dimensional training image so as to adjust the parameters of the initial image reconstruction model and obtain a final image reconstruction model. The method and the device can improve the reconstruction accuracy of the low-resolution slice image sequence.
Description
Technical Field
The present disclosure relates to the field of image analysis technologies, and in particular, to an image reconstruction model training method, an image reconstruction method, and a related device.
Background
The image reconstruction technique is a technique of obtaining shape information of a three-dimensional object by digital processing from data measured outside the object. At present, image reconstruction techniques are widely used in various fields, for example, in the medical field, a three-dimensional image of each part of a human body can be reconstructed by an Imaging technique such as Magnetic Resonance Imaging (MRI) so as to facilitate observation by a doctor during a diagnosis and treatment process.
In the related art, when the number of slices acquired by the imaging technology is too low, the interlayer resolution between the slices is low, so that it is difficult to reconstruct a high-resolution three-dimensional image, which is not beneficial to observation and use of related technicians.
Disclosure of Invention
In order to overcome the problems in the related art, the present disclosure provides an image reconstruction model training method, an image reconstruction method, and related devices.
According to a first aspect of an embodiment of the present disclosure, there is provided an image reconstruction model training method, including: acquiring a three-dimensional training image and a first slice image sequence corresponding to the three-dimensional image in different directions, wherein the resolution of the first slice image sequence is lower than that of the three-dimensional training image; respectively inputting the first slice image sequences in different directions into a preset image reconstruction model to generate a first reconstructed image; training a preset image reconstruction model according to first loss between each generated first reconstruction image and the three-dimensional training image to obtain an initial image reconstruction model; acquiring a two-dimensional training image, wherein the two-dimensional training image comprises a second slice image sequence of an object to be reconstructed in different directions, the object shown by the three-dimensional training image is the same as the object to be reconstructed in type, and the resolution of the second slice image sequence is lower than that of the three-dimensional training image; respectively inputting each second slice image sequence in the two-dimensional training image into the initial image reconstruction model to generate a second reconstruction image; and respectively constraining each generated second reconstructed image according to the two-dimensional training image so as to adjust the parameters of the initial image reconstruction model and obtain the final image reconstruction model.
In some embodiments, the method for obtaining the first sequence of slice images in the different directions comprises: and respectively carrying out down-sampling treatment on the three-dimensional training images along different directions to obtain first slice image sequences in different directions.
In some embodiments, the resolution of the first sequence of slice images is the same as or similar to the resolution of the second sequence of slice images.
In some embodiments, the constraining each generated second reconstructed image according to the two-dimensional training image to adjust the parameters of the initial image reconstruction model to obtain the final image reconstruction model includes: and respectively carrying out the following processing on each generated second reconstruction image to obtain a final image reconstruction model: respectively calculating the loss between the second reconstructed image and the second slice image sequence in each direction in different directions to obtain second losses respectively corresponding to the different directions; and synthesizing the second losses corresponding to different directions respectively, and adjusting the parameters of the initial image reconstruction model.
In some embodiments, before constraining each of the generated second reconstructed images according to the two-dimensional training image so as to adjust parameters of the initial image reconstruction model to obtain a final image reconstruction model, the method further includes: and performing a registration operation on the second slice image sequences in different directions to align pixels of the second slice image sequences in different directions in space.
In some embodiments, synthesizing the second losses respectively corresponding to the different directions to adjust the parameters of the initial image reconstruction model includes: calculating a smoothing loss between each pixel in the second reconstructed image; and adjusting parameters of the initial image reconstruction model by integrating the smooth loss and the second losses corresponding to different directions respectively.
In some embodiments, the three-dimensional training image shows an object as an adult brain and the object to be reconstructed is a fetal brain.
According to a second aspect of the embodiments of the present disclosure, there is provided an image reconstruction method, including: acquiring a slice image sequence of an object to be reconstructed; and inputting the slice image sequence into a pre-trained image reconstruction model, reconstructing a final three-dimensional image of the object to be reconstructed, wherein the image reconstruction model is obtained based on the training of the first aspect of the embodiment of the disclosure.
In some embodiments, in the case that the sequence of slice images includes a sequence of slice images of the object to be reconstructed in a plurality of directions, inputting the sequence of slice images into a pre-trained image reconstruction model, reconstructing a final three-dimensional image of the object to be reconstructed includes: respectively inputting the slice image sequences in multiple directions into an image reconstruction model to obtain multiple initial three-dimensional images; and fusing the plurality of initial three-dimensional images to obtain a final three-dimensional image.
In some embodiments, fusing the plurality of initial three-dimensional images to obtain a final three-dimensional image comprises: for each first pixel in the final three-dimensional image, the following processing is performed: and taking the pixel average value of a plurality of second pixels with the same positions as the first pixels in the plurality of initial three-dimensional images as the pixel value of the first pixels, thereby obtaining a final three-dimensional image.
According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including: a processor; and a memory for storing executable instructions for the processor; wherein the processor is configured to perform the method of the first or second aspect of an embodiment of the disclosure via execution of the executable instructions.
According to a fourth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of the first or second aspect of embodiments of the present disclosure.
The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:
according to the embodiment of the disclosure, the image reconstruction model is preliminarily trained by using the high-resolution three-dimensional training image to obtain an initial image reconstruction model. And further utilizing the two-dimensional training image which has lower resolution and can show the object to be reconstructed to train the initial image reconstruction model again to obtain a final image reconstruction model. Due to the fact that the type of the object shown by the three-dimensional training image is the same as that of the object to be reconstructed, the image reconstruction model capable of accurately reconstructing the object to be reconstructed can be trained based on the idea of transfer learning, and reconstruction accuracy of a low-resolution slice image sequence is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
Fig. 1 shows a schematic diagram of an exemplary MRI image in the related art.
Fig. 2 shows a system architecture diagram of an image reconstruction model training method in an embodiment of the present disclosure.
Fig. 3 shows a schematic flow chart of an image reconstruction model training method in an embodiment of the present disclosure.
Fig. 4 shows a schematic diagram of a training process of a preset image reconstruction model in the embodiment of the present disclosure.
Fig. 5 shows a schematic diagram of a training process of an initial image reconstruction model in an embodiment of the present disclosure.
Fig. 6 shows a schematic flow chart of an image reconstruction method in an embodiment of the present disclosure.
Fig. 7 shows a schematic structural diagram of an image reconstruction model training apparatus in an embodiment of the present disclosure.
Fig. 8 shows a schematic structural diagram of an image reconstruction apparatus in an embodiment of the present disclosure.
Fig. 9 shows a schematic structural diagram of an electronic device in an embodiment of the present disclosure.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the disclosure, as detailed in the appended claims.
The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure. The word "if," as used herein, may be interpreted as "at … …" or "at … …" or "in response to a determination," depending on the context.
Among current imaging techniques, ultrasound is the most commonly used imaging modality, but the detailed structure of the object to be imaged, such as the brain, internal organs, etc. in fetal development, is not well displayed due to low contrast, narrow field of view, low signal-to-noise ratio, etc. If some suspected structures of the object to be imaged are abnormal in the ultrasonic examination, the MRI can be used as a supplementary examination to provide more detailed structural information, because the MRI has better soft tissue contrast, a larger visual field, a high signal-to-noise ratio and no radiation, and can acquire image information of any section and see some structures which are not easy to find in the ultrasonic. MRI is becoming an increasingly important type of imaging examination.
However, the MRI acquisition process is usually long, and the object to be imaged may have motion during this time (e.g. the fetus is uncontrollable moving in the mother), which is prone to motion artifacts. Therefore, clinically with fast imaging, a high-resolution two-dimensional slice can be obtained in less than one second, but different slices are still affected by the motion of the object to be imaged, have displacement deviation with each other, and have a large layer spacing, generally 3-4mm, so that the obtained three-dimensional data is actually stacked by a stack of two-dimensional slices, called stack (stack), and the three-dimensional resolution is low.
As shown in fig. 1, fig. 1 shows an exemplary MRI image diagram in the related art. In fig. 1, from left to right are respectively the cross-sectional, coronal and sagittal fetal brain images acquired by MRI. Since the image is acquired at the transverse position, the image is influenced by the movement of the fetus when viewed from other directions except the fetal brain image at the transverse position, and the resolution is very low.
Since MRI can acquire data from different cross sections, generally MRI can be acquired from three orthogonal directions of transverse position, coronal position and sagittal position respectively to obtain a two-dimensional slice image sequence with high resolution in the layer of the three directions respectively. However, the three-dimensional slice image sequence still has missing information with respect to the three-dimensional image, and thus a high-resolution three-dimensional image cannot be reconstructed.
In view of this, the method provided by the embodiment of the present disclosure provides a training mode combining supervision and unsupervised, and can train an image reconstruction model capable of accurately reconstructing an object to be reconstructed based on the idea of transfer learning, improve the reconstruction accuracy for a low-resolution slice image sequence, and be applicable to reconstruction for a moving object.
It is understood that the medical image analysis method of the present embodiment may be executed on any electronic device, for example, it may be executed on a server, or on a terminal, or may be executed by both the terminal and the server. The above examples should not be construed as limiting the present disclosure.
Exemplarily, fig. 2 shows an exemplary system architecture diagram of a medical image analysis method or a medical image analysis apparatus to which an embodiment of the present disclosure may be applied.
As shown in fig. 2, the system architecture 200 includes an image acquisition apparatus 201, a server 202, and a terminal 203. The image capturing device 201 and the terminal 203 are connected to the server 202 through a network, for example, a wired or wireless network connection.
Illustratively, the image acquisition device 201 may be configured to perform data acquisition for acquiring a subject, and may acquire a two-dimensional slice image sequence and/or a three-dimensional image relating to at least a portion of the subject. The subject may be biological or non-biological. For example, the subject may be a patient, an artificial object, a laboratory, or the like. For example, the object may include a particular part, organ, and/or tissue of the patient. For example, the acquired objects may include the head, neck, chest, heart, stomach, blood vessels, soft tissue, tumors, nodules, etc., or any combination thereof, of the subject.
The server 202 may be a single server, or may be a server cluster or a cloud server composed of a plurality of servers. For example, the server may be an interworking server or a background server among a plurality of heterogeneous systems, may also be an independent physical server, may also be a server cluster or a distributed system formed by a plurality of physical servers, and may also be a cloud server that provides basic cloud computing services such as cloud services, a cloud database, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms, and the like.
An image reconstruction model is deployed on the server 202. The server 202 may execute the image reconstruction model training method or the image reconstruction method provided by the embodiment of the present disclosure according to the two-dimensional slice image sequence and/or the three-dimensional image acquired by the image acquisition apparatus.
In addition, the server 202 can be further configured to send the generated final three-dimensional image analysis result to the terminal 203 so as to present the final three-dimensional image generated based on the image reconstruction model to the user.
The terminal 203 may receive the final three-dimensional image generated by the server 202. The terminal 203 may include a mobile phone, a smart television, a tablet Computer, a notebook Computer, or a Personal Computer (PC), etc. A client, which may be an application client or a browser client, etc., may also be disposed on the terminal 203.
Those skilled in the art will appreciate that the number of image capturing devices, servers, and terminals shown in fig. 2 is merely illustrative, and that there may be any number of image capturing devices, servers, and terminals according to actual needs, and the present disclosure is not limited thereto.
The present exemplary embodiment will be described in detail below with reference to the drawings and examples.
First, an embodiment of the present disclosure provides an image reconstruction model training method, which may be performed by any electronic device.
Fig. 3 is a schematic flowchart illustrating an image reconstruction model training method in an embodiment of the present disclosure, and as shown in fig. 3, the image reconstruction model training method provided in the embodiment of the present disclosure includes the following steps.
S301, a three-dimensional training image and a first slice image sequence corresponding to the three-dimensional image in different directions are obtained.
It should be noted that the three-dimensional training image may be obtained by an image acquisition apparatus, and for example, the three-dimensional training image may include an ultrasound apparatus, an X-ray apparatus, a nuclear magnetic resonance apparatus, a nuclear medicine apparatus, a medical optical apparatus, a thermal imaging apparatus, and the like, which is not limited in this disclosure.
Illustratively, the three-dimensional training images may be various types of images. For example, the medical image may include an ultrasound image, an X-ray, a Computed Tomography (CT), a Magnetic Resonance Imaging (MRI), a Digital Subtraction Angiography (DSA), a Positron Emission Tomography (PET), and the like according to a division of an apparatus for acquiring the medical image. Further, the three-dimensional training image may include a brain tissue image, a spinal cord image, a fundus image, a blood vessel image, a pancreas image, a lung image, and the like, in accordance with the object division shown by the three-dimensional training image.
In some embodiments, the three-dimensional training image is an image of a stationary object acquired by MRI with higher resolution both intra-and inter-laminar.
In some embodiments, in order for the preset image reconstruction model in the subsequent S302 to learn the upsampled features in the image, the resolution of the first slice image sequence may be lower than the resolution of the three-dimensional training image.
For example, the intra-layer resolution of the first sequence of slice images may be the same as or similar to the three-dimensional training image, while the inter-layer resolution of the first sequence of slice images is lower than the resolution of the three-dimensional training image.
For example, when the first slice image sequence is a slice image sequence acquired at a transverse position, the resolution of the first slice image sequence at the transverse position is higher and may be the same as or similar to the resolution of the three-dimensional training image, and the resolutions of the sagittal position and the coronal position are lower than the resolution of the three-dimensional training image.
It should be noted that the first slice image sequence may be obtained by performing low-resolution acquisition on an object shown by the three-dimensional training image, or may be obtained by performing down-sampling on the three-dimensional training image.
Illustratively, the obtaining method of the first slice image sequence may be: and respectively carrying out down-sampling treatment on the three-dimensional training image along different directions to obtain a first slice image sequence in different directions. It is to be appreciated that the three-dimensional training images in the disclosed embodiments may be decomposed into a sequence of low-layer-spacing (high-layer-resolution) two-dimensional slice images. The down-sampling process of the three-dimensional image can be realized by extracting a plurality of layers of two-dimensional slices in the two-dimensional slice image sequence, namely, the resolution between layers is reduced.
In some embodiments, in order to enable the first slice image sequence to accurately simulate the blurring condition of the moving object, the embodiments of the present disclosure may further add motion blur during the down-sampling process. For example, a degree of blurring caused by rotation and/or translation is added to the first slice image sequence.
S302, the first slice image sequences in different directions are respectively input into a preset image reconstruction model, and a first reconstructed image is generated.
It should be noted that the preset image reconstruction model may adopt an encoder-decoder network model.
Wherein the encoder may extract intra-layer features and inter-layer correlation features of the sequence of slice images in order to learn spatial dependencies upon upsampling. In order to ensure that the sizes of the matched feature maps at the two ends of the encoder and the decoder are consistent, the low-dimensional feature map output by the encoder part needs to be subjected to upsampling operation and then spliced with the feature map of the decoder part through jumping connection. In this regard, the decoder performs upsampling in both the slice dimension and the slice layer of the image, restoring the in-layer dimensions of the slice image sequence, while generating new slices between the original slice image sequences.
In some embodiments, one first reconstructed image may be generated for each of the first slice image sequences of each of the different directions by inputting the first slice image sequence of each of the different directions into a preset image reconstruction model.
And S303, training the preset image reconstruction model according to the first loss between each generated first reconstruction image and the three-dimensional training image to obtain an initial image reconstruction model.
In some embodiments, referring to fig. 4, fig. 4 is a schematic diagram illustrating a training process of the pre-set image reconstruction model. By inputting the first slice image sequences with the resolution of 128 × 128 × 20 into the preset image reconstruction models, respectively, the first reconstructed image with the resolution same as that of the three-dimensional training image (128 × 128 × 80) is output.
In order to ensure that the output first reconstructed image is similar to the real three-dimensional training image obtained through acquisition as much as possible, and the anatomical structure characteristics and the low-resolution to high-resolution mapping details of the object to be reconstructed are learned in a large number of training images, the training of the preset image reconstruction model can be realized by calculating the first loss between each first reconstructed image and the three-dimensional training image, so that the initial image reconstruction model is obtained.
S304, acquiring a two-dimensional training image.
It should be noted that the two-dimensional training image may be a second sequence of slice images of the object to be reconstructed in different directions. The two-dimensional training image is acquired in a manner similar to the three-dimensional training image, except that since the object to be reconstructed is a moving object, a high-resolution three-dimensional image of the object cannot be acquired by the image acquisition means in the related art. The second slice image sequence is typically a two-dimensional slice image sequence with very low inter-layer resolution.
In some embodiments, the three-dimensional training image shows an object of the same class as the object to be reconstructed, and the resolution of the second sequence of slice images is lower than the resolution of the three-dimensional training image.
It can be understood that, based on the idea of migration learning, the embodiment of the present disclosure can apply a model obtained by training an object shown in a three-dimensional training image as an initial point to model training for an object to be reconstructed. That is to say, because the object shown by the three-dimensional training image and the object to be reconstructed have a certain similarity, the image reconstruction model is trained in advance by using the three-dimensional training image, so that a certain basis and support can be provided for the process based on the two-dimensional training image training, and the accuracy of model expression is improved.
In some embodiments, the image reconstruction model in embodiments of the present disclosure is able to learn better up-sampled features when the resolution of the second sequence of slice images is the same as or similar to the resolution of the first sequence of slice images.
S305, inputting each second slice image sequence in the two-dimensional training image into the initial image reconstruction model respectively, and generating a second reconstruction image.
Exemplarily, referring to fig. 5, fig. 5 shows a schematic diagram of a training process for an initial image reconstruction model in an embodiment of the present disclosure. Similarly to S302, second slice image sequences of different directions of the object to be reconstructed in the two-dimensional training image are respectively input to the initial image reconstruction model, so that one second reconstruction image is respectively generated for the second slice image sequence of each direction.
And S306, respectively constraining each generated second reconstructed image according to the two-dimensional training image so as to adjust parameters of the initial image reconstruction model and obtain a final image reconstruction model.
It should be noted that, for the object to be reconstructed in the embodiment of the present disclosure, due to its own reason (for example, the object to be reconstructed has motion to blur the acquired image), real high-resolution three-dimensional image data cannot be obtained. In order to complete a training task for reconstructing an object to be reconstructed, the embodiment of the present disclosure directly constrains pixels of a generated second reconstructed image in space through pixels in an acquired real two-dimensional training image.
Illustratively, with continuing reference to fig. 5, the two-dimensional training image includes a second sequence of slice images of the object to be reconstructed in different directions, and for each generated second reconstructed image, second losses respectively corresponding to the different directions may be obtained by respectively calculating a loss between the second reconstructed image and the second sequence of slice images in each of the different directions. The parameters of the initial image reconstruction model can be adjusted by integrating the second losses corresponding to different directions (for example, adding the second losses corresponding to different directions to be used as the overall loss in the model parameter adjustment process), so as to obtain the final image reconstruction model.
In some embodiments, to accurately determine the pixel correspondence of the two-dimensional training image and the second reconstructed image in space, a registration operation may be performed on the second sequence of slice images in different directions to align pixels of the second sequence of slice images in different directions in space before calculating the second loss.
Referring to fig. 5 again, in order to ensure that the second reconstructed image is excessively smooth in the generating process and avoid abrupt bright-dark pixel change or line break, the above-mentioned overall loss may further include the own smooth loss of the second reconstructed image.
Specifically, after the smoothing loss between each pixel in the second reconstructed image is calculated, the smoothing loss and the second losses respectively corresponding to different directions may be integrated (for example, the smoothing loss and the second losses corresponding to different directions are added to be used as an overall loss in the model parameter adjustment process), and the initial image reconstruction model may be trained to obtain the final image reconstruction model.
According to the embodiment of the disclosure, the image reconstruction model is preliminarily trained by using the high-resolution three-dimensional training image to obtain the initial image reconstruction model. And further utilizing the two-dimensional training image which has lower resolution and can show the object to be reconstructed to train the initial image reconstruction model again to obtain a final image reconstruction model. Due to the fact that the type of the object shown by the three-dimensional training image is the same as that of the object to be reconstructed, the image reconstruction model capable of accurately reconstructing the object to be reconstructed can be trained based on the idea of transfer learning, and reconstruction accuracy of a low-resolution slice image sequence is improved.
In addition, in the medical field, the image reconstruction model trained by the method provided by the embodiment of the disclosure can accurately reconstruct an MRI image of a moving object. Such as fetal brain images, cardiac images, etc.
Based on the same inventive concept, the embodiment of the present disclosure also provides an image reconstruction method, which may be executed by any electronic device.
Fig. 6 shows a schematic flowchart of an image reconstruction method in an embodiment of the present disclosure, and as shown in fig. 6, the image reconstruction method provided in the embodiment of the present disclosure includes the following steps.
S601, acquiring a slice image sequence of the object to be reconstructed.
And S602, inputting the slice image sequence into a pre-trained image reconstruction model, and reconstructing a final three-dimensional image of the object to be reconstructed.
It should be noted that the image reconstruction model in the embodiment of the present disclosure is obtained by pre-training based on the method provided in the embodiment of the training method for the image reconstruction model.
For example, in the case that the slice image sequence includes a slice image sequence of the object to be reconstructed in multiple directions, the slice image sequence is input into a pre-trained image reconstruction model, and the manner of reconstructing the final three-dimensional image of the object to be reconstructed may be: respectively inputting the slice image sequences in multiple directions into an image reconstruction model to obtain multiple initial three-dimensional images; and fusing the plurality of initial three-dimensional images to obtain a final three-dimensional image.
Illustratively, the manner of fusing a plurality of initial three-dimensional images to obtain a final three-dimensional image may be: and regarding each first pixel in the final three-dimensional image, taking the pixel average value of a plurality of second pixels which have the same positions with the first pixel in a plurality of initial three-dimensional images as the pixel value of the first pixel, thereby obtaining the final three-dimensional image.
The image reconstruction method provided by the embodiment of the disclosure can realize high-resolution image reconstruction by inputting a slice image sequence in a single direction, so as to be suitable for a situation that the slice image sequence is difficult to acquire in multiple directions (for example, a claustrophobia patient may have fear on MRI imaging for a long time).
Of course, by inputting a sequence of image slices in multiple directions, the images reconstructed by the present disclosure can be made to a higher precision and accuracy for reference by the relevant skilled person.
Based on the same inventive concept, an image reconstruction model training device is also provided in the embodiments of the present disclosure, as in the following embodiments. Because the principle of solving the problem of the embodiment of the apparatus is similar to that of the embodiment of the image reconstruction model training method, the embodiment of the apparatus can be implemented by referring to the embodiment of the image reconstruction model training method, and repeated details are not repeated.
Fig. 7 is a schematic structural diagram of an image reconstruction model training apparatus in an embodiment of the present disclosure, and as shown in fig. 7, the image reconstruction model training apparatus 700 includes: a first obtaining module 701, a first generating module 702, a first training module 703, a second obtaining module 704, a second generating module 705, and a second training module 706.
Specifically, the first obtaining module 701 is configured to obtain a three-dimensional training image and a first slice image sequence in different directions corresponding to the three-dimensional image, where a resolution of the first slice image sequence is lower than a resolution of the three-dimensional training image. The first generating module 702 is configured to input the first slice image sequences in different directions into a preset image reconstruction model, respectively, and generate a first reconstructed image. The first training module 703 is configured to train a preset image reconstruction model according to a first loss between each generated first reconstructed image and the three-dimensional training image, so as to obtain an initial image reconstruction model. The second obtaining module 704 is configured to obtain a two-dimensional training image, where the two-dimensional training image includes a second slice image sequence of the object to be reconstructed in different directions, the object shown in the three-dimensional training image has the same category as the object to be reconstructed, and a resolution of the second slice image sequence is lower than a resolution of the three-dimensional training image. The second generating module 705 is configured to input each second slice image sequence in the two-dimensional training image into the initial image reconstruction model, and generate a second reconstructed image. The second training module 706 is configured to constrain each generated second reconstructed image according to the two-dimensional training image, so as to adjust parameters of the initial image reconstruction model, and obtain a final image reconstruction model.
In some embodiments, the first generating module 702 is further configured to perform down-sampling processing on the three-dimensional training image along different directions, respectively, to obtain a first slice image sequence in different directions.
In some embodiments, the resolution of the first sequence of slice images is the same as or similar to the resolution of the second sequence of slice images.
In some embodiments, the second training module 706 is configured to perform the following processing on each generated second reconstructed image to obtain a final image reconstruction model: respectively calculating the loss between the second reconstructed image and the second slice image sequence in each direction in different directions to obtain second losses respectively corresponding to the different directions; and synthesizing the second losses corresponding to different directions respectively, and adjusting the parameters of the initial image reconstruction model.
In some embodiments, the second training module 706 is further configured to perform a registration operation on the second sequence of slice images in different directions to spatially align pixels of the second sequence of slice images in different directions.
In some embodiments, the second training module 706 is further configured to calculate a smoothness loss between pixels in the second reconstructed image; and adjusting parameters of the initial image reconstruction model by integrating the smooth loss and the second losses corresponding to different directions respectively.
In some embodiments, the three-dimensional training image shows an object as an adult brain and the object to be reconstructed is a fetal brain.
Based on the same inventive concept, an image reconstruction apparatus is also provided in the embodiments of the present disclosure, such as the following embodiments. Because the principle of the embodiment of the apparatus for solving the problem is similar to that of the embodiment of the image reconstruction method, the embodiment of the apparatus can be implemented by referring to the implementation of the embodiment of the image reconstruction method, and repeated details are not repeated.
Fig. 8 is a schematic structural diagram of an image reconstruction apparatus in an embodiment of the present disclosure, and as shown in fig. 8, the image reconstruction apparatus 800 includes: an acquisition module 801 and a reconstruction module 802.
In particular, the acquisition module 801 is configured to acquire a sequence of slice images of an object to be reconstructed. The reconstruction module 802 is configured to input the slice image sequence into a pre-trained image reconstruction model, and reconstruct a final three-dimensional image of the object to be reconstructed. The image reconstruction model is obtained by training based on the method provided by the embodiment of the image reconstruction model training method.
In some embodiments, the reconstruction module 802 is further configured to input the slice image sequences in multiple directions into the image reconstruction model, respectively, to obtain multiple initial three-dimensional images; and fusing the plurality of initial three-dimensional images to obtain a final three-dimensional image.
In some embodiments, the reconstruction module 802 is further configured to, for each first pixel in the final three-dimensional image: and taking the pixel average value of a plurality of second pixels with the same positions as the first pixels in the plurality of initial three-dimensional images as the pixel value of the first pixels, thereby obtaining a final three-dimensional image.
As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method or program product. Accordingly, various aspects of the disclosure may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
An electronic device 900 according to this embodiment of the disclosure is described below with reference to fig. 9. The electronic device 900 shown in fig. 9 is only an example and should not bring any limitation to the functions and applicable scope of the embodiments of the present disclosure.
As shown in fig. 9, the electronic device 900 is embodied in the form of a general purpose computing device. Components of electronic device 900 may include, but are not limited to: the at least one processing unit 910, the at least one memory unit 920, and a bus 930 that couples various system components including the memory unit 920 and the processing unit 910.
Where the storage unit stores program code, which may be executed by the processing unit 910, to cause the processing unit 910 to perform the steps according to various exemplary embodiments of the present disclosure described in the above section "exemplary method" of this specification.
In some embodiments, the processing unit 910 may perform the following steps of the above-described embodiment of the image reconstruction model training method: acquiring a three-dimensional training image and a first slice image sequence corresponding to the three-dimensional image in different directions, wherein the resolution of the first slice image sequence is lower than that of the three-dimensional training image; respectively inputting the first slice image sequences in different directions into a preset image reconstruction model to generate a first reconstructed image; training a preset image reconstruction model according to first loss between each generated first reconstruction image and the three-dimensional training image to obtain an initial image reconstruction model; acquiring a two-dimensional training image, wherein the two-dimensional training image comprises a second slice image sequence of an object to be reconstructed in different directions, the object shown by the three-dimensional training image is the same as the object to be reconstructed in type, and the resolution of the second slice image sequence is lower than that of the three-dimensional training image; respectively inputting each second slice image sequence in the two-dimensional training image into the initial image reconstruction model to generate a second reconstruction image; and respectively constraining each generated second reconstructed image according to the two-dimensional training image so as to adjust the parameters of the initial image reconstruction model and obtain a final image reconstruction model.
In some embodiments, the processing unit 910 may further perform the following steps of the above-described image reconstruction method embodiment: acquiring a slice image sequence of an object to be reconstructed; and inputting the slice image sequence into a pre-trained image reconstruction model, reconstructing a final three-dimensional image of the object to be reconstructed, wherein the image reconstruction model is obtained by training based on the method provided by the embodiment of the image reconstruction model training method.
The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 9201 and/or a cache memory unit 9202, and may further include a read only memory unit (ROM) 9203.
The electronic device 900 may also communicate with one or more external devices 940 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 900, and/or any device (e.g., router, modem, etc.) that enables the electronic device 900 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 950. Also, the electronic device 900 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 960. As shown in FIG. 9, the network adapter 960 communicates with the other modules of the electronic device 900 via the bus 930. It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, to name a few.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium, which may be a readable signal medium or a readable storage medium. Having stored thereon a program product capable of carrying out the methods of the present disclosure. In some possible embodiments, various aspects of the disclosure may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the disclosure described in the above-mentioned "exemplary methods" section of this specification, when the program product is run on the terminal device.
More specific examples of the computer-readable storage medium in the present disclosure may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
In the present disclosure, a computer readable storage medium may include a propagated data signal with readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Alternatively, program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
In particular implementations, program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Moreover, although the steps of the methods of the present disclosure are depicted in the drawings in a particular order, this does not require or imply that the steps must be performed in this particular order, or that all of the depicted steps must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions, etc.
Through the description of the above embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, and may also be implemented by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
Claims (12)
1. An image reconstruction model training method is characterized by comprising the following steps:
acquiring a three-dimensional training image and a first slice image sequence corresponding to the three-dimensional image in different directions, wherein the resolution of the first slice image sequence is lower than that of the three-dimensional training image;
respectively inputting the first slice image sequences in different directions into a preset image reconstruction model to generate a first reconstructed image;
training the preset image reconstruction model according to a first loss between each generated first reconstruction image and the three-dimensional training image to obtain an initial image reconstruction model;
acquiring a two-dimensional training image, wherein the two-dimensional training image comprises a second slice image sequence of an object to be reconstructed in different directions, the object shown by the three-dimensional training image is the same as the object to be reconstructed in type, and the resolution of the second slice image sequence is lower than that of the three-dimensional training image;
respectively inputting each second slice image sequence in the two-dimensional training image into the initial image reconstruction model to generate a second reconstruction image;
and respectively constraining each generated second reconstructed image according to the two-dimensional training image so as to adjust the parameters of the initial image reconstruction model and obtain a final image reconstruction model.
2. The method of claim 1, wherein the obtaining of the first sequence of slice images in the different directions comprises:
and respectively carrying out down-sampling treatment on the three-dimensional training images along different directions to obtain first slice image sequences in different directions.
3. The method of claim 2, wherein the resolution of the first sequence of slice images is the same as or similar to the resolution of the second sequence of slice images.
4. The method of claim 1, wherein the constraining each generated second reconstructed image according to the two-dimensional training image to adjust parameters of the initial image reconstruction model to obtain a final image reconstruction model comprises:
and respectively carrying out the following processing on each generated second reconstruction image to obtain the final image reconstruction model:
respectively calculating the loss between the second reconstructed image and the second slice image sequence in each direction in the different directions to obtain second losses respectively corresponding to the different directions;
and synthesizing the second losses respectively corresponding to the different directions, and adjusting the parameters of the initial image reconstruction model.
5. The method of claim 1, further comprising, before constraining each of the generated second reconstructed images separately from the two-dimensional training image to adjust parameters of the initial image reconstruction model to obtain a final image reconstruction model:
and performing a registration operation on the second slice image sequences in different directions to align pixels of the second slice image sequences in different directions in space.
6. The method of claim 4, wherein the synthesizing of the second losses corresponding to the different directions respectively adjusts parameters of the initial image reconstruction model, including:
calculating a smoothing loss between each pixel in the second reconstructed image;
and synthesizing the smooth loss and second losses respectively corresponding to the different directions, and adjusting parameters of the initial image reconstruction model.
7. The method according to claim 1, wherein the three-dimensional training image shows an object as an adult brain and the object to be reconstructed is a fetal brain.
8. An image reconstruction method, comprising:
acquiring a slice image sequence of an object to be reconstructed;
inputting the slice image sequence into a pre-trained image reconstruction model, and reconstructing a final three-dimensional image of the object to be reconstructed, wherein the image reconstruction model is obtained by training based on the method of any one of claims 1 to 7.
9. The method of claim 8, wherein, in the case that the sequence of slice images includes a sequence of slice images of the object to be reconstructed in multiple directions, the inputting the sequence of slice images into a pre-trained image reconstruction model, the reconstructing the final three-dimensional image of the object to be reconstructed includes:
respectively inputting the slice image sequences in the multiple directions into the image reconstruction model to obtain multiple initial three-dimensional images;
and fusing the plurality of initial three-dimensional images to obtain the final three-dimensional image.
10. The method according to claim 9, wherein said fusing said plurality of initial three-dimensional images to obtain said final three-dimensional image comprises:
for each first pixel in the final three-dimensional image, the following processing is performed:
and taking the pixel average value of a plurality of second pixels with the same positions as the first pixels in the plurality of initial three-dimensional images as the pixel value of the first pixels, thereby obtaining the final three-dimensional image.
11. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the method of any of claims 1 to 10 via execution of the executable instructions.
12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method of any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310162463.7A CN115965837A (en) | 2023-02-23 | 2023-02-23 | Image reconstruction model training method, image reconstruction method and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310162463.7A CN115965837A (en) | 2023-02-23 | 2023-02-23 | Image reconstruction model training method, image reconstruction method and related equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115965837A true CN115965837A (en) | 2023-04-14 |
Family
ID=87354911
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310162463.7A Pending CN115965837A (en) | 2023-02-23 | 2023-02-23 | Image reconstruction model training method, image reconstruction method and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115965837A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116152241A (en) * | 2023-04-18 | 2023-05-23 | 湖南炅旭生物科技有限公司 | Brain image processing method, system, electronic equipment and storage medium |
-
2023
- 2023-02-23 CN CN202310162463.7A patent/CN115965837A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116152241A (en) * | 2023-04-18 | 2023-05-23 | 湖南炅旭生物科技有限公司 | Brain image processing method, system, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11847781B2 (en) | Systems and methods for medical acquisition processing and machine learning for anatomical assessment | |
| US10803354B2 (en) | Cross-modality image synthesis | |
| CN104346821B (en) | Automatic planning for medical imaging | |
| JP2021521993A (en) | Image enhancement using a hostile generation network | |
| WO2019211307A1 (en) | Modality-agnostic method for medical image representation | |
| CN110338795B (en) | Radial golden angle magnetic resonance heart film imaging method, device and equipment | |
| US10867375B2 (en) | Forecasting images for image processing | |
| US20220139062A1 (en) | Learning apparatus, learning method, and learning program, region-of-interest extraction apparatus, region-of-interest extraction method, and region-of-interest extraction program, and learned extraction model | |
| US9355454B2 (en) | Automatic estimation of anatomical extents | |
| CN112005314A (en) | System and method for training a deep learning model of an imaging system | |
| CN115965837A (en) | Image reconstruction model training method, image reconstruction method and related equipment | |
| Advincula et al. | Development and future trends in the application of visualization toolkit (VTK): the case for medical image 3D reconstruction | |
| CN111243052A (en) | Image reconstruction method and device, computer equipment and storage medium | |
| CN112004471A (en) | System and method for imaging system shortcut mode | |
| US20220076808A1 (en) | External device-enabled imaging support | |
| US20230214664A1 (en) | Learning apparatus, method, and program, image generation apparatus, method, and program, trained model, virtual image, and recording medium | |
| CN112419175A (en) | Weight-sharing dual-region generation countermeasure network and image generation method thereof | |
| CN115761134A (en) | Twin model generation method, system, nuclear magnetic resonance device, and medium | |
| US20230245317A1 (en) | Low latency interactive segmentation of medical images within a web-based deployment architecture | |
| CN111919264A (en) | System and method for synchronizing an imaging system and an edge calculation system | |
| CN115908610A (en) | Method for obtaining attenuation correction coefficient image based on single-mode PET image | |
| CN115311191A (en) | Generating reformatted views of three-dimensional anatomical scans using deep learning estimated scan scheme masks | |
| CN111583354B (en) | Training method of medical image processing unit and medical image motion estimation method | |
| EP4231234A1 (en) | Deep learning for registering anatomical to functional images | |
| CN115546174B (en) | Image processing method, device, computing equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |