CN115701616A - Training method for generating countermeasure network, and C-arm machine image restoration method and device - Google Patents
Training method for generating countermeasure network, and C-arm machine image restoration method and device Download PDFInfo
- Publication number
- CN115701616A CN115701616A CN202110880853.9A CN202110880853A CN115701616A CN 115701616 A CN115701616 A CN 115701616A CN 202110880853 A CN202110880853 A CN 202110880853A CN 115701616 A CN115701616 A CN 115701616A
- Authority
- CN
- China
- Prior art keywords
- image
- repaired
- sub
- arm machine
- countermeasure network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Processing (AREA)
Abstract
The application provides a training method for generating a confrontation network, a C-arm machine image restoration method and a device, wherein the method comprises the following steps: constructing and generating a countermeasure network; acquiring a sample C arm machine image, and cutting the sample C arm machine image into a plurality of sample sub-images; generating a training data set according to the sample subimages, wherein the training data set comprises at least one pair of matched sample subimages and mask subimages, and the mask subimages are obtained by superposing mask patterns for simulating the circular shadows on the sample subimages; and training the generated countermeasure network by using the training data set to obtain the trained generated countermeasure network, wherein the trained generated countermeasure network is used for image restoration of the C-arm machine image to be restored. Through the technical scheme of this application, can reduce to the apparent memory space that the generation was required to the antagonistic network trains to can be based on the image restoration of the generation antagonistic network to C arm machine image of training realization with the effect of preferred after the training.
Description
Technical Field
The application relates to the technical field of medical image processing, in particular to a training method for generating a countermeasure network, and a C-arm machine image restoration method and device.
Background
Under the influence of the image shooting technology of the C-arm machine, a plurality of circular shadows may exist on the C-arm machine image, and the circular shadows often cause incomplete display of structures such as bones, muscles and the like in the image, thereby seriously influencing medical diagnosis made by a doctor based on the C-arm machine image.
At present, although the generation countermeasure network can be used for repairing the shadow part in the image, the image repairing method based on the generation countermeasure network consumes a large amount of memory space, and has a high requirement on the hardware environment, for example, a GPU with a memory size of 12G can only support image repairing of an image with a resolution of 256 × 256, whereas a C-arm image has a large resolution, and it is difficult to perform image repairing of the C-arm image through the generation countermeasure network.
Disclosure of Invention
In view of the above, the present application provides a training method for generating an antagonistic network, a C-arm machine image restoration method and an apparatus.
Specifically, the method is realized through the following technical scheme:
according to a first aspect of the present application, a training method for generating an antagonistic network for C-arm machine image inpainting is provided, including:
constructing and generating a countermeasure network; the generation countermeasure network is used for repairing the input original image with the circular shadow and outputting a repaired image with the circular shadow removed;
acquiring a sample C arm machine image, and cutting the sample C arm machine image into a plurality of sample sub-images;
generating a training data set according to the sample subimages, wherein the training data set comprises at least one pair of matched sample subimages and mask subimages, and the mask subimages are obtained by superposing mask patterns for simulating the circular shadows on the sample subimages;
and training the generated countermeasure network by using the training data set to obtain the trained generated countermeasure network, wherein the trained generated countermeasure network is used for image restoration of the C-arm machine image to be restored.
According to a second aspect of the present application, a C-arm machine image restoration method based on generation of a countermeasure network is provided, including:
acquiring a C-arm machine image to be repaired;
cutting the C-arm machine image to be repaired to obtain a plurality of sub-images to be repaired;
inputting the subimage to be repaired into the generation countermeasure network, and processing the subimage to be repaired through the generation countermeasure network to obtain a repaired subimage, wherein the generation countermeasure network is obtained by training through the training method for generating the countermeasure network in any one of the first aspect;
and splicing the repaired sub-images to obtain a complete repaired image.
According to a third aspect of the present application, there is provided a training apparatus for generating an antagonistic network for C-arm machine image inpainting, comprising:
a network construction unit for constructing a generation countermeasure network; the generation countermeasure network is used for repairing the input original image with the circular shadow and outputting a repaired image with the circular shadow removed;
the system comprises a sample acquisition unit, a data processing unit and a data processing unit, wherein the sample acquisition unit is used for acquiring a sample C-arm machine image and cutting the sample C-arm machine image into a plurality of sample sub-images;
a data set generating unit, configured to generate a training data set according to the sample sub-images, where the training data set includes at least one pair of matched sample sub-images and mask sub-images, and the mask sub-images are obtained by superimposing mask patterns for simulating the circular shadows on the sample sub-images;
and the network training unit is used for training the generated confrontation network by utilizing the training data set to obtain the trained generated confrontation network, and the trained generated confrontation network is used for image restoration of the C-arm machine image to be restored.
According to a fourth aspect of the present application, a C-arm machine image restoration device based on a generation countermeasure network is provided, including:
the image acquisition unit is used for acquiring a C-arm machine image to be repaired;
the image cutting unit is used for cutting the C-arm machine image to be repaired to obtain a plurality of sub-images to be repaired;
the image restoration unit is configured to input the sub-image to be restored into the generation countermeasure network, so as to process the sub-image to be restored through the generation countermeasure network to obtain a restored sub-image, where the generation countermeasure network is obtained by training the training method for generating the countermeasure network according to any one of the first aspect;
and the image splicing unit is used for splicing the repaired sub-images to obtain a complete repaired image.
According to a fifth aspect of the present application, there is provided an electronic device comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor implements the method as described in the embodiments of the first and second aspects above by executing the executable instructions.
According to a fourth aspect of embodiments of the present application, there is provided a computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the method as described in the embodiments of the first and second aspects above.
According to the technical scheme, the method and the device have the advantages that the high-resolution sample C-arm machine image is cut, the training of generation of the countermeasure network is achieved through the cut sub-images, the C-arm machine image to be repaired is cut into the plurality of sub-images in the image repairing process, the image repairing is conducted on each sub-image, the repaired sub-images are spliced, and accordingly the repaired image of the whole C-arm machine image can be obtained. The technical scheme of the application enables training of the generation countermeasure network for repairing the C-arm machine image with large resolution to be possible, and image repairing with good effect can be achieved on the C-arm machine image based on the generation countermeasure network after training.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.
FIG. 1 is a flow diagram illustrating a training method for generating a confrontational network for C-arm machine image inpainting according to an illustrative embodiment of the present application;
FIGS. 2a-2b are a pair of matched sample sub-image and mask sub-image, according to an exemplary embodiment of the present application;
FIG. 3 is a flowchart illustrating a C-arm machine image inpainting method based on generation of a countermeasure network according to an exemplary embodiment of the present application;
FIG. 4 is a schematic diagram illustrating image cropping of a sub-image to be repaired according to an exemplary embodiment of the present application;
FIG. 5 is a schematic diagram illustrating a portion for image stitching in a repaired sub-image to be repaired according to an exemplary embodiment of the present application;
FIG. 6 is a schematic diagram illustrating image stitching according to an exemplary embodiment of the present application;
7a-7b illustrate an image to be replaced and corresponding sub-images to be repaired according to an exemplary embodiment of the present application;
FIG. 8 is a schematic diagram illustrating a training electronic device for generating a confrontational network for C-arm machine image inpainting according to an illustrative embodiment of the present application;
FIG. 9 is a block diagram illustrating a training apparatus for generation of a countermeasure network for C-arm machine image inpainting according to an exemplary embodiment of the present application;
FIG. 10 is a schematic diagram illustrating a C-arm machine image inpainting electronic device based on a generative confrontation network according to an illustrative embodiment of the present application;
fig. 11 is a block diagram of a C-arm machine image restoration device based on a generation countermeasure network according to an exemplary embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at" \8230; "or" when 8230; \8230; "or" in response to a determination ", depending on the context.
Next, examples of the present application will be described in detail.
Fig. 1 is a flowchart illustrating a training method for generating a countermeasure network for C-arm machine image inpainting according to an exemplary embodiment of the present application. As shown in fig. 1, the method may include the steps of:
step 102: constructing and generating a countermeasure network; the generation countermeasure network is used for repairing the input original image with the circular shadow and outputting a repaired image with the circular shadow removed.
The generation countermeasure network is a deep learning model and comprises a generation network and a discrimination network, wherein the generation network is used for generating random samples similar to real samples and deceiving the random samples as false samples to the discrimination network; the judgment network is used for judging the content output by the generation network and learning to distinguish the real sample from the false sample generated by the generation network. The generation of the confrontation network utilizes the mutual game between the generation network and the discrimination network to continuously improve the network performance, and finally balance is achieved, so that a high-quality generation network capable of generating a real sample and a discrimination network with strong judgment capability are obtained.
In the technical scheme of the application, based on the self characteristic that a plurality of circular shadows are usually formed in a C-arm machine image, a generation countermeasure network is constructed to repair an input original image with the circular shadows and output a repaired image with the circular shadows removed, so that substitute contents of the circular shadows are synthesized from a given C-arm machine image to be repaired, the repaired C-arm machine image is made to be visually vivid and semantically reasonable, wherein the specific construction mode for generating the countermeasure network can refer to records in related technologies, and the generation countermeasure network is not described herein any more.
In an embodiment, since the generation countermeasure network for image restoration in the related art is generally provided with three RGB channel inputs, and the C-arm machine image to be restored in the present application can be regarded as a grayscale image, the input for generating the countermeasure network can be constructed as a single channel when the generation countermeasure network is constructed, so as to input the C-arm machine image to be restored in the grayscale mode. Similarly, the generation countermeasure network of the three-channel input can be directly modified, the network parameters are adjusted, and the input channel is modified from the RGB three-channel to the single channel. This application only leaves the required single channel of C arm machine image through setting up the input channel who generates the confrontation network, can reduce unnecessary data and calculate, improves the training speed that generates the confrontation network to can reduce two thirds's apparent memory demand, reduce to generate and confront the requirement of network training to hardware environment.
Step 104: and acquiring a sample C-arm machine image, and cutting the sample C-arm machine image into a plurality of sample sub-images.
Wherein the sample C-arm machine image is a pre-collected perfect shadow-free high-quality C-arm machine image. As described above, due to the hardware environment of the generation countermeasure network, the GPU cannot generally train the entire C-arm machine image, and therefore, for each sample C-arm machine image, it needs to be cropped into a sub-image with a smaller size until it can be processed by the GPU. In an embodiment, the resolution size of the sample sub-image to be cropped may be set based on hardware attributes of the electronic device performing the generation of the anti-network training. For example, if the maximum image resolution that the generation countermeasure network can train is 512 × 512 and the sample C-arm machine image resolution is 1024 × 1024, the sample C-arm machine image can be clipped along two central lines of the horizontal and vertical sides to be equally divided into 4 sample sub-images 512 × 512. On one hand, the generated countermeasure network can support the training process of repairing the C-arm machine image with large resolution by cutting the sample C-arm machine image; on the other hand, even if the GPU video memory size of the electronic device performing the generation countermeasure network training is enough to support the processing of the whole sample C-arm machine image, the occupation of the video memory space in the subsequent training process can be reduced by clipping the sample C-arm machine image, and taking the example of clipping the sample C-arm machine image with the resolution of 1024 × 1024 into the sample sub-image with the resolution of 512 × 512, the video memory usage of three quarters can be reduced in the subsequent training process.
Step 106: and generating a training data set according to the sample subimages, wherein the training data set comprises at least one pair of matched sample subimages and mask subimages, and the mask subimages are obtained by superposing mask patterns for simulating the circular shadows on the sample subimages.
In generating the countermeasure network, the training data set includes, in addition to the sample sub-images obtained in step 104 as real samples, images to be repaired containing shadows that enable the generation network to process to obtain false samples. In the technical scheme of the application, the shadow in the C-arm machine image can be replaced by the mask pattern, the mask image for simulating the shadow of the C-arm machine image is superposed on the sample sub-image to obtain the mask sub-image, so that the generation network can process the mask sub-image in the training process, the repaired mask sub-image is used as a false sample, and the judgment network needs to distinguish the repaired mask sub-image from the sample sub-image. The training data set required for generating the countermeasure network can be obtained by processing the sample subimages, and the training data set at least comprises a group of image pairs, and each group of image pairs comprises a sample subimage and a matched mask subimage.
In one embodiment, the mask patterns superimposed on the sample sub-images are not completely random because the shadow to be repaired on the C-arm machine image has the characteristics of being circular and substantially the same size. For any sample sub-image, the mask patterns superimposed thereon are all circular and have the same size. Since the size and the position of the shadow to be repaired on the C-arm machine image are not fixed, the size and the position of the mask pattern superimposed on different sample sub-images are random when the mask sub-images are generated. The convergence rate of the generation of the countermeasure network can be increased by generating the mask pattern according to a specific rule to replace the shadow in the image to reduce the randomness of the shadow to be repaired.
Further, since the positions of the circular shadows in the C-arm machine image to be restored are not regularly distributed, when the C-arm machine image is cut, the cutting path may pass through a certain circular shadow to cut the circular shadow into two halves. Thus, the present application can superimpose several semicircular mask patterns at the edges of the sample sub-image to simulate the above situation. Fig. 2a and 2b are a pair of matched sample sub-images and mask sub-images, where fig. 2a is a sample sub-image, fig. 2b is a mask sub-image after a mask sub-pattern for simulating a circle is superimposed, and a white dot in fig. 2b is a superimposed mask pattern generated according to a specific rule. By superposing the semicircular mask patterns on the edges of the sample sub-images to simulate the shadow state which may appear after cutting, the restoration effect of the C-arm machine image to be restored through the generation countermeasure network can be improved.
Step 108: and training the generated countermeasure network by using the training data set to obtain the trained generated countermeasure network, wherein the trained generated countermeasure network is used for image restoration of the C-arm machine image to be restored.
The method is the same as the training method for generating the countermeasure network in the related technology, the generation of the countermeasure network comprises the generation network and the judgment network, the mask subimage is input into the generation network to obtain the repaired mask subimage, the repaired mask subimage and the corresponding sample subimage are used as two inputs of the judgment network, and the judgment network feeds back the judgment of the real data and the data generated by the generation network to the generation network. After receiving the judgment feedback, the generation network generates false data better than the previous time again, the judgment network judges the false data generated by the generation network again by learning the true data and the false data of the previous time, and the training is finished until the loss function is converged in such a way of continuous iterative optimization to obtain the trained generation confrontation network.
In an embodiment, the generation of the countermeasure network may be pre-trained prior to formal training of the generation of the countermeasure network. Inputting a pair of sample sub-images and mask sub-images into a generation network for pre-training, and training by using the following loss:
in formula (1), X represents maskless raw data input to a model, G (X) represents an output after input to the model with a mask,
representing pixel-by-pixel multiplication and MASK representing the MASK layer. The method can generate pre-training of the countermeasure network based on the loss function, save the trained model parameters as initialization parameters of the countermeasure network, and train the generation network through the pre-training generation network and the randomly initialized discrimination network to generate the countermeasure network, so that the convergence rate of the generation countermeasure network is effectively improved.
According to the technical scheme, the high-resolution sample C-arm machine image is cut, and the training for generating the countermeasure network is realized by utilizing the cut sub-images, so that the training for repairing the large-resolution C-arm machine image to generate the countermeasure network becomes possible, and the video memory space consumed by the production of the countermeasure network training is effectively reduced.
Fig. 3 is a flowchart illustrating a C-arm machine image restoration method based on a generation countermeasure network according to an exemplary embodiment of the present application. As shown in fig. 3, the method may include the steps of:
step 302: and acquiring an image of the C-arm machine to be repaired.
Step 304: and cutting the C-arm machine image to be repaired to obtain a plurality of sub-images to be repaired.
As described above, due to the hardware environment of the generation countermeasure network, the GPU cannot generally perform image restoration on the entire C-arm machine image, and the generation countermeasure network may have a better restoration effect when restoring an image having the same size as the training data set thereof. Therefore, for the C-arm machine image to be repaired, it needs to be cropped to a smaller size corresponding to the generation of the sub-image to be repaired at the image resolution set by the countermeasure network. For example, if the image resolution set by the generation countermeasure network is 512 × 512 and the image resolution of the C-arm machine to be repaired is 1024 × 1024, the C-arm machine to be repaired may be clipped along two central lines of the horizontal and vertical axes of the C-arm machine to be repaired, and the C-arm machine to be repaired may be divided into 4 sub-images to be repaired by 512 × 512 on average. The image of the C-arm machine to be restored is cut into the sub-image to be restored with a smaller size, so that the image restoration can be performed through the generation countermeasure network trained in advance.
In an embodiment, the C-arm machine image to be repaired may be cropped multiple times, considering that the circular shadow in the C-arm machine image to be repaired may be cropped into two sub-images to be repaired right on the cropping path, which may cause image discontinuity in the final stitching. Firstly, according to the image resolution set by the adopted generation countermeasure network, cutting a C-arm machine image to be repaired into a plurality of first-class sub-images to be repaired; secondly, re-cropping the C-arm machine image to be repaired according to the image resolution set by the adopted generation countermeasure network aiming at the cropping path during the first cropping to obtain at least one second type of sub-image to be repaired, wherein the second type of sub-image to be repaired covers the edge handover area of a plurality of adjacent first type of sub-images to be repaired; furthermore, the C-arm machine image to be repaired can be re-cut according to the image resolution set by the adopted generation countermeasure network to obtain a plurality of third-class sub-images to be repaired, and the third-class sub-images to be repaired cover the vertex handover areas of the adjacent first-class sub-images to be repaired. For example, the image resolution set by the generation countermeasure network for C-arm machine image restoration is 512 × 512, as shown in fig. 4, the image is a schematic diagram of image cropping of a C-arm machine image to be restored with a resolution of 1024 × 1024, and an image in a dotted line is a sub-image to be restored obtained by cropping. The sub-images 401, 403, 407 and 409 to be repaired are 4 first-class sub-images to be repaired which are obtained by cutting along the horizontal and vertical central lines of the C-arm machine to be repaired; the sub-images to be repaired 402, 404, 406 and 408 are 4 second sub-images to be repaired covering the joint area of the edges of the two images of the first type of sub-images to be repaired, which are obtained by rescaling the first clipping path, namely the horizontal and vertical central lines; the sub-image to be repaired 405 is a third type sub-image to be repaired covering the vertex intersection area of the first type sub-image to be repaired.
Step 306: and inputting the subimage to be repaired into the generation countermeasure network, and processing the subimage to be repaired through the generation countermeasure network to obtain a repaired subimage.
The generation of the countermeasure network is trained by the training method for generating the countermeasure network according to any one of the embodiments. In the present application, the specific implementation manner of repairing the image by using the trained generated countermeasure network may refer to the records in the related art, and the details are not repeated in the present application again.
Step 308: and splicing the repaired sub-images to obtain a complete repaired image.
In an embodiment, the first type of sub-image to be repaired, the second type of sub-image to be repaired, and the third type of sub-image to be repaired after the repair may be respectively spliced. Firstly, splicing the repaired first type of subimages to be repaired to obtain a first spliced image; then, the repaired second type of subimages to be repaired can be spliced and replace the corresponding content in the first spliced image to obtain a second spliced image; and finally, replacing the corresponding content in the second spliced image by the repaired third-class subimage to be repaired to obtain a complete repaired image.
Further, for the repaired second type sub-image to be repaired and the repaired third type sub-image to be repaired, only part of the content in the image may be selected for splicing and replacing. The repaired second type of subimage to be repaired can be cut into a first type of image to be replaced, the width of which is twice the diameter of the circular shadow in the C-arm machine image to be repaired, and the part, corresponding to the edge junction area of the first type of subimage to be repaired, in the first spliced image is replaced by the first type of image to be replaced; and cutting the repaired third type of subimage to be repaired into a second type of image to be replaced, wherein the image width and the image length of the second type of image to be replaced are two times of the diameter of the circular shadow in the C-arm machine image to be repaired, and replacing the part, corresponding to the vertex connection area of the first type of subimage to be repaired, of the first spliced image with the second type of image to be replaced. Taking the image to be repaired shown in fig. 4 as an example, for the second type of sub-images to be repaired 402, 404, 406, and 408 and the third type of sub-images to be repaired 405, part of the content in the repaired image can be selected for image splicing, as shown in fig. 5, the shaded part represents the part of the repaired sub-images to be repaired that will be used for image splicing, where 501, 503, 507, and 509 are the first type of repaired sub-images, 502, 504, 506, and 508 are the first type of images to be replaced, and 505 is the second type of images to be replaced. Taking the repaired first-class sub-image to be repaired and the repaired image to be replaced shown in fig. 5 as an example, and fig. 6 is an image splicing schematic diagram, where the repaired first-class sub-image to be repaired 501, 503, 507, and 509 can be used as a bottom layer to be spliced to obtain a first spliced image with a resolution of 1024 × 1024, but the spliced first spliced image may have a fuzzy discontinuity at a spliced joint portion, so that the first-class sub-image to be repaired and the second-class sub-image to be replaced need to be supplemented, the first-class sub-image to be replaced 502, 504, 506, and 508 are respectively spliced and covered at corresponding positions on the first spliced image and replace covered contents thereof to obtain a second spliced image, and the second-class sub-image to be replaced 505 is covered at a corresponding position on the second spliced image and replace covered contents thereof, thereby obtaining a final complete repaired image.
In an embodiment, although the cropping radius of the image to be replaced may be set by a user according to the size of the circular shadow in the image of the C-arm machine to be repaired, the image edges of the first type of image to be replaced and the second type of image to be replaced after the cropping are obtained by repairing the incomplete circular shadow, so the position information of the circular shadow in the image of the C-arm machine to be repaired may be determined in a hough transform manner, and the like, where the determination manner of the position information of the circular shadow may refer to related records in the related art, and this application is not described herein again. The images for splicing can be processed in a targeted manner through the determined position information of the circular shadows, so that the first type of images to be replaced only contain complete circular shadows in the corresponding areas of the second type of sub-images to be repaired, and the second type of images to be replaced only contain complete circular shadows in the corresponding areas of the third type of sub-images to be repaired. Fig. 7a is a to-be-repaired sub-image where an image to be repaired is located, as shown in the figure, the edge of the to-be-repaired sub-image includes an incomplete circular shadow, so that the content of the to-be-replaced image can be replaced by the image at the bottom layer of the incomplete circular shadow portion in the image stitching process in a manner of cutting the position of the incomplete circular shadow, for example, the to-be-replaced image after the incomplete circular shadow portion is cut and removed in fig. 7b, and when the image is stitched through the image, the content after the complete circular shadow corresponding to the lower layer is repaired can be displayed at the position of the incomplete circular shadow, so that the final repairing effect of the complete repaired image is improved.
According to the technical scheme, the C-arm machine image to be repaired is cut into the plurality of sub-images, the sub-images are respectively subjected to image repairing through the generation of the countermeasure network, the repaired sub-images are spliced to obtain the repaired image of the whole C-arm machine image, and therefore the C-arm machine image repairing based on the generation of the countermeasure network is achieved, and a good image repairing effect can be obtained.
Corresponding to the method embodiments, the present specification also provides an embodiment of an apparatus.
Fig. 8 is a schematic diagram illustrating a training electronic device for generating a countermeasure network for C-arm machine image inpainting according to an exemplary embodiment of the present application. Referring to fig. 8, at the hardware level, the electronic device includes a processor 802, an internal bus 804, a network interface 806, a memory 808, and a non-volatile memory 810, although it may also include hardware required for other services. The processor 802 reads the corresponding computer program from the non-volatile memory 810 into the memory 808 and then runs. Of course, besides the software implementation, the present application does not exclude other implementations, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or logic devices.
FIG. 9 is a block diagram illustrating a training apparatus for generation of a countermeasure network for C-arm machine image inpainting according to an exemplary embodiment of the present application. Referring to fig. 9, the apparatus comprises a network construction unit 902, a sample acquisition unit 904, a data set generation unit 906, and a network training unit 908, wherein:
the network construction unit 902 is configured to construct a generative confrontation network; the generation countermeasure network is used for repairing the input original image with the circular shadow and outputting a repaired image with the circular shadow removed.
Optionally, the input of the generation countermeasure network is a single channel, and is used for inputting the C-arm machine image to be repaired in the grayscale mode.
The sample acquisition unit 904 is configured to acquire a sample C-arm machine image and crop the sample C-arm machine image into a number of sample sub-images.
The dataset generation unit 906 is configured to generate a training dataset including at least one pair of matched sample sub-images and mask sub-images resulting from the sample sub-images superimposing a mask pattern for simulating the circular shading, from the sample sub-images.
Optionally, the mask pattern adopted by any mask sub-image includes at least one of: a circular mask, a semicircular mask having the same radius as the circular mask and located at an edge in the any mask sub-image.
The network training unit 908 is configured to train the generated confrontation network with the training data set, resulting in a trained generated confrontation network, where the trained generated confrontation network is used for image inpainting of the C-arm machine image to be inpainted.
Optionally, the apparatus further comprises:
a pre-training unit 910 is configured to input a pair of the sample sub-images and the mask sub-images into the generation network to pre-train the generation network.
The parameter initialization unit 912 is configured to use the pre-trained model parameters as initialization parameters of the generation network.
The implementation process of the functions and actions of each unit in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.
Fig. 10 is a schematic structural diagram of an electronic device for repairing a C-arm machine image based on a generation countermeasure network according to an exemplary embodiment of the present application. Referring to fig. 10, at the hardware level, the electronic device includes a processor 1002, an internal bus 1004, a network interface 1006, a memory 1008, and a non-volatile memory 1010, although it may include hardware required for other services. The processor 1002 reads a corresponding computer program from the nonvolatile memory 1010 into the memory 1008 and then runs the program. Of course, besides the software implementation, the present application does not exclude other implementations, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or logic devices.
Fig. 11 is a block diagram of a training apparatus for generating a confrontational network for C-arm machine image inpainting according to an exemplary embodiment of the present application. Referring to fig. 11, the apparatus includes an image acquisition unit 1102, an image cropping unit 1104, an image inpainting unit 1106, and an image stitching unit 1108, in which:
the image acquisition unit 1102 is configured to acquire a C-arm machine image to be repaired.
The image cropping unit 1104 is configured to crop the C-arm machine image to be repaired to obtain a plurality of sub-images to be repaired.
The image repairing unit 1106 is configured to input the sub-image to be repaired into the generation countermeasure network, so as to process the sub-image to be repaired through the generation countermeasure network to obtain a repaired sub-image, where the generation countermeasure network is obtained by training the training apparatus for generating a countermeasure network according to any of the embodiments.
The image stitching unit 1108 is configured to stitch the repaired sub-images to obtain a complete repaired image.
Optionally, the cutting the image of the C-arm machine to be repaired to obtain a plurality of sub-images to be repaired includes: according to the image resolution set by the generation countermeasure network, the C-arm machine image to be repaired is cut into a plurality of first sub-images to be repaired; re-cutting the C-arm machine image to be repaired according to the image resolution to obtain at least one second type of sub-image to be repaired, wherein the second type of sub-image to be repaired covers the edge joint area of a plurality of adjacent first type of sub-images to be repaired; and re-cutting the C-arm machine image to be repaired according to the image resolution to obtain a plurality of third sub-images to be repaired, wherein the third sub-images to be repaired cover the vertex connection area of a plurality of adjacent first sub-images to be repaired.
Optionally, the splicing the repaired sub-images to obtain a complete repaired image includes: splicing the repaired first type of subimages to be repaired to obtain a first spliced image; replacing corresponding content in the first spliced image by the repaired second type of subimages to be repaired to obtain a second spliced image; and replacing the corresponding content in the second spliced image by using the repaired third type of subimage to be repaired to obtain a complete repaired image.
Optionally, the replacing, by the repaired second-class sub-image to be repaired, the corresponding content in the first stitched image to obtain a second stitched image includes: and cutting the repaired second type of subimage to be repaired into a first type of image to be replaced, and replacing a part, corresponding to the edge cross-connecting area, of the first spliced image with the first type of image to be replaced, wherein the image width of the first type of image to be replaced is twice of the diameter of a circular shadow in the C-arm machine image to be repaired.
Optionally, the replacing, by the repaired third type of sub-image to be repaired, the corresponding content in the second stitched image to obtain a complete repaired image includes: and cutting the repaired third type of subimage to be repaired into a second type of image to be replaced, and replacing the part, corresponding to the vertex junction area, of the first spliced image with the second type of image to be replaced, wherein the image width and the image length of the second type of image to be replaced are two times of the diameter of the circular shadow in the C-arm machine image to be repaired.
Optionally, the first type of image to be replaced only contains a complete circular shadow in a corresponding area of the second type of sub-image to be repaired; and the second type of image to be replaced only contains complete circular shadow in the corresponding area of the third type of sub-image to be repaired.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the application. One of ordinary skill in the art can understand and implement it without inventive effort.
In an exemplary embodiment, there is also provided a non-transitory computer readable storage medium, e.g., a memory, comprising instructions executable by a processor of a C-arm machine image inpainting apparatus based on generation of a countermeasure network to implement a method as described in any of the above embodiments, such as the method may comprise:
acquiring a C-arm machine image to be repaired;
cutting the C-arm machine image to be repaired to obtain a plurality of sub-images to be repaired;
inputting the sub-image to be repaired into the generation countermeasure network, and processing the sub-image to be repaired through the generation countermeasure network to obtain a repaired sub-image, wherein the generation countermeasure network is obtained by training through any one of the training methods for generating the countermeasure network;
and splicing the repaired sub-images to obtain a complete repaired image.
The non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc., which is not limited in this application.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (13)
1. A training method for generating a confrontation network for C-arm machine image inpainting, the method comprising:
constructing and generating a confrontation network; the generation countermeasure network is used for repairing the input original image with the circular shadow and outputting a repaired image with the circular shadow removed;
acquiring a sample C-arm machine image, and cutting the sample C-arm machine image into a plurality of sample sub-images;
generating a training data set according to the sample subimages, wherein the training data set comprises at least one pair of matched sample subimages and mask subimages, and the mask subimages are obtained by superposing mask patterns for simulating the circular shadows on the sample subimages;
and training the generated countermeasure network by using the training data set to obtain the trained generated countermeasure network, wherein the trained generated countermeasure network is used for image restoration of the C-arm machine image to be restored.
2. The method of claim 1, wherein the input for generating the countermeasure network is a single channel for inputting the C-arm machine image to be repaired in a grayscale mode.
3. The method of claim 1, wherein the mask pattern used for any sub-image of the mask comprises at least one of:
a circular mask, a semicircular mask having the same radius as the circular mask and located at an edge in the any mask sub-image.
4. The method of claim 1, wherein generating the countermeasure network comprises generating a network and a discriminant network, and wherein the method further comprises:
inputting the pair of sample subimages and the mask subimages into the generation network to pre-train the generation network;
and taking the model parameters after pre-training as initialization parameters of the generation network.
5. A C-arm machine image restoration method based on a generation countermeasure network is characterized by comprising the following steps:
acquiring a C-arm machine image to be repaired;
cutting the C-arm machine image to be repaired to obtain a plurality of sub-images to be repaired;
inputting the sub-image to be repaired into the generation countermeasure network, and processing the sub-image to be repaired through the generation countermeasure network to obtain a repaired sub-image, wherein the generation countermeasure network is obtained by training according to the training method for generating the countermeasure network, which is disclosed by any one of claims 1-4;
and splicing the repaired sub-images to obtain a complete repaired image.
6. The method according to claim 5, wherein the cropping the image of the C-arm machine to be repaired to obtain a plurality of sub-images to be repaired comprises:
according to the image resolution set by the generation countermeasure network, the C-arm machine image to be repaired is cut into a plurality of first sub-images to be repaired;
re-cutting the C-arm machine image to be repaired according to the image resolution to obtain at least one second type of sub-image to be repaired, wherein the second type of sub-image to be repaired covers the edge joint area of a plurality of adjacent first type of sub-images to be repaired;
and re-cutting the C-arm machine image to be repaired according to the image resolution to obtain a plurality of third sub-images to be repaired, wherein the third sub-images to be repaired cover the vertex connection area of a plurality of adjacent first sub-images to be repaired.
7. The method according to claim 6, wherein said stitching the repaired sub-images to obtain a complete repaired image comprises:
splicing the repaired first type of subimages to be repaired to obtain a first spliced image;
replacing corresponding content in the first spliced image by the repaired second type of subimages to be repaired to obtain a second spliced image;
and replacing the corresponding content in the second spliced image by using the repaired third type of subimage to be repaired to obtain a complete repaired image.
8. The method of claim 7,
replacing corresponding content in the first spliced image by using the repaired second-class subimage to be repaired to obtain a second spliced image, wherein the method comprises the following steps: cutting the repaired second type of subimage to be repaired into a first type of image to be replaced, and replacing a part, corresponding to the edge cross-connecting area, of the first spliced image with the first type of image to be replaced, wherein the image width of the first type of image to be replaced is twice of the diameter of a circular shadow in the C-arm machine image to be repaired;
replacing corresponding content in the second spliced image by using the repaired third-class sub-image to be repaired to obtain a complete repaired image, wherein the method comprises the following steps: and cutting the repaired third type of subimage to be repaired into a second type of image to be replaced, and replacing the part, corresponding to the vertex junction area, of the first spliced image with the second type of image to be replaced, wherein the image width and the image length of the second type of image to be replaced are two times of the diameter of the circular shadow in the C-arm machine image to be repaired.
9. The method of claim 8,
the first type of image to be replaced only contains complete circular shadow in the corresponding area of the second type of sub-image to be repaired;
and the second type of image to be replaced only contains complete circular shadow in the corresponding area of the third type of sub-image to be repaired.
10. A training apparatus for generating a countermeasure network for C-arm machine image inpainting, the apparatus comprising:
a network construction unit for constructing a generation countermeasure network; the generation countermeasure network is used for repairing the input original image with the circular shadow and outputting a repaired image with the circular shadow removed;
the sample acquisition unit is used for acquiring a sample C-arm machine image and cutting the sample C-arm machine image into a plurality of sample sub-images;
a data set generating unit, configured to generate a training data set according to the sample sub-images, where the training data set includes at least one pair of matched sample sub-images and mask sub-images, and the mask sub-images are obtained by superimposing mask patterns for simulating the circular shadows on the sample sub-images;
and the network training unit is used for training the generated countermeasure network by using the training data set to obtain the trained generated countermeasure network, and the trained generated countermeasure network is used for image restoration of the C-arm machine image to be restored.
11. A C-arm machine image restoration device based on a generation countermeasure network, the device is characterized by comprising:
the image acquisition unit is used for acquiring a C-arm machine image to be repaired;
the image cutting unit is used for cutting the C-arm machine image to be repaired to obtain a plurality of sub-images to be repaired;
an image restoration unit, configured to input the sub-image to be restored into the generation countermeasure network, so as to process the sub-image to be restored through the generation countermeasure network to obtain a restored sub-image, where the generation countermeasure network is obtained by training according to the training method for generating a countermeasure network according to any one of claims 1 to 4;
and the image splicing unit is used for splicing the repaired sub-images to obtain a complete repaired image.
12. An electronic device, comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor implements the method of any one of claims 1-9 by executing the executable instructions.
13. A computer-readable storage medium having stored thereon computer instructions, which, when executed by a processor, carry out the steps of the method according to any one of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110880853.9A CN115701616A (en) | 2021-08-02 | 2021-08-02 | Training method for generating countermeasure network, and C-arm machine image restoration method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110880853.9A CN115701616A (en) | 2021-08-02 | 2021-08-02 | Training method for generating countermeasure network, and C-arm machine image restoration method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115701616A true CN115701616A (en) | 2023-02-10 |
Family
ID=85142439
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110880853.9A Pending CN115701616A (en) | 2021-08-02 | 2021-08-02 | Training method for generating countermeasure network, and C-arm machine image restoration method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115701616A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115860067A (en) * | 2023-02-16 | 2023-03-28 | 深圳华声医疗技术股份有限公司 | Method and device for training generation confrontation network, computer equipment and storage medium |
-
2021
- 2021-08-02 CN CN202110880853.9A patent/CN115701616A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115860067A (en) * | 2023-02-16 | 2023-03-28 | 深圳华声医疗技术股份有限公司 | Method and device for training generation confrontation network, computer equipment and storage medium |
| CN115860067B (en) * | 2023-02-16 | 2023-09-05 | 深圳华声医疗技术股份有限公司 | Method, device, computer equipment and storage medium for generating countermeasure network training |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8537155B2 (en) | Image processing apparatus and method | |
| US8629868B1 (en) | Systems and methods for simulating depth of field on a computer generated display | |
| CN109844819A (en) | System and method for dynamic barriers disposition | |
| Lee et al. | Real-time tracking of visually attended objects in virtual environments and its application to LOD | |
| CN111882627A (en) | Image processing method, video processing method, device, equipment and storage medium | |
| CN110009573B (en) | Model training method, image processing method, device, electronic equipment and storage medium | |
| CN108765520B (en) | Text information rendering method and device, storage medium and electronic device | |
| WO2013078404A1 (en) | Perceptual rating of digital image retouching | |
| KR20130089649A (en) | Method and arrangement for censoring content in three-dimensional images | |
| CN109255763A (en) | Image processing method, device, equipment and storage medium | |
| CN111047506A (en) | Environmental map generation and hole filling | |
| CN112884792B (en) | Lung image segmentation method and device, electronic equipment and storage medium | |
| US20220092840A1 (en) | Systems and Methods for Generating a Skull Surface for Computer Animation | |
| CN113657357B (en) | Image processing method, image processing device, electronic equipment and storage medium | |
| KR20210041155A (en) | GAN based training data including bad image generating apparatus and method therefor | |
| CN110916707A (en) | Two-dimensional bone image acquisition method, system and device | |
| CN115701616A (en) | Training method for generating countermeasure network, and C-arm machine image restoration method and device | |
| US20220207790A1 (en) | Image generation method and apparatus, and computer | |
| JP2013511109A (en) | Image processing method and apparatus therefor | |
| US10832420B2 (en) | Dynamic local registration system and method | |
| Chambe et al. | HDR-LFNet: Inverse tone mapping using fusion network | |
| CN113487475B (en) | Interactive image editing method, system, readable storage medium and electronic equipment | |
| CN116863044A (en) | Face model generation method and device, electronic equipment and readable storage medium | |
| US11120606B1 (en) | Systems and methods for image texture uniformization for multiview object capture | |
| CN110766079B (en) | Training data generation method and device for screen abnormal picture detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |