CN111462000A - Image recovery method and device based on pre-training self-encoder - Google Patents
Image recovery method and device based on pre-training self-encoder Download PDFInfo
- Publication number
- CN111462000A CN111462000A CN202010188588.3A CN202010188588A CN111462000A CN 111462000 A CN111462000 A CN 111462000A CN 202010188588 A CN202010188588 A CN 202010188588A CN 111462000 A CN111462000 A CN 111462000A
- Authority
- CN
- China
- Prior art keywords
- image
- network model
- encoder
- self
- trained
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000011084 recovery Methods 0.000 title claims abstract description 146
- 238000012549 training Methods 0.000 title claims abstract description 120
- 238000000034 method Methods 0.000 title claims abstract description 60
- 239000000835 fiber Substances 0.000 claims description 23
- 239000013307 optical fiber Substances 0.000 claims description 20
- 238000004891 communication Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 9
- 238000012545 processing Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 description 15
- 239000007787 solid Substances 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Optical Communication System (AREA)
Abstract
The embodiment of the invention provides an image recovery method and device based on a pre-training self-encoder, wherein the method comprises the following steps: acquiring a speckle image; inputting the speckle image into an image recovery network model to obtain a recovered image; wherein the image recovery network model comprises a trained first encoder subnetwork and a first decoder subnetwork; initial parameters of the first decoder subnetwork are determined according to parameters in a pre-trained self-encoder network model, and the self-encoder network model is obtained through training according to a first training set; the image restoration network model is trained according to a second training set, and the second training set comprises a second sample original image and a sample speckle image. By pre-training the self-encoder network model and performing parameter initialization on the first decoder subnetwork in the image recovery network model by adopting the parameters in the trained self-encoder network model, the information of the training data set can be fully utilized, and the convergence speed of the network during training is accelerated.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an image restoration method and apparatus based on a pre-training self-encoder.
Background
In recent years, optical fibers are widely used in imaging systems, and compared with conventional optical imaging systems using lenses, optical fibers have the advantages of small diameter, low loss and low cost, and are ideal invasive light guide media. Compared with the common optical fiber, the multimode optical fiber allows multiple modes to be transmitted simultaneously, and has the characteristics of large core diameter and low coupling connection cost.
However, since the multimode fiber can transmit hundreds of guided wave modes, each mode has different phase velocity, which causes different image information to lose the original phase relation, and finally the output end outputs random speckle images. In fact, the image information is not lost in the process, the image information is subjected to complex coding, and therefore, the recovery of the original image information from the speckle image is a key technology of the multimode fiber imaging system.
At present, in order to recover original image information from speckle images, a main method is to use a Unet network in a convolutional neural network, the network structure of the Unet is designed for an image segmentation task, the whole network is in a U-shaped structure, jump layer connection is adopted, a shallow feature map is directly connected with a deep feature map, so that shallow feature information can be directly transmitted to the deep layer of the network, and the shrinkage data feature of a pooling layer is abandoned for reducing information loss.
The network structure of the Unet is designed for an image segmentation task, and although the network structure can also be applied to a multimode optical fiber image transmission system, a deep learning method taking the Unet as a core has a great limitation in solving the problem of multimode optical fiber imaging recovery. The concrete expression is as follows: the convergence rate of the model is slow, when the data set tends to be complex, excessive detail information can reduce the recovery effect of the image structure, and the reliability of image recovery is greatly limited due to the characteristics of slow convergence and low structural similarity.
Therefore, the existing method for recovering the original image information according to the speckle image transmitted by the multimode fiber has the technical problems of low model convergence speed and low image recovery quality.
Disclosure of Invention
The embodiment of the invention aims to provide an image recovery method and device based on a pre-training self-encoder, and aims to solve the technical problems that the existing method for recovering original image information according to speckle images transmitted by multimode fibers is low in model convergence speed and low in image recovery quality.
The specific technical scheme is as follows:
in order to achieve the above object, an embodiment of the present invention provides an image restoration method based on a pre-trained self-encoder, where the method includes:
acquiring a speckle image, wherein the speckle image is obtained by transmitting an original image through a multimode fiber;
inputting the speckle image into an image recovery network model to obtain a recovered image;
wherein the image recovery network model comprises a trained first encoder subnetwork and a first decoder subnetwork; initial parameters of the first decoder subnetwork are determined according to parameters in a pre-trained self-encoder network model, the self-encoder network model being trained according to a first training set, the first training set comprising: a first sample original image; the image recovery network model is obtained by training according to a second training set, the second training set comprises a second sample original image and a sample speckle image, and the sample speckle image is obtained by transmitting the second sample original image through a multimode fiber.
Optionally, the self-encoder network model is trained as follows:
acquiring an initial self-encoder network model and the first training set, wherein the initial self-encoder network model comprises a second encoder sub-network and a second decoder sub-network to be trained;
inputting the first sample original image into the initial self-encoder network model to obtain a first output image;
determining a loss value based on the first output image and the first sample raw image;
determining whether the self-encoder network model converges based on the loss value;
if not, updating the parameter values of the second encoder sub-network and the second decoder sub-network, and returning to a self-encoder network model after the first sample original image is input with the updated parameter values to obtain a first output image;
and if so, determining the current self-encoder network model as the trained self-encoder network model.
Optionally, the method further includes:
acquiring an initial image recovery network model, and initializing parameters of a first decoder sub-network in the initial image recovery network model by using parameters of a second decoder sub-network in the trained self-encoder network model to obtain the image recovery network model to be trained.
Optionally, the image recovery network model is trained according to the following method:
acquiring an image recovery network model to be trained and the second training set;
inputting the sample speckle image into the image recovery network model to be trained to obtain a second output image;
determining a second loss value based on the second output image and the second sample original image;
determining whether the image restoration network model converges based on the second loss value;
if not, updating the parameter value of the image recovery network model, and returning to the step of inputting the sample speckle image into the current image recovery network model to obtain a second output image;
and if so, determining the current image recovery network model as the trained image recovery network model.
In order to achieve the above object, an embodiment of the present invention further provides an image restoration apparatus based on a pre-trained self-encoder, where the apparatus includes:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring speckle images, and the speckle images are obtained by transmitting original images through multimode optical fibers;
the recovery module is used for inputting the speckle images into an image recovery network model to obtain recovered images; wherein the image recovery network model comprises a trained first encoder subnetwork and a first decoder subnetwork; initial parameters of the first decoder subnetwork are determined according to parameters in a pre-trained self-encoder network model, the self-encoder network model being trained according to a first training set, the first training set comprising: a first sample original image; the image recovery network model is obtained by training according to a second training set, the second training set comprises a second sample original image and a sample speckle image, and the sample speckle image is obtained by transmitting the second sample original image through a multimode fiber.
Optionally, the apparatus further comprises: a first training module to train the self-coder network model as follows:
acquiring an initial self-encoder network model and the first training set, wherein the initial self-encoder network model comprises a second encoder sub-network and a second decoder sub-network to be trained;
inputting the first sample original image into the initial self-encoder network model to obtain a first output image;
determining a loss value based on the first output image and the first sample raw image;
determining whether the self-encoder network model converges based on the loss value;
if not, updating the parameter values of the second encoder sub-network and the second decoder sub-network, and returning to a self-encoder network model after the first sample original image is input with the updated parameter values to obtain a first output image;
and if so, determining the current self-encoder network model as the trained self-encoder network model.
Optionally, the apparatus further comprises: an initialization module to:
acquiring an initial image recovery network model, and initializing parameters of a first decoder sub-network in the initial image recovery network model by using parameters of a second decoder sub-network in the trained self-encoder network model to obtain the image recovery network model to be trained.
Optionally, the apparatus further comprises: a second training module, configured to train the image recovery network model as follows:
acquiring an image recovery network model to be trained and the second training set;
inputting the sample speckle image into the image recovery network model to be trained to obtain a second output image;
determining a second loss value based on the second output image and the second sample original image;
determining whether the image restoration network model converges based on the second loss value;
if not, updating the parameter value of the image recovery network model, and returning to the step of inputting the sample speckle image into the current image recovery network model to obtain a second output image;
and if so, determining the current image recovery network model as the trained image recovery network model.
In order to achieve the above object, an embodiment of the present invention further provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing any image recovery method step based on the pre-training self-encoder when executing the program stored in the memory.
In order to achieve the above object, an embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements any of the above method steps.
The embodiment of the invention has the following beneficial effects:
the image recovery method and the image recovery device based on the pre-training self-encoder provided by the embodiment of the invention are applied to obtain the speckle images, wherein the speckle images are obtained by transmitting the original images through multimode optical fibers; inputting the speckle images into an image recovery network model to obtain recovered images, wherein the image recovery network model comprises a trained first encoder sub-network and a trained first decoder sub-network; initial parameters of the first decoder subnetwork are determined from parameters in a pre-trained self-encoder network model trained from a first training set comprising: a first sample original image; the image recovery network model is obtained by training according to a second training set, the second training set comprises a second sample original image and a sample speckle image, and the sample speckle image is obtained by transmitting the second sample original image through a multimode fiber. It can be seen that the first decoder subnetwork in the image recovery network model is initialized with parameters by pre-training the self-encoder network model and adopting the parameters in the trained self-encoder network model, so that the training data set information can be fully utilized, the convergence rate of the network during training is accelerated, and the image recovery quality can be significantly improved compared with the existing image recovery method.
Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flowchart of an image restoration method based on a pre-trained self-encoder according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating an image recovery network model according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating a training process of a self-encoder network model according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating a self-encoder network model according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating a training process of an image retrieval network model according to an embodiment of the present invention;
FIG. 6 is a schematic structural diagram of an image restoration apparatus based on a pre-trained self-encoder according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In order to solve the problems of low model convergence speed and low image recovery quality of the conventional method for recovering original image information according to speckle images transmitted by multimode optical fibers, the embodiment of the invention provides an image recovery method and device based on a pre-training self-encoder, electronic equipment and a computer-readable storage medium.
Referring to fig. 1, fig. 1 is a schematic flowchart of an image restoration method based on a pre-trained self-encoder according to an embodiment of the present invention, where the method includes the following steps:
s101: and acquiring a speckle image, wherein the speckle image is obtained by transmitting an original image through a multimode fiber.
In the embodiment of the invention, after the original image is transmitted by the multimode optical fiber, the image information loses the original phase relation and becomes the speckle image. The recovery of the original image from the speckle image is a key step in multimode fiber imaging systems.
The image restoration method based on the pre-training self-encoder provided by the embodiment of the invention can be used for restoring the original image according to the speckle image transmitted by the multimode fiber. The method can be applied to electronic equipment such as computers.
In this step, a speckle image can be acquired. As one example, the electronics can acquire a speckle image from the output of the image sensor.
S102: inputting the speckle image into an image recovery network model to obtain a recovered image; wherein the image recovery network model comprises a trained first encoder subnetwork and a first decoder subnetwork; initial parameters of the first decoder subnetwork are determined from parameters in a pre-trained self-encoder network model trained from a first training set comprising: a first sample original image; the image recovery network model is obtained by training according to a second training set, the second training set comprises a second sample original image and a sample speckle image, and the sample speckle image is obtained by transmitting the second sample original image through a multimode fiber.
In the embodiment of the invention, the neural network model is adopted for image restoration, the neural network model for restoring the speckle images is expressed as the image restoration network model, and the speckle images are input into the trained image restoration network model, so that the restored images output by the model can be obtained. The main improvement point of the present invention is the model of the neural network and the process of adding the pre-trained self-encoder and using the parameters in the pre-trained self-encoder to perform parameter initialization, which is specifically referred to below.
In an embodiment of the present invention, the structure of the image restoration network model can be seen in fig. 2, which includes a first encoder subnetwork and a first decoder subnetwork. Wherein the initial network parameters of the first decoder subnetwork are derived by pre-training a self-encoder network model. In the embodiment of the invention, before the image recovery network model is trained, the self-encoder network model is trained in advance, the self-encoder network model is trained according to a first training set, and the first training set comprises a first sample original image, namely the input of the self-encoder network model is the original image which is not transmitted by the multimode optical fiber. The purpose of pre-training the self-encoder network model is to initialize the model parameters of the first decoder subnetwork with the partial model parameters in the trained self-encoder network model.
The function of the self-encoder network model is to perform characterization learning on input information by using the input information as a learning target. Specifically, the self-encoder network model comprises an encoder part and a decoder part, wherein the encoder part is used for generating characteristic codes of images according to input images, and the decoder part is used for restoring the images according to the characteristic codes. By training the self-encoder network model, the image output from the self-encoder model is similar to the input image as much as possible, and the characteristic encoding generated by the encoder part can be understood to better reserve the information of the original input data, and the decoder part can also better recover the image information according to the characteristic encoding.
In the embodiment of the invention, the parameters in the trained self-encoder network model are used for carrying out parameter initialization on the first decoder subnetwork in the image recovery network model, and then the image recovery network model is trained on the basis.
The image recovery network model is obtained by training according to a second training set, the second training set comprises a second sample original image and a sample speckle image, and the sample speckle image is obtained by transmitting the second sample original image through a multimode fiber.
The specific training process of the image recovery network model can be seen in the following, and speckle images obtained after multimode fiber transmission are input into the trained image recovery network model, so that recovered images can be obtained.
Referring to fig. 2, a speckle image is input into a first encoder subnetwork in an image network recovery model, downsampling is completed through a convolutional layer to obtain a feature map, and the feature map is mapped into feature codes of the image through a full connection layer. The first decoder subnetwork restores the feature codes into a feature map through upsampling, and then the feature map is amplified to an original size to obtain a restored image, wherein the restored image is an original image which corresponds to the speckle image and is not transmitted through multimode optical fibers.
The image recovery method based on the pre-training self-encoder provided by the embodiment of the invention is applied to obtain the speckle images, wherein the speckle images are obtained by transmitting the original images through multimode optical fibers; inputting the speckle images into an image recovery network model to obtain recovered images, wherein the image recovery network model comprises a trained first encoder sub-network and a trained first decoder sub-network; initial parameters of the first decoder subnetwork are determined from parameters in a pre-trained self-encoder network model trained from a first training set comprising: a first sample original image; the image recovery network model is obtained by training according to a second training set, the second training set comprises a second sample original image and a sample speckle image, and the sample speckle image is obtained by transmitting the second sample original image through a multimode fiber. It can be seen that the first decoder subnetwork in the image recovery network model is initialized with parameters by pre-training the self-encoder network model and adopting the parameters in the trained self-encoder network model, so that the training data set information can be fully utilized, the convergence rate of the network during training is accelerated, and the image recovery quality can be significantly improved compared with the existing image recovery method.
In an embodiment of the present invention, referring to fig. 3, the self-encoder network model may be trained as follows, including the following specific steps:
s301: an initial self-encoder network model is obtained, and a first training set, the initial self-encoder network model including a second encoder subnetwork and a second decoder subnetwork to be trained.
Wherein the initial self-encoder network model is a self-encoder network model to be trained, and the structure of the self-encoder network model can be seen in fig. 4, and includes a second encoder sub-network and a second decoder sub-network.
S302: and inputting the first sample original image into an initial self-encoder network model to obtain a first output image.
In the embodiment of the invention, the training set of the self-encoder network model is the first sample original image. The training process is carried out iteratively, and in each iteration process, the first sample original image is input into the self-encoder network model, so that a first output image corresponding to the iteration process in the current round can be obtained.
S303: based on the first output image and the first sample raw image, a loss value is determined.
In the embodiment of the present invention, the loss value is obtained by using, but not limited to, Mean Squared Error (MSE) formula as the loss function.
S304: determining whether the self-encoder network model converges based on the loss value, and if not, executing S305; if so, S306 is executed.
S305: the parameter values of the second encoder subnetwork and the second decoder subnetwork are updated and a return is made to S302.
S306: and determining the current self-encoder network model as the trained self-encoder network model.
In the trained self-encoder network model, the second encoder sub-network can reserve data information in the feature codes generated according to the input images, and the second decoder sub-network can recover image information according to the feature codes.
Specifically, referring to fig. 4, the first original image of the sample is an image that is not transmitted through the multimode optical fiber, the sub-network of the second encoder completes downsampling through the convolutional layer to obtain a feature map, and the feature map is mapped into a feature code of the image through the full connection layer; the second decoder subnetwork can recover the original picture according to the characteristic coding.
In an embodiment of the present invention, the step of initializing parameters of the first decoder network model in the image recovery network model by using the trained model parameters in the self-encoder network model may specifically include:
and acquiring an initial image recovery network model, and initializing the parameters of a first decoder sub-network in the initial image recovery network model by using the parameters of a second decoder sub-network in the trained self-encoder network model to obtain the image recovery network model to be trained.
Specifically, the network structure of the self-encoder network model and the network structure of the image recovery network model may be the same, and the essence of the network structure includes the encoding network structure and the decoding network structure, so that the parameters of the first decoder sub-network in the initial image recovery network model may be initialized by using the parameters of the second decoder sub-network in the trained self-encoder network model, and the initialized image recovery network model serves as the image recovery network model to be trained.
The pre-trained second decoder sub-network has a function of recovering from the feature codes to the original images, so that the network parameters in the pre-trained second decoder sub-network are transferred to the first decoder sub-network in the image recovery network model, and the convergence speed of the image recovery network model training process can be remarkably increased.
In an embodiment of the present invention, referring to fig. 5, the image recovery network model may be trained according to the following method, which includes the following specific steps:
s501: and acquiring an image recovery network model to be trained and a second training set.
In the embodiment of the present invention, the training set used for training the image recovery network model is represented as a second training set, and may include a second sample original image and a sample speckle image, where the sample speckle image is obtained by transmitting the second sample original image through a multimode optical fiber.
S502: and inputting the sample speckle image into an image recovery network model to be trained to obtain a second output image.
S503: a second loss value is determined based on the second output image and the second sample original image.
S504: determining whether the image recovery network model converges based on the second loss value, and if not, executing S505; if so, S506 is executed.
S505: the parameter values of the image recovery network model are updated and the process returns to S502.
S506: and determining the current image recovery network model as the trained image recovery network model.
In the embodiment of the invention, the training process of the image recovery network model and the training process of the self-encoder network model are the same in nature and can be referred to each other. The difference lies in that the training sets of the two are different, the training set of the image recovery network model is an original image and a corresponding speckle image, and after the image recovery network training is finished, the speckle image is input, and then the recovered image can be output.
Corresponding to the image restoration method based on the pre-trained self-encoder provided by the embodiment of the present invention, an embodiment of the present invention further provides an image restoration apparatus based on the pre-trained self-encoder, and referring to fig. 6, the image restoration apparatus may include the following modules:
the acquiring module 601 is configured to acquire a speckle image, where the speckle image is obtained by transmitting an original image through a multimode optical fiber.
A restoration module 602, configured to input the speckle image into an image restoration network model to obtain a restored image; wherein the image recovery network model comprises a trained first encoder subnetwork and a first decoder subnetwork; initial parameters of the first decoder subnetwork are determined from parameters in a pre-trained self-encoder network model trained from a first training set comprising: a first sample original image; the image recovery network model is obtained by training according to a second training set, the second training set comprises a second sample original image and a sample speckle image, and the sample speckle image is obtained by transmitting the second sample original image through a multimode fiber.
In an embodiment of the present invention, on the basis of the apparatus shown in fig. 6, a first training module may be further included, where the first training module is configured to train the self-coder network model as follows:
acquiring an initial self-encoder network model and a first training set, wherein the initial self-encoder network model comprises a second encoder sub-network and a second decoder sub-network to be trained;
inputting the first sample original image into an initial self-encoder network model to obtain a first output image;
determining a loss value based on the first output image and the first sample raw image;
determining whether the self-encoder network model converges based on the loss value;
if not, updating the parameter values of the second encoder sub-network and the second decoder sub-network, and returning to the self-encoder network model after the first sample original image is input with the updated parameter values to obtain a first output image;
and if so, determining the current self-encoder network model as the trained self-encoder network model.
In an embodiment of the present invention, on the basis of the apparatus shown in fig. 6, an initialization module may further be included, and the initialization module is configured to:
and acquiring an initial image recovery network model, and initializing the parameters of a first decoder sub-network in the initial image recovery network model by using the parameters of a second decoder sub-network in the trained self-encoder network model to obtain the image recovery network model to be trained.
In an embodiment of the present invention, on the basis of the apparatus shown in fig. 6, a second training module may further be included, and the second training module is configured to train the image recovery network model as follows:
acquiring an image recovery network model to be trained and a second training set;
inputting the sample speckle image into an image recovery network model to be trained to obtain a second output image;
determining a second loss value based on the second output image and the second sample original image;
determining whether the image restoration network model converges based on the second loss value;
if not, updating the parameter value of the image recovery network model, and returning to the step of inputting the sample speckle image into the current image recovery network model to obtain a second output image;
and if so, determining the current image recovery network model as the trained image recovery network model.
The image recovery device based on the pre-training self-encoder provided by the embodiment of the invention is used for acquiring the speckle images, wherein the speckle images are obtained by transmitting the original images through multimode optical fibers; inputting the speckle images into an image recovery network model to obtain recovered images, wherein the image recovery network model comprises a trained first encoder sub-network and a trained first decoder sub-network; initial parameters of the first decoder subnetwork are determined from parameters in a pre-trained self-encoder network model trained from a first training set comprising: a first sample original image; the image recovery network model is obtained by training according to a second training set, the second training set comprises a second sample original image and a sample speckle image, and the sample speckle image is obtained by transmitting the second sample original image through a multimode fiber. It can be seen that the first decoder subnetwork in the image recovery network model is initialized with parameters by pre-training the self-encoder network model and adopting the parameters in the trained self-encoder network model, so that the training data set information can be fully utilized, the convergence rate of the network during training is accelerated, and the image recovery quality can be significantly improved compared with the existing image recovery method.
Corresponding to the embodiment of the image restoration method based on the pre-trained self-encoder, the embodiment of the present invention further provides an electronic device, as shown in fig. 7, including a processor 701, a communication interface 702, a memory 703 and a communication bus 704, where the processor 701, the communication interface 702 and the memory 703 complete mutual communication through the communication bus 704,
a memory 703 for storing a computer program;
the processor 701 is configured to implement the following steps when executing the program stored in the memory 703:
acquiring a speckle image, wherein the speckle image is obtained by transmitting an original image through a multimode fiber;
inputting the speckle image into an image recovery network model to obtain a recovered image;
wherein the image recovery network model comprises a trained first encoder subnetwork and a first decoder subnetwork; initial parameters of the first decoder subnetwork are determined from parameters in a pre-trained self-encoder network model trained from a first training set comprising: a first sample original image; the image recovery network model is obtained by training according to a second training set, the second training set comprises a second sample original image and a sample speckle image, and the sample speckle image is obtained by transmitting the second sample original image through a multimode fiber.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component.
The electronic equipment provided by the embodiment of the invention is applied to obtain the speckle images, wherein the speckle images are obtained by transmitting the original images through the multimode optical fiber; inputting the speckle images into an image recovery network model to obtain recovered images, wherein the image recovery network model comprises a trained first encoder sub-network and a trained first decoder sub-network; initial parameters of the first decoder subnetwork are determined from parameters in a pre-trained self-encoder network model trained from a first training set comprising: a first sample original image; the image recovery network model is obtained by training according to a second training set, the second training set comprises a second sample original image and a sample speckle image, and the sample speckle image is obtained by transmitting the second sample original image through a multimode fiber. It can be seen that the first decoder subnetwork in the image recovery network model is initialized with parameters by pre-training the self-encoder network model and adopting the parameters in the trained self-encoder network model, so that the training data set information can be fully utilized, the convergence rate of the network during training is accelerated, and the image recovery quality can be significantly improved compared with the existing image recovery method.
The embodiment of the invention also provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and the computer program is used for realizing any one of the method steps when being executed by a processor.
The computer instructions may be stored on or transmitted from one computer-readable storage medium to another computer-readable storage medium, e.g., from one website site, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DS L)) or wireless (e.g., infrared, wireless, microwave, etc.) means to another website site, computer, server, or data center, the computer-readable storage medium may be any available medium accessible by a computer or including one or more available media, such as a Solid State Disk, a magnetic storage medium (e.g., a floppy Disk, a Solid State Disk, a floppy Disk, a Solid State Disk, a magnetic storage medium (e.g., a Solid State Disk, a floppy Disk, a Solid State Disk, a magnetic storage medium, a hard Disk, a Solid State Disk, a magnetic storage medium, a magnetic Disk, a magnetic storage medium, a magnetic Disk, a magnetic storage medium, a magnetic Disk, or the like.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the image restoration apparatus, the electronic device, and the computer-readable storage medium based on the pre-trained self-encoder, since they are substantially similar to the image restoration method based on the pre-trained self-encoder, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the image restoration method based on the pre-trained self-encoder.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (10)
1. An image restoration method based on a pre-trained self-encoder, the method comprising:
acquiring a speckle image, wherein the speckle image is obtained by transmitting an original image through a multimode fiber;
inputting the speckle image into an image recovery network model to obtain a recovered image;
wherein the image recovery network model comprises a trained first encoder subnetwork and a first decoder subnetwork; initial parameters of the first decoder subnetwork are determined according to parameters in a pre-trained self-encoder network model, the self-encoder network model being trained according to a first training set, the first training set comprising: a first sample original image; the image recovery network model is obtained by training according to a second training set, the second training set comprises a second sample original image and a sample speckle image, and the sample speckle image is obtained by transmitting the second sample original image through a multimode fiber.
2. The method of claim 1, wherein the self-encoder network model is trained as follows:
acquiring an initial self-encoder network model and the first training set, wherein the initial self-encoder network model comprises a second encoder sub-network and a second decoder sub-network to be trained;
inputting the first sample original image into the initial self-encoder network model to obtain a first output image;
determining a loss value based on the first output image and the first sample raw image;
determining whether the self-encoder network model converges based on the loss value;
if not, updating the parameter values of the second encoder sub-network and the second decoder sub-network, and returning to a self-encoder network model after the first sample original image is input with the updated parameter values to obtain a first output image;
and if so, determining the current self-encoder network model as the trained self-encoder network model.
3. The method of claim 2, further comprising:
acquiring an initial image recovery network model, and initializing parameters of a first decoder sub-network in the initial image recovery network model by using parameters of a second decoder sub-network in the trained self-encoder network model to obtain the image recovery network model to be trained.
4. The method of claim 3, wherein the image recovery network model is trained as follows:
acquiring an image recovery network model to be trained and the second training set;
inputting the sample speckle image into the image recovery network model to be trained to obtain a second output image;
determining a second loss value based on the second output image and the second sample original image;
determining whether the image restoration network model converges based on the second loss value;
if not, updating the parameter value of the image recovery network model, and returning to the step of inputting the sample speckle image into the current image recovery network model to obtain a second output image;
and if so, determining the current image recovery network model as the trained image recovery network model.
5. An apparatus for image restoration based on a pre-trained self-encoder, the apparatus comprising:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring speckle images, and the speckle images are obtained by transmitting original images through multimode optical fibers;
the recovery module is used for inputting the speckle images into an image recovery network model to obtain recovered images; wherein the image recovery network model comprises a trained first encoder subnetwork and a first decoder subnetwork; initial parameters of the first decoder subnetwork are determined according to parameters in a pre-trained self-encoder network model, the self-encoder network model being trained according to a first training set, the first training set comprising: a first sample original image; the image recovery network model is obtained by training according to a second training set, the second training set comprises a second sample original image and a sample speckle image, and the sample speckle image is obtained by transmitting the second sample original image through a multimode fiber.
6. The apparatus of claim 5, further comprising: a first training module to train the self-coder network model as follows:
acquiring an initial self-encoder network model and the first training set, wherein the initial self-encoder network model comprises a second encoder sub-network and a second decoder sub-network to be trained;
inputting the first sample original image into the initial self-encoder network model to obtain a first output image;
determining a loss value based on the first output image and the first sample raw image;
determining whether the self-encoder network model converges based on the loss value;
if not, updating the parameter values of the second encoder sub-network and the second decoder sub-network, and returning to a self-encoder network model after the first sample original image is input with the updated parameter values to obtain a first output image;
and if so, determining the current self-encoder network model as the trained self-encoder network model.
7. The apparatus of claim 6, further comprising: an initialization module to:
acquiring an initial image recovery network model, and initializing parameters of a first decoder sub-network in the initial image recovery network model by using parameters of a second decoder sub-network in the trained self-encoder network model to obtain the image recovery network model to be trained.
8. The apparatus of claim 7, further comprising: a second training module, configured to train the image recovery network model as follows:
acquiring an image recovery network model to be trained and the second training set;
inputting the sample speckle image into the image recovery network model to be trained to obtain a second output image;
determining a second loss value based on the second output image and the second sample original image;
determining whether the image restoration network model converges based on the second loss value;
if not, updating the parameter value of the image recovery network model, and returning to the step of inputting the sample speckle image into the current image recovery network model to obtain a second output image;
and if so, determining the current image recovery network model as the trained image recovery network model.
9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any of claims 1 to 4 when executing a program stored in the memory.
10. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010188588.3A CN111462000B (en) | 2020-03-17 | 2020-03-17 | Image recovery method and device based on pre-training self-encoder |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010188588.3A CN111462000B (en) | 2020-03-17 | 2020-03-17 | Image recovery method and device based on pre-training self-encoder |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111462000A true CN111462000A (en) | 2020-07-28 |
| CN111462000B CN111462000B (en) | 2023-11-28 |
Family
ID=71679043
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010188588.3A Active CN111462000B (en) | 2020-03-17 | 2020-03-17 | Image recovery method and device based on pre-training self-encoder |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111462000B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112929666A (en) * | 2021-03-22 | 2021-06-08 | 北京金山云网络技术有限公司 | Method, device and equipment for training coding and decoding network and storage medium |
| CN113033457A (en) * | 2021-04-08 | 2021-06-25 | 北京邮电大学 | Soft fault recognition model training and soft fault recognition method and device |
| CN113313776A (en) * | 2021-05-27 | 2021-08-27 | Oppo广东移动通信有限公司 | Image processing method, image processing apparatus, storage medium, and electronic device |
| CN113592074A (en) * | 2021-07-28 | 2021-11-02 | 北京世纪好未来教育科技有限公司 | Training method, generating method and device, and electronic device |
| CN114598386A (en) * | 2022-01-24 | 2022-06-07 | 北京邮电大学 | Method and device for detecting soft fault of optical network communication |
| CN114627012A (en) * | 2022-03-09 | 2022-06-14 | 上海应用技术大学 | Old photo restoration method based on automatic encoder |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180101957A1 (en) * | 2016-10-06 | 2018-04-12 | Qualcomm Incorporated | Neural network for image processing |
| CN109859144A (en) * | 2019-02-22 | 2019-06-07 | 上海商汤智能科技有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN109949345A (en) * | 2019-02-21 | 2019-06-28 | 西北大学 | The method neural network based for obtaining lack of balance SAR image variation detection figure |
| CN110163267A (en) * | 2019-05-09 | 2019-08-23 | 厦门美图之家科技有限公司 | A kind of method that image generates the training method of model and generates image |
| CN110263848A (en) * | 2019-06-18 | 2019-09-20 | 北京邮电大学 | A kind of multimode fibre imaging method and device based on production confrontation network |
| US10426442B1 (en) * | 2019-06-14 | 2019-10-01 | Cycle Clarity, LLC | Adaptive image processing in assisted reproductive imaging modalities |
| WO2020006762A1 (en) * | 2018-07-06 | 2020-01-09 | 深圳前海达闼云端智能科技有限公司 | Method for training image restoration model, image restoration method and apparatus, medium, and device |
| CN110738605A (en) * | 2019-08-30 | 2020-01-31 | 山东大学 | Image denoising method, system, device and medium based on transfer learning |
-
2020
- 2020-03-17 CN CN202010188588.3A patent/CN111462000B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180101957A1 (en) * | 2016-10-06 | 2018-04-12 | Qualcomm Incorporated | Neural network for image processing |
| WO2020006762A1 (en) * | 2018-07-06 | 2020-01-09 | 深圳前海达闼云端智能科技有限公司 | Method for training image restoration model, image restoration method and apparatus, medium, and device |
| CN109949345A (en) * | 2019-02-21 | 2019-06-28 | 西北大学 | The method neural network based for obtaining lack of balance SAR image variation detection figure |
| CN109859144A (en) * | 2019-02-22 | 2019-06-07 | 上海商汤智能科技有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN110163267A (en) * | 2019-05-09 | 2019-08-23 | 厦门美图之家科技有限公司 | A kind of method that image generates the training method of model and generates image |
| US10426442B1 (en) * | 2019-06-14 | 2019-10-01 | Cycle Clarity, LLC | Adaptive image processing in assisted reproductive imaging modalities |
| CN110263848A (en) * | 2019-06-18 | 2019-09-20 | 北京邮电大学 | A kind of multimode fibre imaging method and device based on production confrontation network |
| CN110738605A (en) * | 2019-08-30 | 2020-01-31 | 山东大学 | Image denoising method, system, device and medium based on transfer learning |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112929666A (en) * | 2021-03-22 | 2021-06-08 | 北京金山云网络技术有限公司 | Method, device and equipment for training coding and decoding network and storage medium |
| CN112929666B (en) * | 2021-03-22 | 2023-04-14 | 北京金山云网络技术有限公司 | Method, device and equipment for training coding and decoding network and storage medium |
| CN113033457A (en) * | 2021-04-08 | 2021-06-25 | 北京邮电大学 | Soft fault recognition model training and soft fault recognition method and device |
| CN113033457B (en) * | 2021-04-08 | 2022-11-04 | 北京邮电大学 | Soft fault recognition model training and soft fault recognition method and device |
| CN113313776A (en) * | 2021-05-27 | 2021-08-27 | Oppo广东移动通信有限公司 | Image processing method, image processing apparatus, storage medium, and electronic device |
| CN113592074A (en) * | 2021-07-28 | 2021-11-02 | 北京世纪好未来教育科技有限公司 | Training method, generating method and device, and electronic device |
| CN113592074B (en) * | 2021-07-28 | 2023-12-12 | 北京世纪好未来教育科技有限公司 | Training method, generating method and device and electronic equipment |
| CN114598386A (en) * | 2022-01-24 | 2022-06-07 | 北京邮电大学 | Method and device for detecting soft fault of optical network communication |
| CN114598386B (en) * | 2022-01-24 | 2023-08-01 | 北京邮电大学 | Soft fault detection method and device for optical network communication |
| CN114627012A (en) * | 2022-03-09 | 2022-06-14 | 上海应用技术大学 | Old photo restoration method based on automatic encoder |
| CN114627012B (en) * | 2022-03-09 | 2024-04-19 | 上海应用技术大学 | Old photo restoration method based on automatic encoder |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111462000B (en) | 2023-11-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111462000A (en) | Image recovery method and device based on pre-training self-encoder | |
| CN113240056B (en) | Multi-mode data joint learning model training method and device | |
| CN108596267B (en) | Image reconstruction method, terminal equipment and computer readable storage medium | |
| CN113128494A (en) | Method, device and system for recognizing text in image | |
| CN114022394B (en) | Image restoration method and device, electronic equipment and storage medium | |
| CN116884391B (en) | Multimode fusion audio generation method and device based on diffusion model | |
| CN112149733B (en) | Model training method, model quality determining method, model training device, model quality determining device, electronic equipment and storage medium | |
| WO2023082453A1 (en) | Image processing method and device | |
| CN115115918B (en) | Visual learning method based on multi-knowledge fusion | |
| CN110991298B (en) | Image processing method and device, storage medium and electronic device | |
| CN116206314A (en) | Model training method, formula identification method, device, medium and equipment | |
| CN114464168A (en) | Training method of voice processing model, and noise reduction method and device of voice data | |
| CN116110378B (en) | Model training method, voice recognition device and electronic equipment | |
| CN112995673B (en) | Sample image processing method and device, electronic equipment and medium | |
| CN117036190A (en) | Image denoising method and device and electronic equipment | |
| CN111325832A (en) | Modeling method, modeling device and electronic equipment | |
| CN115272667B (en) | Farmland image segmentation model training method and device, electronic equipment and medium | |
| CN112995433B (en) | Time sequence video generation method and device, computing equipment and storage medium | |
| CN110852043B (en) | Text transcription method, device, equipment and storage medium | |
| CN114912568A (en) | Method, apparatus and computer-readable storage medium for data processing | |
| CN111898658A (en) | Image classification method and device and electronic equipment | |
| CN116629947B (en) | Method, device, equipment and medium for generating flow site processing information | |
| CN116558541B (en) | Model training method and device, and track prediction method and device | |
| CN116468071B (en) | Model training method, device, equipment and storage medium | |
| CN113792784B (en) | Method, electronic device and storage medium for user clustering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |