CN112270648A - Unsupervised image transformation method and unsupervised image transformation device based on loop countermeasure network - Google Patents
Unsupervised image transformation method and unsupervised image transformation device based on loop countermeasure network Download PDFInfo
- Publication number
- CN112270648A CN112270648A CN202011019829.8A CN202011019829A CN112270648A CN 112270648 A CN112270648 A CN 112270648A CN 202011019829 A CN202011019829 A CN 202011019829A CN 112270648 A CN112270648 A CN 112270648A
- Authority
- CN
- China
- Prior art keywords
- network
- image
- backward
- loop
- unsupervised
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/088—Non-supervised learning, e.g. competitive learning
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The application discloses an unsupervised image transformation method and device based on a loop countermeasure network, wherein the method comprises the following steps: step S1: constructing a loop countermeasure network, wherein the loop countermeasure network comprises a forward generation network and a backward generation network for converting an input image into an output image, and a forward judgment network and a backward judgment network for judging whether a corresponding generation result is real; step S2: designing a network loss function, wherein the network loss function comprises: step S3: manufacturing a data set by adopting an unsupervised mechanism; step S4: training and generating a network and a discrimination network by taking the minimum cost function as a target; step S5: and inputting the target image to the trained loop countermeasure network to obtain an image transformation result. The method learns a pair of reciprocal mappings by simultaneously training two generative countermeasure networks on misaligned data sets, and limits the position of image content on an image coordinate system by adding additional saliency constraints, thereby avoiding mapping biases.
Description
Technical Field
The application relates to the technical field of artificial intelligence and machine learning, in particular to an unsupervised image transformation method and device based on a loop countermeasure network.
Background
Artificial intelligence techniques typified by deep learning have been widely used in image information processing. The deep neural network can be regarded as a universal high-dimensional function fitter, and after reasonable training is carried out on a given training set, excellent generalization capability can be shown on a test set. The application of deep learning to image tasks is mainly divided into two categories, the first is classification, which aims to map the input image into a category label, in this case the input is a high-dimensional image and the output is a low-dimensional category probability distribution. The other is image transformation, which aims to map an input image to another image, and in this application scenario, the input and output are high-dimensional images. The mapping of images to images is to implement an image transformation, analyze the data content or reveal implicit, imperceptible patterns within the data in order to improve the data quality.
In recent years, with the gradual improvement of artificial intelligence theory and the rapid development of deep learning technology, it has become possible to train a deep neural network to complete a specific task without supervision. In some applications involving natural images, unsupervised image style migration has been implemented. However, these methods cannot guarantee stable and accurate convergence, and in a scene with a stricter requirement, especially in applications involving biomedical images and microscopic images, the image transformation method based on unsupervised learning requires new innovation to be applied.
Content of application
The present invention is directed to solving, at least to some extent, one of the technical problems in the related art.
To this end, a first objective of the present invention is to propose an unsupervised image transformation method based on a loop countermeasure network.
The second purpose of the invention is to provide an unsupervised image transformation device based on a loop countermeasure network.
A third object of the invention is to propose an electronic device.
A fourth object of the invention is to propose a computer-readable storage medium.
In order to achieve the above object, an embodiment of a first aspect of the present application provides an unsupervised image transformation method based on a loop countermeasure network, including the following steps:
step S1: constructing a loop countermeasure network, wherein the loop countermeasure network comprises a forward generation network and a backward generation network which are used for converting an input image into an output image, and a forward judgment network and a backward judgment network which are used for judging whether a corresponding generation result is real or not;
step S2: designing a network loss function, the network loss function comprising: the first part is the countermeasure loss of the cyclic countermeasure network; the second part is a cycle consistency loss, so that the difference between the cyclically generated image and the original image should be minimal; the third part is significance constraint, belonging to weak constraint, to limit the position of image content in the output image coordinate system;
step S3: manufacturing a data set by adopting an unsupervised mechanism, wherein a large graph of a source domain mode is split into small graphs of a first preset size and placed into a source domain, and a large graph of a target domain style is split into small graphs of a second preset size and placed into a target domain;
step S4: training and generating a network and a discrimination network by taking the minimum cost function as a target; and
step S5: and inputting the target image to the trained loop countermeasure network to obtain an image transformation result.
In addition, the unsupervised image transformation method based on the loop countermeasure network according to the above embodiment of the present invention may further have the following additional technical features:
optionally, the forward generation network and the backward generation network employ a deep residual network comprising three layers of down-sampled convolutional layers followed by nine consecutive residual blocks and up-sampled output layers, wherein the included span connections transmit information lossless to subsequent convolutional layers.
Optionally, the forward direction discrimination network and the backward direction discrimination network adopt a patch gan structure, so as to judge the true and false results of the output result on the scale of the receptive field through the abstract representation of the continuously connected downsampling layer learning data.
Optionally, the first part employs a general cross-entropy method against loss; the cycle consistency constraint is expressed as a measure of the L1 norm between the input image and the cycle-generated image; the saliency constraints are generated by a threshold segmentation algorithm, wherein after a segmentation threshold is determined such that the image should be as close as possible to the image generated by the forward generation network and the output image should be as close as possible to the circularly generated image, the saliency constraints are applied to both the forward and backward mappings.
Optionally, the training with the minimized cost function as the target to generate the network and the discriminant network includes:
and optimizing network parameters by adopting a preset Adam optimizer.
In order to achieve the above object, a second aspect of the present application provides an unsupervised image transformation apparatus based on a loop countermeasure network, including:
the system comprises a building module, a judging module and a judging module, wherein the building module is used for building a loop countermeasure network, and the loop countermeasure network comprises a forward generation network and a backward generation network which are used for converting an input image into an output image and a forward judging network and a backward judging network which are used for judging whether a corresponding generation result is real or not;
a design module to design a network loss function, the network loss function comprising: the first part is the countermeasure loss of the cyclic countermeasure network; the second part is a cycle consistency loss, so that the difference between the cyclically generated image and the original image should be minimal; the third part is significance constraint, belonging to weak constraint, to limit the position of image content in the output image coordinate system;
the system comprises a production module, a data set and a data processing module, wherein the production module is used for producing the data set by adopting an unsupervised mechanism, splitting a large graph of a source domain mode into small graphs of a first preset size and placing the small graphs into a source domain, and splitting a large graph of a target domain style into small graphs of a second preset size and placing the small graphs into a target domain;
the generating module is used for training and generating a network and a discrimination network by taking the minimum cost function as a target; and
and the acquisition module is used for inputting the target image to the trained cyclic countermeasure network to obtain an image transformation result.
Optionally, the forward generation network and the backward generation network employ a deep residual network comprising three layers of down-sampled convolutional layers followed by nine consecutive residual blocks and up-sampled output layers, wherein the included span connections transmit information lossless to subsequent convolutional layers.
Optionally, the forward direction discrimination network and the backward direction discrimination network adopt a patch gan structure, so as to judge the true and false results of the output result on the scale of the receptive field through the abstract representation of the continuously connected downsampling layer learning data.
To achieve the above object, an embodiment of a third aspect of the present application provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor and arranged to perform the loop countermeasure network based unsupervised image transformation method of the above embodiment.
In order to achieve the above object, a fourth aspect of the present application provides a computer-readable storage medium storing computer instructions for causing the computer to execute the loop countermeasure network-based unsupervised image transformation method according to the above embodiment.
Therefore, an image transformation result is obtained by constructing a loop countermeasure network, designing a network loss function, manufacturing a data set by adopting an unsupervised mechanism, training a generation network and a judgment network by taking a minimum cost function as a target, and inputting a target image to the trained loop countermeasure network. Therefore, under the condition that one-to-one corresponding input-output training pairs are not needed, the network is made to learn the mapping function, the acquisition and calibration of the alignment data can be effectively avoided, the application range of deep learning is widened, the application efficiency of the deep learning is improved, namely two generated confrontation networks are simultaneously trained on unaligned data sets to learn one-to-one reciprocal mapping, and the position of the image content on the image coordinate system is limited by adding extra significance constraints, so that the mapping deviation is avoided.
Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a flowchart of an unsupervised image transformation method based on a loop countermeasure network according to an embodiment of the present application;
FIG. 2 is a flow diagram of an unsupervised image transformation method based on a loop countermeasure network according to one embodiment of the present application;
FIG. 3 is a schematic diagram of an unsupervised image transformation method based on a loop countermeasure network according to an embodiment of the present application;
FIG. 4 is a schematic network structure diagram of an unsupervised image transformation method based on a loop countermeasure network according to an embodiment of the present application;
FIG. 5 is an exemplary diagram of an unsupervised image transformation device based on a loop countermeasure network according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary and intended to be used for explaining the present application and should not be construed as limiting the present application.
The unsupervised image transformation method and device based on the loop countermeasure network according to the embodiment of the invention are described below with reference to the accompanying drawings, and first, the unsupervised image transformation method based on the loop countermeasure network according to the embodiment of the invention will be described with reference to the accompanying drawings.
Specifically, fig. 1 is a schematic flowchart of an unsupervised image transformation method based on a loop countermeasure network according to an embodiment of the present application.
As shown in fig. 1, the unsupervised image transformation method based on the loop countermeasure network comprises the following steps:
step S1: and constructing a loop countermeasure network, wherein the loop countermeasure network comprises a forward generation network and a backward generation network which are used for converting the input image into the output image, and a forward judgment network and a backward judgment network which are used for judging whether the corresponding generation result is real or not.
Optionally, in some embodiments, the forward generation network and the backward generation network employ a depth residual network comprising three layers of down-sampled convolutional layers followed by nine consecutive residual blocks and up-sampled output layers, wherein the contained span connections transmit information losslessly to subsequent convolutional layers.
It will be appreciated that, in conjunction with fig. 2 and 3, the cyclic confrontation network comprises two generative confrontation networks, one for learning the forward mapping, referred to as the forward confrontation network (G), as shown in fig. 3; the other is used to learn a backward mapping, called a backward countermeasure network (F). Each countermeasure network in turn comprises a generator for converting an input image into an output image and a discriminator for determining whether the result of the corresponding generator is authentic. In summary, the loop countermeasure network includes a total of four networks: the system comprises a forward generation network, a forward judgment network, a backward generation network and a backward judgment network.
Preferably, the forward countermeasure network and the backward countermeasure network in step S1 may be selected as the same structure, and the design of the network structure may be compressed as the design generator and the arbiter under the preferred design. Preferably, the generator employs a deep residual network, as shown in fig. 3. The generator consists of three layers of downsampled convolutional layers, followed by nine consecutive residual blocks, and finally an upsampled output layer. The span connection contained in the method can transmit information to a subsequent convolution layer in a lossless manner, so that the problem of gradient disappearance can be effectively avoided, and the generated network is easier to train.
Optionally, in some embodiments, the forward discriminant network and the backward discriminant network adopt a PatchGAN structure to determine the true and false results of the output result on the scale of the receptive field through an abstract representation of continuously connected downsampled learning data.
That is to say, the discriminator may adopt a PatchGAN structure, and the truth of the output result is judged on the scale of the receptive field through the abstract representation of the continuously connected downsampling layer learning data, and this structure can improve the perception of the high-frequency fine structure, and is favorable for generating a highly vivid fine image.
Step S2: designing a network loss function, wherein the network loss function comprises: the first part is the countermeasure loss of the cyclic countermeasure network; the second part is a cycle consistency loss, so that the difference between the cyclically generated image and the original image should be minimal; the third part is a significance constraint, which belongs to a weak constraint, so as to limit the position of the image content in the output image coordinate system.
Optionally, in some embodiments, the first part employs a general cross-entropy approach to combat losses; the cycle consistency constraint is expressed as a measure of the L1 norm between the input image and the cycle-generated image; the saliency constraints are generated by a threshold segmentation algorithm, wherein after a segmentation threshold is determined such that the image should be as close as possible to the image generated by the forward generation network, the output image should be as close as possible to the circularly generated image, and the saliency constraints are applied to both the forward and backward mappings.
It will be appreciated that the network penalty function may be made up of three parts, the first part being the penalty for the forward and backward competing networks, used to train the network. The second part is the loss of cyclic consistency, which ensures that the image can return to itself after being forward mapped and backward mapped, i.e. the difference between the cyclically generated image and the original image should be minimal. The third part is a saliency constraint, which is a weak constraint, used to define the position of the image content in the output image coordinate system. These three-term loss functions trade off the relative relationship of the constraints by adding different weights to control their relative strengths.
Preferably, the first part adopts a general cross entropy method for resisting loss; the cyclic consistency constraint may be expressed as a measure of the L1 norm between the input image and the cyclically generated image; the saliency constraint can be implemented by a thresholding algorithm, i.e. after the segmentation threshold is chosen, the input image and the forward network generated image should be as close as possible, and the output image should be as close as possible to the circularly generated image, which constraint needs to be applied to both the forward and backward mappings.
Specifically, the network loss function includes the countervailing loss, the cyclic consistency loss, the significance constraint of the forward and backward countermeasure networks, expressed as:
wherein the loss is resisted
Can be expressed as:
the cycle consistent loss can be expressed as:
the significance constraint is expressed as:
in the above formulation language, G and F represent a forward generator and a backward generator, respectively, and DAAnd DBRespectively representing a forward discriminator and a backward discriminator, a and b respectively representing images in a source domain and a target domain; e denotes the statistical expectation, which in practice is achieved by an arithmetic average element by element.
Step S3: and manufacturing a data set by adopting an unsupervised mechanism, wherein a large graph of a source domain mode is split into small graphs of a first preset size and placed into a source domain, and a large graph of a target domain style is split into small graphs of a second preset size and placed into a target domain.
That is to say, in the embodiment of the present application, an unsupervised training manner is adopted, so that it is not necessary to prepare an "input-output image pair" of pixel-level registration when a data set is created, split a large image in a source domain mode into small images of a certain size and place the small images in the source domain, and split a large image in a target domain style into small images of a certain size and place the small images in the target domain. Step S4: and training the generated network and the discriminant network by taking the minimized cost function as a target.
Optionally, in some embodiments, as shown in fig. 4, training the generated network and the discriminant network with the minimized cost function as a target includes: and optimizing network parameters by adopting a preset Adam optimizer.
That is to say, the embodiment of the present application may preferentially adopt the Adam optimizer with the best performance at present to perform the optimization of the network parameters, different optimizers may bring a certain difference of the training effect, and as the optimization algorithm progresses, a better optimizer should be used.
Step S5: and inputting the target image to the trained loop countermeasure network to obtain an image transformation result.
It should be noted that, in this step, the network parameters trained in the above steps are used, and once the network parameters are optimized, the network parameters can be stored for all the same-type data.
In conclusion, a deep generation countermeasure network is built, a cyclic countermeasure network is built, significance constraints are added, preparation of a source domain and a target domain and training and application of the networks are carried out, two deep countermeasure networks are trained simultaneously, the forward network is responsible for learning forward mapping, the backward network is used for learning backward mapping, and after full training in an unsupervised mode, the two networks form a complete closed loop, so that a pair of reversible mappings is built. In addition, in order to avoid the network from converging to an unexpected solution, additional significance constraint is added in a loss function of the network, so that the position of the image content in an image coordinate system is limited, the network is effectively prevented from converging to an unreasonable solution, and the stability and the accuracy of the network are greatly improved.
According to the unsupervised image transformation method based on the loop countermeasure network, the loop countermeasure network is built, the network loss function is designed, the unsupervised mechanism is adopted to manufacture the data set, the minimum cost function is used as the target training to generate the network and the judgment network, the target image is input to the trained loop countermeasure network, and the image transformation result is obtained. Therefore, under the condition that one-to-one corresponding input-output training pairs are not needed, the network is made to learn the mapping function, the acquisition and calibration of the alignment data can be effectively avoided, the application range of deep learning is widened, the application efficiency of the deep learning is improved, namely two generated confrontation networks are simultaneously trained on unaligned data sets to learn one-to-one reciprocal mapping, and the position of the image content on the image coordinate system is limited by adding extra significance constraints, so that the mapping deviation is avoided.
Next, an unsupervised image transformation device based on a loop countermeasure network according to an embodiment of the present application will be described with reference to the drawings.
Fig. 5 is a block schematic diagram of an unsupervised image transformation device based on a loop countermeasure network according to an embodiment of the present application.
As shown in fig. 5, the unsupervised image transformation device 10 based on the loop countermeasure network includes: .
The system comprises a building module 100, a design module 200, a manufacturing module 300, a generating module 400 and an obtaining module 500.
The construction module 100 is used for constructing a loop countermeasure network, wherein the loop countermeasure network comprises a forward generation network and a backward generation network for converting an input image into an output image, and a forward judgment network and a backward judgment network for judging whether a corresponding generation result is real;
the design module 200 is used to design a network loss function, which includes: the first part is the countermeasure loss of the cyclic countermeasure network; the second part is a cycle consistency loss, so that the difference between the cyclically generated image and the original image should be minimal; the third part is significance constraint, belonging to weak constraint, to limit the position of image content in the output image coordinate system;
the making module 300 is configured to make a data set by using an unsupervised mechanism, where a large graph in a source domain mode is split into small graphs of a first preset size and placed in a source domain, and a large graph in a target domain style is split into small graphs of a second preset size and placed in a target domain;
the generating module 400 is configured to train a generating network and a discriminating network with the minimized cost function as a target; and
the obtaining module 500 is configured to input the target image to the trained loop countermeasure network, and obtain an image transformation result.
Optionally, the forward generation network and the backward generation network employ a deep residual network comprising three layers of down-sampled convolutional layers followed by nine consecutive residual blocks and up-sampled output layers, wherein the included span connections transmit information lossless to subsequent convolutional layers.
Optionally, the forward direction discrimination network and the backward direction discrimination network adopt a patch gan structure, so as to judge the true and false results of the output result on the scale of the receptive field through the abstract representation of the continuously connected downsampling layer learning data.
It should be noted that the foregoing explanation on the embodiment of the loop countermeasure network-based unsupervised image transformation method is also applicable to the loop countermeasure network-based unsupervised image transformation apparatus of this embodiment, and is not repeated here.
According to the unsupervised image transformation device based on the loop countermeasure network, the loop countermeasure network is built, the network loss function is designed, the unsupervised mechanism is adopted to manufacture the data set, the minimum cost function is used as the target to train the generation network and the judgment network, the target image is input to the trained loop countermeasure network, and the image transformation result is obtained. Therefore, under the condition that one-to-one corresponding input-output training pairs are not needed, the network is made to learn the mapping function, the acquisition and calibration of the alignment data can be effectively avoided, the application range of deep learning is widened, the application efficiency of the deep learning is improved, namely two generated confrontation networks are simultaneously trained on unaligned data sets to learn one-to-one reciprocal mapping, and the position of the image content on the image coordinate system is limited by adding extra significance constraints, so that the mapping deviation is avoided. .
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device may include:
a memory 1201, a processor 1202, and a computer program stored on the memory 1201 and executable on the processor 1202.
The processor 1202, when executing the program, implements the loop countermeasure network-based unsupervised image transformation method provided in the above-described embodiments.
Further, the electronic device further includes:
a communication interface 1203 for communication between the memory 1201 and the processor 1202.
A memory 1201 for storing computer programs executable on the processor 1202.
The memory 1201 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
If the memory 1201, the processor 1202 and the communication interface 1203 are implemented independently, the communication interface 1203, the memory 1201 and the processor 1202 may be connected to each other through a bus and perform communication with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 6, but this is not intended to represent only one bus or type of bus.
Optionally, in a specific implementation, if the memory 1201, the processor 1202, and the communication interface 1203 are integrated on a chip, the memory 1201, the processor 1202, and the communication interface 1203 may complete mutual communication through an internal interface.
The present embodiment also provides a computer-readable storage medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the above unsupervised image transformation method based on a loop countermeasure network.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or N embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "N" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more N executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of implementing the embodiments of the present application.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or N wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.
Claims (10)
1. An unsupervised image transformation method based on a loop countermeasure network is characterized by comprising the following steps:
step S1: constructing a loop countermeasure network, wherein the loop countermeasure network comprises a forward generation network and a backward generation network which are used for converting an input image into an output image, and a forward judgment network and a backward judgment network which are used for judging whether a corresponding generation result is real or not;
step S2: designing a network loss function, the network loss function comprising: the first part is the countermeasure loss of the cyclic countermeasure network; the second part is a cycle consistency loss, so that the difference between the cyclically generated image and the original image should be minimal; the third part is significance constraint, belonging to weak constraint, to limit the position of image content in the output image coordinate system;
step S3: manufacturing a data set by adopting an unsupervised mechanism, wherein a large graph of a source domain mode is split into small graphs of a first preset size and placed into a source domain, and a large graph of a target domain style is split into small graphs of a second preset size and placed into a target domain;
step S4: training and generating a network and a discrimination network by taking the minimum cost function as a target; and
step S5: and inputting the target image to the trained loop countermeasure network to obtain an image transformation result.
2. The method of claim 1, wherein the forward generation network and the backward generation network employ a deep residual network comprising three layers of down-sampled convolutional layers followed by nine consecutive residual blocks and up-sampled output layers, wherein a span connection is included to losslessly transmit information to subsequent convolutional layers.
3. The method of claim 2, wherein the forward and backward decision networks employ a PatchGAN structure to determine the true and false results of the output result on a receptive field scale by continuously connected down-sampling layers to learn abstract representations of data.
4. The method of claim 1, wherein the first portion employs a general cross-entropy method against loss; the cycle consistency constraint is expressed as a measure of the L1 norm between the input image and the cycle-generated image; the saliency constraints are generated by a threshold segmentation algorithm, wherein after a segmentation threshold is determined such that the image should be as close as possible to the image generated by the forward generation network and the output image should be as close as possible to the circularly generated image, the saliency constraints are applied to both the forward and backward mappings.
5. The method of claim 1, wherein the training to generate the net and the discriminant net for the target with the minimized cost function comprises:
and optimizing network parameters by adopting a preset Adam optimizer.
6. An unsupervised image transformation device based on a loop countermeasure network, comprising:
the system comprises a building module, a judging module and a judging module, wherein the building module is used for building a loop countermeasure network, and the loop countermeasure network comprises a forward generation network and a backward generation network which are used for converting an input image into an output image and a forward judging network and a backward judging network which are used for judging whether a corresponding generation result is real or not;
a design module to design a network loss function, the network loss function comprising: the first part is the countermeasure loss of the cyclic countermeasure network; the second part is a cycle consistency loss, so that the difference between the cyclically generated image and the original image should be minimal; the third part is significance constraint, belonging to weak constraint, to limit the position of image content in the output image coordinate system;
the system comprises a production module, a data set and a data processing module, wherein the production module is used for producing the data set by adopting an unsupervised mechanism, splitting a large graph of a source domain mode into small graphs of a first preset size and placing the small graphs into a source domain, and splitting a large graph of a target domain style into small graphs of a second preset size and placing the small graphs into a target domain;
the generating module is used for training and generating a network and a discrimination network by taking the minimum cost function as a target; and
and the acquisition module is used for inputting the target image to the trained cyclic countermeasure network to obtain an image transformation result.
7. The apparatus of claim 6, wherein the forward generation network and the backward generation network employ a depth residual network comprising three layers of down-sampled convolutional layers followed by nine consecutive residual blocks and up-sampled output layers, wherein a span connection is included to losslessly transmit information to subsequent convolutional layers.
8. The apparatus of claim 7, wherein the forward and backward decision networks employ a PatchGAN structure to determine the true and false results of the output result on a receptive field scale by continuously connected down-sampling layers to learn abstract representations of data.
9. An electronic device, comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the loop countermeasure network based unsupervised image transformation method of any of claims 1-5.
10. A computer-readable storage medium, on which a computer program is stored, which program is executable by a processor for implementing the loop countermeasure network based unsupervised image transformation method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011019829.8A CN112270648A (en) | 2020-09-24 | 2020-09-24 | Unsupervised image transformation method and unsupervised image transformation device based on loop countermeasure network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011019829.8A CN112270648A (en) | 2020-09-24 | 2020-09-24 | Unsupervised image transformation method and unsupervised image transformation device based on loop countermeasure network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112270648A true CN112270648A (en) | 2021-01-26 |
Family
ID=74349572
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011019829.8A Withdrawn CN112270648A (en) | 2020-09-24 | 2020-09-24 | Unsupervised image transformation method and unsupervised image transformation device based on loop countermeasure network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112270648A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112785542A (en) * | 2021-02-07 | 2021-05-11 | 中国人民解放军国防科技大学 | Method and device for converting remote sensing image into network map, computer equipment and medium |
| CN113095998A (en) * | 2021-03-17 | 2021-07-09 | 南京理工大学 | Method for reducing unsupervised image conversion network parameter number based on reversible network |
| CN113160101A (en) * | 2021-04-14 | 2021-07-23 | 中山大学 | Method for synthesizing high-simulation image |
| CN114528976A (en) * | 2022-01-24 | 2022-05-24 | 北京智源人工智能研究院 | Equal variable network training method and device, electronic equipment and storage medium |
| CN114693897A (en) * | 2021-04-28 | 2022-07-01 | 上海联影智能医疗科技有限公司 | Unsupervised inter-layer super-resolution for medical images |
| CN114821681A (en) * | 2022-06-27 | 2022-07-29 | 深圳市魔力信息技术有限公司 | Fingerprint augmentation method |
-
2020
- 2020-09-24 CN CN202011019829.8A patent/CN112270648A/en not_active Withdrawn
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112785542A (en) * | 2021-02-07 | 2021-05-11 | 中国人民解放军国防科技大学 | Method and device for converting remote sensing image into network map, computer equipment and medium |
| CN112785542B (en) * | 2021-02-07 | 2022-09-09 | 中国人民解放军国防科技大学 | Method and device for converting remote sensing image into network map, computer equipment and medium |
| CN113095998A (en) * | 2021-03-17 | 2021-07-09 | 南京理工大学 | Method for reducing unsupervised image conversion network parameter number based on reversible network |
| CN113160101A (en) * | 2021-04-14 | 2021-07-23 | 中山大学 | Method for synthesizing high-simulation image |
| CN113160101B (en) * | 2021-04-14 | 2023-08-01 | 中山大学 | Method for synthesizing high-simulation image |
| CN114693897A (en) * | 2021-04-28 | 2022-07-01 | 上海联影智能医疗科技有限公司 | Unsupervised inter-layer super-resolution for medical images |
| CN114528976A (en) * | 2022-01-24 | 2022-05-24 | 北京智源人工智能研究院 | Equal variable network training method and device, electronic equipment and storage medium |
| CN114821681A (en) * | 2022-06-27 | 2022-07-29 | 深圳市魔力信息技术有限公司 | Fingerprint augmentation method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112270648A (en) | Unsupervised image transformation method and unsupervised image transformation device based on loop countermeasure network | |
| CN110059741B (en) | Image recognition method based on semantic capsule fusion network | |
| CN110287873B (en) | Non-cooperative target pose measurement method and system based on deep neural network and terminal equipment | |
| CN108447062A (en) | A kind of dividing method of the unconventional cell of pathological section based on multiple dimensioned mixing parted pattern | |
| CN110046698A (en) | Heterogeneous figure neural network generation method, device, electronic equipment and storage medium | |
| CN111899353A (en) | Three-dimensional scanning point cloud hole filling method based on generation countermeasure network | |
| Huang et al. | Deep and wide multiscale recursive networks for robust image labeling | |
| CN112529146B (en) | Neural network model training method and device | |
| CN110245683B (en) | Residual error relation network construction method for less-sample target identification and application | |
| Peyghambarzadeh et al. | Point-PlaneNet: Plane kernel based convolutional neural network for point clouds analysis | |
| CN107609634A (en) | A kind of convolutional neural networks training method based on the very fast study of enhancing | |
| CN111311702B (en) | Image generation and identification module and method based on BlockGAN | |
| Son et al. | SAUM: Symmetry-aware upsampling module for consistent point cloud completion | |
| Zhou et al. | Octr: Octree-based transformer for 3d object detection | |
| CN113362242B (en) | Image restoration method based on multi-feature fusion network | |
| CN117115339B (en) | Building three-dimensional reconstruction and damage identification method based on NeRF D nerve radiation field | |
| Mosella-Montoro et al. | 2d–3d geometric fusion network using multi-neighbourhood graph convolution for rgb-d indoor scene classification | |
| CN113743417A (en) | Semantic segmentation method and semantic segmentation device | |
| CN110674774A (en) | Improved deep learning facial expression recognition method and system | |
| CN111210382A (en) | Image processing method, image processing device, computer equipment and storage medium | |
| CN115375877A (en) | Three-dimensional point cloud classification method and device based on channel attention mechanism | |
| CN108629374A (en) | A kind of unsupervised multi-modal Subspace clustering method based on convolutional neural networks | |
| CN109934835B (en) | Contour detection method based on deep strengthening network adjacent connection | |
| CN112802072B (en) | Medical image registration method and system based on counterstudy | |
| Henry et al. | Pix2Pix GAN for image-to-image Translation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210126 |