WO2023179291A1 - 图像修复方法、装置、设备、介质及产品 - Google Patents

图像修复方法、装置、设备、介质及产品 Download PDF

Info

Publication number
WO2023179291A1
WO2023179291A1 PCT/CN2023/077871 CN2023077871W WO2023179291A1 WO 2023179291 A1 WO2023179291 A1 WO 2023179291A1 CN 2023077871 W CN2023077871 W CN 2023077871W WO 2023179291 A1 WO2023179291 A1 WO 2023179291A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
repaired
feature sequence
repair
sequence
Prior art date
Application number
PCT/CN2023/077871
Other languages
English (en)
French (fr)
Inventor
毛晓飞
黄灿
Original Assignee
北京有竹居网络技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京有竹居网络技术有限公司 filed Critical 北京有竹居网络技术有限公司
Publication of WO2023179291A1 publication Critical patent/WO2023179291A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present disclosure relates to the field of computer technology, and in particular, to an image repair method, device, equipment, computer-readable storage medium and computer program product.
  • Image restoration refers to restoring the unknown information in the image based on the known information in the image, so as to repair the missing parts of the image.
  • the usual image repair technology is to determine the reference area and the area to be repaired in the image to be repaired, and use the neural network model to determine the pixel values of the area to be repaired based on the pixel values of the reference area for image repair.
  • this image repair technology may cause distortions such as ripples and distortions in the repair area, which does not meet the user's requirements for image repair effects. How to improve the effect of image restoration has become an urgent problem that needs to be solved.
  • the purpose of this disclosure is to provide an image repair method, device, equipment, computer-readable storage medium and computer program product, which can repair the image from the perspective of the entire image and obtain a more realistic repair effect.
  • the present disclosure provides an image repair method, which method includes:
  • the image to be repaired is input into the structural repair model, and the image to be repaired is down-sampled through multiple branches of the structural repair model to obtain a first feature sequence and a second feature sequence, and the first feature sequence is Convert into a third feature sequence with the same length as the second feature sequence, fuse the third feature sequence with the second feature sequence, perform structural repair on the image to be repaired according to the fused feature sequence, and obtain A first repaired image, the first repaired image is an image that repairs the structure of the image to be repaired.
  • an image repair device which includes:
  • a structure repair module configured to input the image to be repaired into a structure repair model, downsample the image to be repaired through multiple branches of the structure repair model, and obtain a first feature sequence and a second feature sequence.
  • the first feature sequence is converted into a third feature sequence with the same length as the second feature sequence, the third feature sequence is fused with the second feature sequence, and the to-be-repaired feature sequence is modified according to the fused feature sequence.
  • the image is structurally repaired to obtain a first repaired image, where the first repaired image is an image that repairs the structure of the image to be repaired.
  • the present disclosure provides an electronic device, including: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the first aspect of the present disclosure. Method steps.
  • the present disclosure provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processing device, the steps of the method described in the first aspect of the present disclosure are implemented.
  • the present disclosure provides a computer program product containing instructions that, when run on a device, cause the device to perform the steps of the method described in the first aspect.
  • the electronic device acquires the image to be repaired, then inputs the image to be repaired into the structural repair model, and downsamples the image to be repaired through multiple branches of the structural repair model to obtain the first feature sequence and the second feature sequence, Convert the first feature sequence into a third feature sequence with the same length as the second feature sequence, fuse the third feature sequence with the second feature sequence, perform structural repair on the image to be repaired based on the fused feature sequence, and obtain the image to be repaired Image of the structure being repaired. Since multiple branches in the structural repair model can downsample the image to be repaired at different scales, then extract the features of the image to be repaired at different scales, and repair the image to be repaired based on the fusion results, it can achieve higher repair accuracy. , the repaired image has better effect.
  • Figure 1 is a schematic flow chart of an image repair method provided by an embodiment of the present disclosure
  • Figure 2 is a schematic diagram of a structural repair model provided by an embodiment of the present disclosure
  • Figure 3 is a schematic diagram of another structural repair model provided by an embodiment of the present disclosure.
  • Figure 4 is a schematic flow chart of another image repair method provided by an embodiment of the present disclosure.
  • Figure 5 is a schematic diagram of a texture/color repair model provided by an embodiment of the present disclosure.
  • Figure 6 is a schematic structural diagram of an image repair device provided by an embodiment of the present disclosure.
  • FIG. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • first and second in the embodiments of the present disclosure are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, features defined as “first” and “second” may explicitly or implicitly include one or more of these features.
  • Image processing technology generally refers to the processing of digital images, and specifically refers to the technology of analyzing and processing digital images through computers. Image processing technology can perform various types of processing on images, such as repairing images with missing parts, that is, image restoration technology.
  • Image repair technology is to determine the repair area and reference area in the image to be repaired for the image to be repaired.
  • the repair area can be an area where part of the pattern is missing, or an area where the clarity does not meet the user's needs.
  • Image repair technology under normal circumstances can directly predict the pixel values of the repair area through a neural network model based on the pixel values of the reference area in the image to be repaired, thereby achieving the repair of the repair area of the image to be repaired.
  • this repair method that directly predicts the pixel values of the repair area through the model only repairs the image from the perspective of pixel values, which may cause distortions such as ripples and distortions in the repaired area, and does not satisfy users' expectations for the realism of image repair. Require.
  • Electronic equipment refers to equipment with data processing capabilities, such as servers or terminals.
  • terminals include but are not limited to smartphones, tablets, laptops, personal digital assistants (personal digital assistants, PDAs) or smart wearable devices.
  • the server may be a cloud server, such as a central server in a central cloud computing cluster, or an edge server in an edge cloud computing cluster.
  • the server can also be a server in a local data center.
  • a local data center refers to a data center directly controlled by the user.
  • the electronic device acquires the image to be repaired, then inputs the image to be repaired into the structural repair model, downsamples the image to be repaired through multiple branches of the structural repair model, obtains the first feature sequence and the second feature sequence, and then adds the first feature sequence to the second feature sequence.
  • a feature sequence is converted into a third feature sequence with the same length as the second feature sequence, the third feature sequence and the second feature sequence are fused, and the structure of the image to be repaired is performed according to the fused feature sequence, thereby obtaining the image to be repaired.
  • the first repair image of the structure for repair is
  • the structural repair model can repair the image to be repaired based on the image features of different scales, and integrate the features of different scales to perform structural repair on the model to be repaired, so as to repair the image to be repaired from the structural aspect, improve the authenticity of the image repair, and satisfy users Need for image restoration.
  • the electronic device can also input the model after structural repair to the texture repair model and/or color repair model to perform texture repair and/or color repair, so as to achieve the repair of the image to be repaired in terms of texture and/or color, and obtain Repaired images that better meet user needs.
  • the following takes an electronic device as a terminal as an example, as shown in Figure 1, to introduce the image repair method provided by the embodiment of the present disclosure.
  • the method includes the following steps:
  • S102 The terminal obtains the image to be repaired.
  • the image to be repaired can be an image with partial missing parts, or an image whose clarity does not meet the user's requirements.
  • the terminal can obtain the image to be repaired in a variety of ways. For example, the terminal may respond to the user's determination request, determine the image stored in the terminal as the image to be repaired, and then call the storage unit in the terminal to obtain the image to be repaired. The terminal may also respond to the user's determination request, determine the image stored in other devices as the image to be repaired, and then obtain the image to be repaired from the other device. The terminal can also obtain the image to be repaired by calling other components. For example, the terminal can take a paper photo through the camera to obtain the image to be repaired in digital format.
  • S104 The terminal inputs the image to be repaired into the structure repair model, downsamples the image to be repaired through multiple branches of the structure repair model, obtains the first feature sequence and the second feature sequence, and converts the first feature sequence into the second feature sequence. For a third feature sequence with the same sequence length, fuse the third feature sequence with the second feature sequence, perform structural repair on the image to be repaired based on the fused feature sequence, and obtain the first repaired image.
  • the first repaired image is an image in which the structure of the image to be repaired is repaired.
  • Converting the first feature sequence into a third feature sequence with the same length as the second feature sequence may include upsampling the first feature sequence to obtain the third feature sequence.
  • the conversion of the first feature sequence into a third feature sequence with the same length as the second feature sequence can also be performed by adding the first feature sequence and the fifth feature sequence and performing upsampling to obtain the third feature sequence, wherein the fifth feature sequence
  • the length of the feature sequence and the first feature sequence are the same.
  • the fifth feature sequence can be obtained by upsampling the fourth feature sequence.
  • the fourth feature sequence can be obtained by downsampling the image to be repaired by another branch of the structural repair model. .
  • the fusion of the third feature sequence and the second feature sequence may be adding the third feature sequence and the second feature sequence and encoding. and decoding.
  • the length of the first feature sequence is four times the length of the fourth feature sequence
  • the length of the fifth feature sequence is the same as the first feature sequence
  • the length of the second feature sequence is four times the length of the first feature sequence.
  • the length of the third characteristic sequence is the same as the length of the second characteristic sequence.
  • the terminal inputs the image to be repaired into the structure repair model.
  • Multiple branches of the structure repair model respectively downsample the image to be repaired through convolution to obtain corresponding feature images, and then proceed to the next step. Sampling to obtain smaller feature maps. Then the feature map is flattened to obtain the first feature sequence, the second feature sequence and the fourth feature sequence of different lengths.
  • the structure repair model encodes and decodes the fourth feature sequence and converts it into a fifth feature sequence that is the same length as the first feature sequence. Then the fifth feature sequence and the first feature sequence are added and encoded and decoded, and then the decoding result is upsampled to obtain the third feature sequence.
  • the structure repair model adds the third feature sequence and the second feature sequence and performs encoding and decoding to obtain the fused feature sequence. In this way, the structure of the image to be repaired can be repaired according to the fused feature sequence, and the first repaired image can be obtained.
  • the electronic device can also downsample the image to be repaired through multiple branches of the structural repair model to obtain a fourth feature sequence, where the fourth feature sequence, the first feature sequence, and the second feature sequence have both lengths. Are not the same.
  • the image to be repaired is down-sampled through three convolutional networks of different scales to obtain three feature maps with different scales.
  • the size of the image to be repaired is 256*256
  • the convolution network 1 (convolution1, conv1) is used for 4 times downsampling to obtain a feature map with a size of 64*64
  • the convolution network 2 (conv2) is used for 8 times downsampling.
  • Sampling a feature map with a size of 32*32 is obtained, and 16 times downsampling is performed through the convolutional network 3 (conv3) to obtain a feature map with a size of 16*16.
  • the structural repair model then downsamples the feature map and reduces the length to 1/2 of the original, obtaining feature maps with sizes of 32*32, 16*16 and 8*8 respectively. Then the above feature maps are flattened respectively, and the two-dimensional feature maps are converted into one-dimensional feature sequences (sequences), where the sequence lengths are 1024, 256 and 64 respectively. After the sequence of length 64 is encoded by N encoders and decoded by N decoders, the resulting sequence is upsampled to obtain a sequence of length 256. The length obtained by upsampling is The sequence of 256 is added to the sequence of length 256 obtained by flattening.
  • the resulting sequence is further upsampled to obtain a sequence of length 1024.
  • the length obtained by upsampling is The sequence of length 1024 is added to the sequence of length 1024 obtained by flattening.
  • the result feature sequence is obtained. Then, the structure of the image to be repaired is repaired according to the resulting feature sequence, and the first repaired image is obtained.
  • the structure repair model has multiple branches, multiple branches can obtain the structural features of the image to be repaired from different scales, so that the image to be repaired is repaired based on the features of different scales, making the structural repair of the image to be repaired more accurate and improving the image quality.
  • the accuracy of structural repair improves the user experience.
  • the structure repair model can learn the spatial layout information in the image, consider the uniform distribution characteristics of objects in the image, and repair the rough contours in the image.
  • the structure repair model can be obtained through training images.
  • the terminal may mask the training image to obtain the mask image, where the size of the training image may be 256*256.
  • the branches obtain training feature maps of different scales respectively. For example, through conv1, 4 times downsampling is performed to obtain a training feature map with a size of 64*64, and through conv2, 8 times downsampling is performed to obtain a training feature map with a size of 32*32.
  • conv3 performs 16 times downsampling to obtain a training feature map with a size of 16*16.
  • the structure repair model then downsamples the above training feature map, reducing the length of the training feature map to 1/2 of the original, and obtaining training feature maps with sizes of 32*32, 16*16 and 8*8 respectively. Then the above training feature maps are flattened respectively, and the two-dimensional training feature maps are converted into one-dimensional training feature sequences (sequences), where the sequence lengths are 1024, 256 and 64 respectively. After the sequence with a length of 64 is encoded by N encoders and decoded by N decoders, the obtained training result sequence is used to repair the mask image, and the first sub-repaired image is obtained, and then the first sub-repaired image is obtained. The sub-repaired image and the unmasked training image are calculated to obtain the first mean-squared loss (mse loss).
  • Mse loss mean-squared loss
  • the structure repair model upsamples the training result sequence to obtain a sequence of length 256, adds the sequence of length 256 obtained by upsampling and the sequence of length 256 obtained by flattening, and encodes the added result.
  • the masked image is repaired with the obtained training result sequence to obtain a second sub-repaired image, and then the second sub-repaired image and the unmasked training image are calculated to obtain a second mean square function.
  • the structure repair model further upsamples the obtained training result sequence to obtain a sequence of length 1024.
  • the sequence of length 1024 obtained by upsampling is added to the sequence of length 1024 obtained by flattening.
  • the result feature sequence is obtained.
  • the structure of the image to be repaired is repaired according to the resulting feature sequence, and the first repaired image is obtained.
  • the first repaired image and the unmasked training image are calculated to obtain a third mean square function.
  • the terminal can update the parameters of the structural model according to the first mean square function, the second mean square function and the third mean square function to achieve optimization of the structural repair model.
  • the terminal can use the first mean square function to optimize the structure to repair the branch where conv1 is located in the model, use the second mean square function to optimize the structure to repair the branches where conv1 and conv2 are in the model, and use the third mean square function to optimize the structure to repair the branch.
  • the terminal that executes the image repair method in this embodiment and the terminal that performs structural model training can be the same terminal, or they can be different terminals.
  • the terminal can transmit its trained structure repair model to multiple other terminals, so that multiple other terminals can directly use the structure repair model to implement the image repair method in the present disclosure.
  • the present disclosure provides an image repair method.
  • the terminal obtains the image to be repaired, then inputs the image to be repaired into the structural repair model, downsamples the image to be repaired through multiple branches of the structural repair model, obtains the first feature sequence and the second feature sequence, and converts the first feature sequence into A third feature sequence with the same length as the second feature sequence is fused with the second feature sequence, and the structure of the image to be repaired is performed according to the fused feature sequence to obtain an image in which the structure of the image to be repaired is repaired.
  • the image repair method also includes the following steps:
  • S406 The terminal inputs the first repaired image to the texture repair model to obtain the second repaired image.
  • the second repaired image is an image obtained by performing texture repair on the first repaired image.
  • the terminal inputs the first repaired image into the texture repair model, downsamples the first repaired image through the model, and then flattens it to obtain the sequence, sends the sequence to the encoder for encoding, and then converts the sequence.
  • the two-dimensional feature map is sent to the deconvolution layer for deconvolution, and then upsampled to obtain a feature map with the same size as the original image, so as to obtain the final feature map based on the fully connected (FC) layer.
  • FC fully connected
  • the first repaired image (image to be repaired) with a size of 256*256 is downsampled 8 times through a convolution layer to obtain a feature map with a size of 32*32, and then the feature map with a size of 32*32 is obtained.
  • the 32*32 feature map is flattened to obtain a sequence of length 1024.
  • the sequence of length 1024 is sent to N encoders for encoding, and the output of the encoder is converted into a two-dimensional feature map of size 32*32.
  • FC layer obtains the final result to realize texture repair of the first repaired image, and the repaired image is the second repaired image.
  • the texture inpainting model can be obtained by training on texture training images.
  • the texture training image can be masked, and the masked texture training image can be downsampled through the convolution layer to obtain the feature map of the image, and then the sequence obtained by flattening the feature map can be sent to the encoder for processing.
  • Encoding convert the output result of the encoder into a feature map and perform deconvolution, further perform upsampling, and finally predict the texture of the masked texture training image through the FC layer, and compare the prediction results with the texture training image. Thereby optimizing the parameters in the texture repair model.
  • S408 The terminal inputs the second repaired image to the color repair model to obtain the third repaired image.
  • the third repaired image is an image obtained by color repairing the second repaired image.
  • the terminal inputs the second repaired image into the color repair model, downsamples the second repaired image through the model, then flattens it to obtain the sequence, sends the sequence to the encoder for encoding, and then converts the sequence
  • the two-dimensional feature map is sent to the deconvolution layer for deconvolution, and then upsampled to obtain a feature map with the same size as the original image, so as to obtain the final result based on the fully connected layer to achieve the first 2.
  • Color restoration of repaired images are provided by color repairing the second repaired image.
  • the second repaired image (image to be repaired) with a size of 256*256 is downsampled 8 times through a convolution layer to obtain a feature map with a size of 32*32, and then the feature map with a size of 32*32 is obtained.
  • the 32*32 feature map is flattened to obtain a sequence of length 1024.
  • the sequence of length 1024 is sent to N encoders for encoding, and the output of the encoder is converted into a two-dimensional feature map of size 32*32.
  • FC layer obtains the final result to achieve color restoration of the second repaired image, and the repaired image is the third repaired image.
  • the color inpainting model can be obtained by training on color training images. Specifically, the color training image can be masked, and the masked color training image can be downsampled through the convolution layer to obtain the feature map of the image, and then the sequence obtained by flattening the feature map can be sent to the encoder for processing. Encoding, convert the output result of the encoder into a feature map and perform deconvolution, further perform upsampling, and finally predict the color of the masked color training image through the FC layer, and compare the prediction results with the color training image. Thereby optimizing the parameters in the color restoration model.
  • the terminal can perform texture repair on the first repaired image after structural repair through S406.
  • the terminal can also perform color repair on the first repaired image after structural repair through S408.
  • the terminal can also perform color repair on the first repaired image after structural repair. Texture repair is performed on the first repaired image after structural repair through S406, and color repair is performed on the second repaired image after texture repair through S408.
  • the terminal that executes the image repair method in this embodiment and the terminal that performs texture model training and color model training can be the same terminal, or they can be different terminals.
  • the terminal can transmit its trained texture repair model and/or color repair model to multiple other terminals, so that multiple other terminals can directly use the texture repair model and/or color repair model, The image repair method in the present disclosure is implemented.
  • the image repair method can gradually achieve accurate repair of the image to be repaired from the three aspects of the structure, texture and color of the image to be repaired from the whole to the part.
  • the structure repair model, texture repair model and color repair model are respectively obtained through the corresponding training images, so that the three models can learn the structural rules, texture rules and color rules of the image respectively.
  • Each model realizes the function corresponding to the model. Accurate repair, thus improving the accuracy of model repair.
  • Figure 6 is a schematic diagram of an image repair device according to an exemplary disclosed embodiment. As shown in Figure 6, the image repair device 600 includes:
  • Acquisition module 602 used to obtain the image to be repaired
  • the structure repair module 604 is used to input the image to be repaired into a structure repair model, and downsample the image to be repaired through multiple branches of the structure repair model to obtain a first feature sequence and a second feature sequence, Convert the first feature sequence into a third feature sequence with the same length as the second feature sequence, fuse the third feature sequence with the second feature sequence, and classify the to-be-identified sequence according to the fused feature sequence. Perform structural repair on the repaired image to obtain a first repaired image, where the first repaired image is an image that repairs the structure of the image to be repaired.
  • the device also includes:
  • a texture repair module and/or a color repair module configured to input the first repaired image into a texture repair model and/or a color repair model, perform texture repair and/or color repair, and obtain a second repaired image.
  • the repaired image is an image obtained by performing texture repair and/or color repair on the first repaired image.
  • the structural repair module 604 is also used to:
  • the image to be repaired is downsampled through multiple branches of the structure repair model to obtain a fourth feature sequence
  • the structural repair module is specifically used for:
  • the fourth feature sequence is upsampled and fused with the first feature sequence to obtain a third feature sequence with the same length as the second feature sequence.
  • the device also includes:
  • a texture repair module configured to input the first repaired image into a texture repair model, perform texture repair, and obtain a second repaired image, where the second repaired image is an image obtained by performing texture repair on the first repaired image;
  • a color repair module configured to input the second repaired image into a color repair model, perform color repair, and obtain a third repaired image, where the third repaired image is an image obtained by color repairing the second repaired image.
  • the length of the second characteristic sequence is four times the length of the first characteristic sequence.
  • the length of the first characteristic sequence is four times the length of the fourth characteristic sequence.
  • the structural repair module 604 is specifically used to:
  • the structural repair model is trained in the following manner:
  • training images including mask images
  • the mask image is downsampled through multiple branches of the structure repair model to obtain a first training feature sequence and a second training feature sequence, and the first training feature sequence is converted into the same as the second training feature sequence.
  • a third training feature sequence with the same sequence length fuse the third training feature sequence with the second training feature sequence, perform structural repair on the mask image according to the fused training feature sequence, and obtain the first training repair image;
  • the parameters of the structure repair model are updated according to the first training repair image and the training image before the mask.
  • the structural repair module 604 is specifically used to:
  • the first repaired image is input to a texture repair model and/or a color repair model, and the first repaired image is downsampled and encoded through the texture repair model and/or the color repair model to obtain a fifth feature sequence. , perform deconvolution on the fifth feature sequence to obtain a feature map, perform texture repair and/or color repair on the first repaired image according to the feature map, and obtain a second repaired image.
  • Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablets), PMPs (Portable Multimedia Players), vehicle-mounted terminals (such as Mobile terminals such as car navigation terminals) and fixed terminals such as digital TVs, desktop computers, etc.
  • the electronic device shown in FIG. 7 is only an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.
  • the electronic device 700 may include a processing device (eg, central processing unit, graphics processor, etc.) 701 that may be loaded into a random access device according to a program stored in a read-only memory (ROM) 702 or from a storage device 708 .
  • the program in the memory (RAM) 703 executes various appropriate actions and processes.
  • various programs and data required for the operation of the electronic device 700 are also stored.
  • the processing device 701, the ROM 702 and the RAM 703 are connected to each other via a bus 704.
  • An input/output (I/O) interface 705 is also connected to bus 704.
  • the following devices may be connected to the I/O interface 705: input devices 706 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 707 such as a computer; a storage device 708 including a magnetic tape, a hard disk, etc.; and a communication device 709. Communication device 709 may allow electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data.
  • FIG. 7 illustrates an electronic device 700 having various means, it should be understood that implementation or availability of all illustrated means is not required. More or fewer means may alternatively be implemented or provided.
  • embodiments of the present disclosure include a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart.
  • the computer program may be downloaded and installed from the network via communication device 709, or from storage device 708, or from ROM 702.
  • the processing device 701 the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
  • the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
  • the computer-readable storage medium may be, for example, but not limited to ——Electric, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard drive, random access memory (RAM), read only memory (ROM), removable Programmd read-only memory (EPROM or flash memory), fiber optics, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above.
  • a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device .
  • Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wire, optical cable, RF (radio frequency), etc., or any suitable combination of the above.
  • the client and server can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can communicate with digital data in any form or medium.
  • Communications e.g., communications network
  • communications networks include local area networks (“LAN”), wide area networks (“WAN”), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or developed in the future network of.
  • the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; it may also exist independently without being assembled into the electronic device.
  • the computer-readable medium carries one or more programs.
  • the electronic device When the one or more programs are executed by the electronic device, the electronic device: performs text detection on the image, obtains the text area in the image, and the text
  • the region includes a plurality of text lines; a graph network model is constructed according to the text region, and each text behavior in the text region is a node of the graph network model; the nodes in the graph network model are processed through a node classification model Classify, and classify edges between nodes in the graph network model through an edge classification model; obtain at least one key value in the image based on the classification results of the nodes and the classification results of the edges. right.
  • Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages - such as "C" or similar programming languages.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider). connected via the Internet).
  • LAN local area network
  • WAN wide area network
  • Internet service provider such as an Internet service provider
  • each box in the flowchart or block diagram may represent a module, segment, or portion of code that contains one or more Executable instructions used to implement specified logical functions.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved.
  • each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration can be implemented by special purpose hardware-based systems that perform the specified functions or operations. , or can be implemented using a combination of specialized hardware and computer instructions.
  • the modules involved in the embodiments of the present disclosure can be implemented in software or hardware. Among them, the name of the module does not constitute a limitation on the module itself under certain circumstances.
  • FPGAs Field Programmable Gate Arrays
  • ASICs Application Specific Integrated Circuits
  • ASSPs Application Specific Standard Products
  • SOCs Systems on Chips
  • CPLD Complex Programmable Logical device
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
  • the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any suitable combination of the foregoing.
  • machine-readable storage media would include one or more wire-based electrical connections, laptop disks, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • RAM random access memory
  • ROM read only memory
  • EPROM or flash memory erasable programmable read only memory
  • CD-ROM portable compact disk read-only memory
  • magnetic storage device or any suitable combination of the above.
  • Example 1 provides an image repair method.
  • the method includes: acquiring an image to be repaired; inputting the image to be repaired into a structural repair model, and using Multiple branches downsample the image to be repaired to obtain a first feature sequence and a second feature sequence, convert the first feature sequence into a third feature sequence with the same length as the second feature sequence, and convert the first feature sequence into a third feature sequence with the same length as the second feature sequence.
  • the third feature sequence is fused with the second feature sequence, and the image to be repaired is structurally repaired according to the fused feature sequence to obtain a first repaired image, where the first repaired image is the image to be repaired. Image of the structure being repaired.
  • Example 2 provides the method of Example 1, which method further includes: inputting the first repaired image to a texture repair model and/or a color repair model, and performing texture repair and/or color repair. or color repair, to obtain a second repaired image, where the second repaired image is an image obtained by performing texture repair and/or color repair on the first repaired image.
  • Example 3 provides the method of Example 1, which method further includes: downsampling the image to be repaired through multiple branches of the structural repair model to obtain a fourth feature sequence. ; Converting the first feature sequence into a third feature sequence with the same length as the second feature sequence includes: upsampling the fourth feature sequence and fusing it with the first feature sequence to obtain The second characteristic sequence is a third characteristic sequence with the same length.
  • Example 4 provides the method of Example 1.
  • the method further includes: inputting the first repaired image to a texture repair model, performing texture repair, and obtaining a second repaired image. Describe the second practice
  • the complex image is an image obtained by texture repair of the first repaired image; the second repaired image is input to the color repair model, color repair is performed, and a third repaired image is obtained, and the third repaired image is the image of the third repaired image. 2. Repair the image to perform color repair on the image.
  • Example 5 provides the method of Example 1, the length of the second feature sequence is four times the length of the first feature sequence.
  • Example 6 provides the method of Example 3, the length of the first feature sequence is four times the length of the fourth feature sequence.
  • Example 7 provides the method of Example 1, the fusing the third feature sequence with the second feature sequence includes: fusing the third feature sequence with the The second feature sequences are added, encoded and decoded to obtain the fused feature sequence.
  • Example 8 provides the method of Example 1.
  • the structure repair model is trained in the following manner: obtaining a training image, the training image including a mask image; Multiple branches of the model downsample the mask image to obtain a first training feature sequence and a second training feature sequence, and convert the first training feature sequence into a third training feature sequence with the same length as the second training feature sequence.
  • Example 9 provides the method of Example 2, which includes inputting the first repaired image to a texture repair model and/or a color repair model to perform texture repair and/or color repair,
  • Obtaining the second repaired image includes: inputting the first repaired image into a texture repair model and/or a color repair model, and performing downstream processing on the first repaired image through the texture repair model and/or the color repair model.
  • Sampling and encoding obtain a fifth feature sequence, performing deconvolution on the fifth feature sequence to obtain a feature map, performing texture repair and/or color repair on the first repaired image according to the feature map, and obtaining a second repaired image.
  • Example 10 provides an image repair device.
  • the device includes: an acquisition module for acquiring an image to be repaired; a structure repair module for inputting the image to be repaired into A structural repair model, which downsamples the image to be repaired through multiple branches of the structural repair model to obtain a first feature sequence and a second feature sequence, and converts the first feature sequence into the same sequence as the second feature sequence.
  • a repaired image is an image that repairs the structure of the image to be repaired.
  • Example 11 provides the device of Example 10, the device further comprising: a texture repair module and/or a color repair module for inputting the first repair image to a texture repair model. and/or color repair model, perform texture repair and/or color repair, and obtain a second repair image, where the second repair image is an image obtained by performing texture repair and/or color repair on the first repair image.
  • Example 12 provides the device of Example 10, and the structure repair module is further configured to: downsample the image to be repaired through multiple branches of the structure repair model to obtain a third Four feature sequences; the structure repair module is specifically used to: upsample the fourth feature sequence and fuse it with the first feature sequence, Obtain a third feature sequence with the same length as the second feature sequence.
  • Example 13 provides the device of Example 10, the device further comprising: a texture repair module, configured to input the first repaired image to a texture repair model, perform texture repair, and obtain a second repaired image, where the second repaired image is an image obtained by performing texture repair on the first repaired image; a color repair module configured to input the second repaired image into a color repair model, perform color repair, and obtain the second repaired image.
  • a texture repair module configured to input the first repaired image to a texture repair model, perform texture repair, and obtain a second repaired image, where the second repaired image is an image obtained by performing texture repair on the first repaired image
  • a color repair module configured to input the second repaired image into a color repair model, perform color repair, and obtain the second repaired image.
  • Three repaired images the third repaired image is an image obtained by color repairing the second repaired image.
  • Example 14 provides the device of Example 10, the length of the second feature sequence is four times the length of the first feature sequence.
  • Example 15 provides the device of Example 12, the length of the first feature sequence is four times the length of the fourth feature sequence.
  • Example 16 provides the device of Example 10, the structure repair module is specifically configured to: add the third feature sequence and the second feature sequence, and perform encoding and Decode to obtain the fused feature sequence.
  • Example 17 provides the device of Example 10.
  • the structure repair model is trained in the following manner: acquiring a training image, the training image including a mask image; Multiple branches of the model downsample the mask image to obtain a first training feature sequence and a second training feature sequence, and convert the first training feature sequence into a third training feature sequence with the same length as the second training feature sequence. Three training feature sequences, fuse the third training feature sequence with the second training feature sequence, perform structural repair on the mask image according to the fused training feature sequence, and obtain the first training repair image; according to the first Training the inpainted image and the training image before masking update the parameters of the structure inpainting model.
  • Example 18 provides the device of Example 11, the structure repair module is specifically configured to: input the first repair image to a texture repair model and/or a color repair model, through the The texture repair model and/or the color repair model performs downsampling and encoding on the first repaired image to obtain a fifth feature sequence, and performs deconvolution on the fifth feature sequence to obtain a feature map. According to the feature map Perform texture repair and/or color repair on the first repaired image to obtain a second repaired image.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Image Processing (AREA)

Abstract

本公开提供了一种图像修复方法、装置、设备、介质以及产品，该方法包括：获取待修复图像，然后将待修复图像输入至结构修复模型，通过结构修复模型的多个分支对待修复图像进行下采样，获得第一特征序列和第二特征序列，将第一特征序列转化为与第二特征序列长度相同的第三特征序列，将第三特征序列与第二特征序列融合，根据融合后的特征序列对待修复图像进行结构修复，获得对待修复图像的结构进行修复的图像。如此，可以获得修复精度较高，效果较好的修复图像。

Description

图像修复方法、装置、设备、介质及产品
本公开要求于2022年3月21日提交的申请号为202210278165.X、申请名称为“图像修复方法、装置、设备、介质及产品”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本公开涉及计算机技术领域，尤其涉及一种图像修复方法、装置、设备、计算机可读存储介质以及计算机程序产品。
背景技术
随着图像处理技术的不断成熟，用户对于通过图像处理技术进行图像修复的修复效果提出了更高的要求。图像修复是指基于图像中的已知信息去还原图像中的未知信息，实现对于图像中缺失部分的修复。
通常情况下的图像修复技术是确定出待修复图像中的参考区域与待修复区域，通过神经网络模型，根据参考区域的像素值确定待修复区域的像素值进行图像修复。但是这种图像修复技术可能导致修复区域存在波纹、扭曲等失真情况，不满足用户对于图像修复效果的要求。如何提高图像修复的效果成为亟需解决的问题。
发明内容
本公开的目的在于：提供了一种图像修复方法、装置、设备、计算机可读存储介质以及计算机程序产品，能够从图像整体的角度对图像进行修复，获得更加真实的修复效果。
第一方面，本公开提供了一种图像修复方法，所述方法包括：
获取待修复图像；
将所述待修复图像输入至结构修复模型，通过所述结构修复模型的多个分支对所述待修复图像进行下采样，得到第一特征序列和第二特征序列，将所述第一特征序列转化为与所述第二特征序列长度相同的第三特征序列，将所述第三特征序列与所述第二特征序列融合，根据融合后的特征序列对所述待修复图像进行结构修复，获得第一修复图像，所述第一修复图像为对所述待修复图像的结构进行修复的图像。
第二方面，本公开提供了一种图像修复装置，所述装置包括：
获取模块，用于获取待修复图像；
结构修复模块，用于将所述待修复图像输入至结构修复模型，通过所述结构修复模型的多个分支对所述待修复图像进行下采样，得到第一特征序列和第二特征序列，将所述第一特征序列转化为与所述第二特征序列长度相同的第三特征序列，将所述第三特征序列与所述第二特征序列融合，根据融合后的特征序列对所述待修复图像进行结构修复，获得第一修复图像，所述第一修复图像为对所述待修复图像的结构进行修复的图像。
第三方面，本公开提供一种电子设备，包括：存储装置，其上存储有计算机程序；处理装置，用于执行所述存储装置中的所述计算机程序，以实现本公开第一方面所述方法的步骤。
第四方面，本公开提供一种计算机可读介质，其上存储有计算机程序，该程序被处理装置执行时实现本公开第一方面所述方法的步骤。
第五方面，本公开提供了一种包含指令的计算机程序产品，当其在设备上运行时，使得设备执行上述第一方面所述方法的步骤。
从以上技术方案可以看出，本公开至少具有如下优点：
在上述技术方案中，电子设备获取待修复图像，然后将待修复图像输入至结构修复模型，通过结构修复模型的多个分支对待修复图像进行下采样，获得第一特征序列和第二特征序列，将第一特征序列转化为与第二特征序列长度相同的第三特征序列，将第三特征序列与第二特征序列融合，根据融合后的特征序列对待修复图像进行结构修复，获得对待修复图像的结构进行修复的图像。由于结构修复模型中的多个分支可以对待修复图像进行不同尺度的下采样，然后提取不同尺度下待修复图像的特征，并根据融合后的结果对待修复图像进行修复，因此可以获得修复精度较高，效果较好的修复图像。
本公开的其他特征和优点将在随后的具体实施方式部分予以详细说明。
附图说明
为了更清楚地说明本公开实施例的技术方法，下面将对实施例中所需使用的附图作以简单地介绍。
图1为本公开实施例提供的一种图像修复方法的流程示意图；
图2为本公开实施例提供的一种结构修复模型的示意图；
图3为本公开实施例提供的另一种结构修复模型的示意图；
图4为本公开实施例提供的另一种图像修复方法的流程示意图；
图5为本公开实施例提供的一种纹理/色彩修复模型的示意图；
图6为本公开实施例提供的一种图像修复装置的结构示意图；
图7为本公开实施例提供的一种电子设备的结构示意图。
具体实施方式
本公开实施例中的术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。
首先对本公开实施例中所涉及到的一些技术术语进行介绍。
图像处理(image processing)技术一般是对数字图像进行处理，具体是指通过计算机对于数字图像进行分析处理的技术。图像处理技术可以对于图像进行多种类型的处理，例如对存在缺失部分的图像进行修复，即图像修复技术。
随着图像处理技术的不断发展，用户对于图像修复技术的效果提出了更高的要求。图像修复技术是针对于待修复图像，确定待修复图像中的修复区域以及参考区域，其中修复区域可以为部分图案存在缺失的区域，也可以为清晰度不满足用户需求的区域。
通常情况下的图像修复技术可以根据待修复图像中参考区域的像素值，通过神经网络模型直接预测修复区域的像素值，从而实现对于待修复图像修复区域的修复。
但是这种直接通过模型预测修复区域像素值的修复方法，仅从像素值的角度对图像进行修复，可能导致修复后区域存在波纹、扭曲等失真的情况，不满足用户对于图像修复真实度的效果要求。
有鉴于此，本公开提供了一种图像修复方法，该方法应用于电子设备。电子设备是指具有数据处理能力的设备，例如可以是服务器，或者是终端。其中，终端包括但不限于智能手机、平板电脑、笔记本电脑、个人数字助理(personal digital assistant，PDA)或者智能穿戴设备等。服务器可以是云服务器，例如是中心云计算集群中的中心服务器，或者是边缘云计算集群中的边缘服务器。当然，服务器也可以是本地数据中心中的服务器。本地数据中心是指用户直接控制的数据中心。
具体地，电子设备获取待修复图像，然后将待修复图像输入至结构修复模型，通过结构修复模型的多个分支对待修复图像进行下采样，得到第一特征序列和第二特征序列，然后将第一特征序列转化为与第二特征序列长度相同的第三特征序列，将第三特征序列与第二特征序列进行融合，根据融合后的特征序列对待修复图像进行结构修复，从而获得对待修复图像的结构进行修复的第一修复图像。
如此，结构修复模型可以根据不同尺度的图像特征对待修复图像进行修复，融合不同尺度的特征对待修复模型进行结构修复，实现从结构方面对于待修复图像的修复，提高图像修复的真实性，满足用户对于图像修复的需求。
进一步地，电子设备还可以将进行结构修复后的模型输入至纹理修复模型和/或色彩修复模型进行纹理修复和/或色彩修复，实现对于待修复图像从纹理和/或色彩方面的修复，获取更满足用户需求的修复后的图像。
为了使得本公开的技术方案更加清楚、易于理解，下面从电子设备为终端为例，如图1所示，对本公开实施例提供的图像修复方法进行介绍，该方法包括以下步骤：
S102：终端获取待修复图像。
其中，待修复图像可以为存在部分缺失的图像，也可以为清晰度不满足用户要求的图像。终端可以通过多种方式获取待修复图像。例如，终端可以响应于用户的确定请求，将该终端中所存储的图像确定为待修复图像，然后调用终端中的存储单元获取待修复图像。终端也可以响应于用户的确定请求，将其他设备所存储的图像确定为待修复图像，然后从其他设备中获取待修复图像。终端也可以通过调用其他组件获取待修复图像，例如终端可以通过摄像头拍摄纸质照片，获得数字格式的待修复图像。
S104：终端将待修复图像输入至结构修复模型，通过结构修复模型的多个分支对待修复图像进行下采样，得到第一特征序列和第二特征序列，将第一特征序列转化为与第二特征序列长度相同的第三特征序列，将第三特征序列与第二特征序列融合，根据融合后的特征序列对待修复图像进行结构修复，获得第一修复图像。
其中，第一修复图像为对待修复图像的结构进行修复的图像。所述将第一特征序列转化为与第二特征序列长度相同的第三特征序列可以为对第一特征序列进行上采样，获得第三特征序列。所述将第一特征序列转化为与第二特征序列长度相同的第三特征序列也可以为将第一特征序列与第五特征序列相加并进行上采样，获得第三特征序列，其中第五特征序列和第一特征序列的长度相同，第五特征序列可以为第四特征序列进行上采样后获得的，第四特征序列可以为结构修复模型的另一分支对待修复图像进行下采样后获得的。所述将第三特征序列与第二特征序列融合可以为将第三特征序列与第二特征序列相加并进行编码 与解码。
在一些可能的实现方式中，第一特征序列的长度为第四特征序列的长度的四倍，第五特征序列与第一特征序列的长度相同，第二特征序列的长度为第一特征序列的长度的四倍，第三特征序列的长度与第二特征序列的长度相同。
示例性地，如图2所示，终端将待修复图像输入至结构修复模型，结构修复模型的多个分支分别对待修复图像通过卷积进行下采样，获得分别对应的特征图像，然后再进行下采样，获得更小的特征图。然后将特征图压平获得不同长度的第一特征序列、第二特征序列和第四特征序列。
结构修复模型将第四特征序列经过编码以及解码，并转化为与第一特征序列长度相同的第五特征序列。然后将第五特征序列和第一特征序列相加并进行编码与解码，然后将解码结果进行上采样，获得第三特征序列。结构修复模型将第三特征序列与第二特征序列相加并进行编码解码，获得融合后的特征序列。如此可以根据融合后的特征序列对待修复图像进行结构修复，获得第一修复图像。
在一些可能的实现方式中，电子设备还可以通过结构修复模型的多个分支对待修复图像进行下采样获得第四特征序列，其中第四特征序列和第一特征序列，第二特征序列的长度均不相同。
如图3所示，在结构修复模型中，待修复图像经过3个不同尺度的卷积网络下采样获得三个尺度不同的特征图。例如，待修复图像的尺寸为256*256，经过卷积网络1(convolution1，conv1)进行4倍下采样，获得尺寸为64*64的特征图，经过卷积网络2(conv2)进行8倍下采样，获得尺寸为32*32的特征图，经过卷积网络3(conv3)进行16倍下采样，获得尺寸为16*16的特征图。
结构修复模型接着对于特征图再进行下采样，长度再缩小为原来的1/2，获得尺寸分别为32*32、16*16和8*8的特征图。再分别对上述特征图进行压平(flatten)，将二维特征图转换为一维特征序列(sequence)，其中序列长度分别为1024、256和64。将其中长度为64的序列经过N个编码(encoder)器编码和N个解码(decoder)器解码后，对得到的结果序列进行上采样，获得长度为256的序列，将上采样获得的长度为256的序列与压平获得的长度为256的序列相加，对相加后的结果进行编码与解码后，得到的结果序列进一步进行上采样，获得长度为1024的序列，将上采样获得的长度为1024的序列与压平获得的长度为1024的序列相加，对相加后的结果进行编码与解码后，得到结果特征序列。然后根据结果特征序列对待修复图像的结构进行修复，获得第一修复图像。
由于结构修复模型具有多个分支，多个分支可以从不同尺度获取待修复图像的结构特征，从而基于不同尺度的特征对于待修复图像进行修复，使待修复图像的结构修复更加准确，提高了图像结构修复的准确性，提高了用户的使用体验。并且，结构修复模型可以学习图像中的空间布局信息，考虑图像中物体均匀分布的特性，实现对于图像中大致轮廓的修复。
其中，结构修复模型可以通过训练图像训练获得。示例性地，终端可以将训练图像掩码，获得掩码图像，其中训练图像的尺寸可以为256*256。然后通过结构修复模型的三个 分支分别获得不同尺度的训练特征图，例如经过conv1进行4倍下采样，获得尺寸为64*64的训练特征图，经过conv2进行8倍下采样，获得尺寸为32*32的训练特征图，经过conv3进行16倍下采样，获得尺寸为16*16的训练特征图。
结构修复模型接着对于上述训练特征图再进行下采样，将训练特征图长度再缩小为原来的1/2，获得尺寸分别为32*32、16*16和8*8的训练特征图。再分别对上述训练特征图进行压平(flatten)，将二维训练特征图转换为一维训练特征序列(sequence)，其中序列长度分别为1024、256和64。将其中长度为64的序列经过N个编码(encoder)器编码和N个解码(decoder)器解码后，获得的训练结果序列对掩码图像进行修复，获得第一子修复图像，然后将第一子修复图像与未进行掩码的训练图像计算获得第一均方函数(mean-squared loss，mse loss)。
同时，结构修复模型将训练结果序列进行上采样获得长度为256的序列，将上采样获得的长度为256的序列与压平获得的长度为256的序列相加，对相加后的结果进行编码与解码后，得到的训练结果序列对掩码图像进行修复，获得第二子修复图像，然后将第二子修复图像与未进行掩码的训练图像计算获取第二均方函数。
并且，结构修复模型对得到的训练结果序列进一步进行上采样，获得长度为1024的序列，将上采样获得的长度为1024的序列与压平获得的长度为1024的序列相加，对相加后的结果进行编码与解码后，得到结果特征序列。然后根据结果特征序列对待修复图像的结构进行修复，获得第一修复图像。将第一修复图像与未进行掩码的训练图像计算获取第三均方函数。
如此，终端可以根据第一均方函数、第二均方函数以及第三均方函数更新结构模型的参数，以实现对于结构修复模型的优化。具体地，终端可以通过第一均方函数优化结构修复模型中conv1所在的支路，通过第二均方函数优化结构修复模型中conv1和conv2所在的支路，通过第三均方函数优化结构修复模型中conv1、conv2和conv3所在的支路。
其中，执行本实施例中图像修复方法的终端和进行结构模型训练的终端可以为同一终端，也可以为不同终端。在一些可能的实现方式中，终端可以将其训练完成的结构修复模型传输至多个其他终端，以使多个其他终端可以直接使用该结构修复模型，实现本公开中的图像修复方法。
基于以上内容的描述，本公开提供了一种图像修复方法。终端获取待修复图像，然后将待修复图像输入至结构修复模型，通过结构修复模型的多个分支对待修复图像进行下采样，获得第一特征序列和第二特征序列，将第一特征序列转化为与第二特征序列长度相同的第三特征序列，将第三特征序列与第二特征序列融合，根据融合后的特征序列对待修复图像进行结构修复，获得对待修复图像的结构进行修复的图像。由于结构修复模型中的多个分支可以对待修复图像进行不同尺度的下采样，然后提取不同尺度下待修复图像的特征，并根据融合后的结果对待修复图像进行修复，因此可以获得修复精度较高，效果较好的修复图像。
在一些可能的实现方式中，如图4所示，该图像修复方法还包括以下步骤：
S406：终端将第一修复图像输入至纹理修复模型，获得第二修复图像。
其中，第二修复图像为对第一修复图像进行纹理修复所获得的图像。具体地，终端将第一修复图像输入至纹理修复模型中，通过该模型对第一修复图像进行下采样，然后压平获得序列，将该序列送入至编码器中进行编码，然后将序列转化为二维特征图，将二维特征图送入反卷积层进行反卷积，接着进行上采样，获得与原图大小相同的特征图，从而根据全连接(fully connected，FC)层获取最终的结果，实现对于第一修复图像的纹理修复。
示例性地，如图5所示，将尺寸为256*256的第一修复图像(待修复图像)经过卷积层进行8倍下采样，获得尺寸为32*32的特征图，然后将尺寸为32*32的特征图压平获得长度为1024的序列，将该长度为1024的序列送入至N个编码器中进行编码，将编码器的输出结果转化为尺寸32*32的二维特征图，然后将该二维特征图进行反卷积，获得尺寸为64*64的特征图，进一步对该尺寸为64*64的特征图进行上采样，获得尺寸为256*256的特征图，然后通过FC层获取最终结果，实现对于第一修复图像的纹理修复，修复后的图像为第二修复图像。
纹理修复模型可以通过纹理训练图像训练获得。具体地，可以对于纹理训练图像进行掩码，将掩码后的纹理训练图像经过卷积层进行下采样，获得该图像的特征图，然后将特征图压平获得的序列送入编码器中进行编码，将编码器输出的结果转换为特征图并进行反卷积，进一步进行上采样，最终通过FC层对掩码后的纹理训练图像的纹理进行预测，将预测结果与纹理训练图像进行对比，从而优化纹理修复模型中的参数。
S408：终端将第二修复图像输入至色彩修复模型，获得第三修复图像。
其中，第三修复图像为对第二修复图像进行色彩修复所获得的图像。具体地，终端将第二修复图像输入至色彩修复模型中，通过该模型对第二修复图像进行下采样，然后压平获得序列，将该序列送入至编码器中进行编码，然后将序列转化为二维特征图，将二维特征图送入反卷积层进行反卷积，接着进行上采样，获得与原图大小相同的特征图，从而根据全连接层获取最终的结果，实现对于第二修复图像的色彩修复。
示例性地，如图5所示，将尺寸为256*256的第二修复图像(待修复图像)经过卷积层进行8倍下采样，获得尺寸为32*32的特征图，然后将尺寸为32*32的特征图压平获得长度为1024的序列，将该长度为1024的序列送入至N个编码器中进行编码，将编码器的输出结果转化为尺寸32*32的二维特征图，然后将该二维特征图进行反卷积，获得尺寸为64*64的特征图，进一步对该尺寸为64*64的特征图进行上采样，获得尺寸为256*256的特征图，然后通过FC层获取最终结果，实现对于第二修复图像的色彩修复，修复后的图像为第三修复图像。
色彩修复模型可以通过色彩训练图像训练获得。具体地，可以对于色彩训练图像进行掩码，将掩码后的色彩训练图像经过卷积层进行下采样，获得该图像的特征图，然后将特征图压平获得的序列送入编码器中进行编码，将编码器输出的结果转换为特征图并进行反卷积，进一步进行上采样，最终通过FC层对掩码后的色彩训练图像的色彩进行预测，将预测结果与色彩训练图像进行对比，从而优化色彩修复模型中的参数。
以上S406和S408为可选步骤，终端可以通过S406对结构修复后的第一修复图像进行纹理修复，终端也可以通过S408对结构修复后的第一修复图像进行色彩修复，终端也可以 通过S406对结构修复后的第一修复图像进行纹理修复，通过S408对纹理修复后的第二修复图像进行色彩修复。执行本实施例中图像修复方法的终端和进行纹理模型训练以及色彩模型训练的终端可以为同一终端，也可以为不同终端。在一些可能的实现方式中，终端可以将其训练完成的纹理修复模型和/或色彩修复模型传输至多个其他终端，以使多个其他终端可以直接使用该纹理修复模型和/或色彩修复模型，实现本公开中的图像修复方法。
当该方法包括S406和S408两个步骤时，该图像修复方法可以从待修复图像的结构、纹理和色彩三个方面从整体到局部逐步实现对于待修复图像的准确修复。其中，结构修复模型、纹理修复模型和色彩修复模型分别通过对应的训练图像获得，使三种模型可以分别学习图像的结构规律、纹理规律以及色彩规律，每一个模型实现对于该模型对应的功能的准确修复，从而提高了模型修复的准确性。
图6是根据一示例性公开实施例示出的一种图像修复装置的示意图，如图6所示，所述图像修复装置600包括：
获取模块602，用于获取待修复图像；
结构修复模块604，用于将所述待修复图像输入至结构修复模型，通过所述结构修复模型的多个分支对所述待修复图像进行下采样，得到第一特征序列和第二特征序列，将所述第一特征序列转化为与所述第二特征序列长度相同的第三特征序列，将所述第三特征序列与所述第二特征序列融合，根据融合后的特征序列对所述待修复图像进行结构修复，获得第一修复图像，所述第一修复图像为对所述待修复图像的结构进行修复的图像。
可选地，所述装置还包括：
纹理修复模块和/或色彩修复模块，用于将所述第一修复图像输入至纹理修复模型和/或色彩修复模型，进行纹理修复和/或色彩修复，获得第二修复图像，所述第二修复图像为对所述第一修复图像进行纹理修复和/或色彩修复的图像。
可选地，所述结构修复模块604还用于：
通过所述结构修复模型的多个分支对所述待修复图像进行下采样得到第四特征序列；
所述结构修复模块具体用于：
将所述第四特征序列上采样并与所述第一特征序列融合，获得与所述第二特征序列长度相同的第三特征序列。
可选地，所述装置还包括：
纹理修复模块，用于将所述第一修复图像输入至纹理修复模型，进行纹理修复，获得第二修复图像，所述第二修复图像为对所述第一修复图像进行纹理修复的图像；
色彩修复模块，用于将所述第二修复图像输入至色彩修复模型，进行色彩修复，获得第三修复图像，所述第三修复图像为对所述第二修复图像进行色彩修复的图像。
可选地，所述第二特征序列的长度为所述第一特征序列长度的四倍。
可选地，所述第一特征序列的长度为所述第四特征序列长度的四倍。
可选地，所述结构修复模块604具体用于：
将所述第三特征序列与所述第二特征序列相加，并进行编码与解码，获得融合后的特征序列。
可选地，所述结构修复模型通过以下方式训练得到：
获取训练图像，所述训练图像中包括掩码图像；
通过所述结构修复模型的多个分支对所述掩码图像进行下采样，得到第一训练特征序列和第二训练特征序列，将所述第一训练特征序列转化为与所述第二训练特征序列长度相同的第三训练特征序列，将所述第三训练特征序列与所述第二训练特征序列融合，根据融合后的训练特征序列对所述掩码图像进行结构修复，获得第一训练修复图像；
根据第一训练修复图像和掩码前的训练图像更新所述结构修复模型的参数。
可选地，所述结构修复模块604具体用于：
将所述第一修复图像输入至纹理修复模型和/或色彩修复模型，通过所述纹理修复模型和/或所述色彩修复模型对所述第一修复图像进行下采样和编码获得第五特征序列，将所述第五特征序列进行反卷积获得特征图，根据所述特征图对所述第一修复图像进行纹理修复和/或色彩修复，获得第二修复图像。
上述各模块的功能在上一实施例中的方法步骤中已详细阐述，在此不做赘述。
下面参考图7，其示出了适于用来实现本公开实施例的电子设备700的结构示意图。本公开实施例中的终端设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、PDA(个人数字助理)、PAD(平板电脑)、PMP(便携式多媒体播放器)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字TV、台式计算机等等的固定终端。图7示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图7所示，电子设备700可以包括处理装置(例如中央处理器、图形处理器等)701，其可以根据存储在只读存储器(ROM)702中的程序或者从存储装置708加载到随机访问存储器(RAM)703中的程序而执行各种适当的动作和处理。在RAM703中，还存储有电子设备700操作所需的各种程序和数据。处理装置701、ROM702以及RAM703通过总线704彼此相连。输入/输出(I/O)接口705也连接至总线704。
通常，以下装置可以连接至I/O接口705：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置706；包括例如液晶显示器(LCD)、扬声器、振动器等的输出装置707；包括例如磁带、硬盘等的存储装置708；以及通信装置709。通信装置709可以允许电子设备700与其他设备进行无线或有线通信以交换数据。虽然图7示出了具有各种装置的电子设备700，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在非暂态计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置709从网络上被下载和安装，或者从存储装置708被安装，或者从ROM 702被安装。在该计算机程序被处理装置701执行时，执行本公开实施例的方法中限定的上述功能。
需要说明的是，本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于 ——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、RF(射频)等等，或者上述的任意合适的组合。
在一些实施方式中，客户端、服务器可以利用诸如HTTP(HyperText Transfer Protocol，超文本传输协议)之类的任何当前已知或未来研发的网络协议进行通信，并且可以与任意形式或介质的数字数据通信(例如，通信网络)互连。通信网络的示例包括局域网(“LAN”)，广域网(“WAN”)，网际网(例如，互联网)以及端对端网络(例如，ad hoc端对端网络)，以及任何当前已知或未来研发的网络。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备：对图像进行文本检测，获得所述图像中的文本区域，所述文本区域包括多个文本行；根据所述文本区域构建图网络模型，所述文本区域中的每个文本行为所述图网络模型的一个节点；通过节点分类模型对所述图网络模型中的节点进行分类，以及通过边分类模型对所述图网络模型中的节点之间的边进行分类；根据对所述节点的分类结果以及对所述边的分类结果，获得所述图像中的至少一个键值对。可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码，上述程序设计语言包括但不限于面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言——诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)——连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个 用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的模块可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，模块的名称在某种情况下并不构成对该模块本身的限定。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如，非限制性地，可以使用的示范类型的硬件逻辑部件包括：现场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、片上系统(SOC)、复杂可编程逻辑设备(CPLD)等等。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
根据本公开的一个或多个实施例，示例1提供了一种图像修复方法，所述方法包括：获取待修复图像；将所述待修复图像输入至结构修复模型，通过所述结构修复模型的多个分支对所述待修复图像进行下采样，得到第一特征序列和第二特征序列，将所述第一特征序列转化为与所述第二特征序列长度相同的第三特征序列，将所述第三特征序列与所述第二特征序列融合，根据融合后的特征序列对所述待修复图像进行结构修复，获得第一修复图像，所述第一修复图像为对所述待修复图像的结构进行修复的图像。
根据本公开的一个或多个实施例，示例2提供了示例1的方法，所述方法还包括：将所述第一修复图像输入至纹理修复模型和/或色彩修复模型，进行纹理修复和/或色彩修复，获得第二修复图像，所述第二修复图像为对所述第一修复图像进行纹理修复和/或色彩修复的图像。
根据本公开的一个或多个实施例，示例3提供了示例1的方法，所述方法还包括：通过所述结构修复模型的多个分支对所述待修复图像进行下采样得到第四特征序列；所述将所述第一特征序列转化为与所述第二特征序列长度相同的第三特征序列，包括：将所述第四特征序列上采样并与所述第一特征序列融合，获得与所述第二特征序列长度相同的第三特征序列。
根据本公开的一个或多个实施例，示例4提供了示例1的方法，所述方法还包括：将所述第一修复图像输入至纹理修复模型，进行纹理修复，获得第二修复图像，所述第二修 复图像为对所述第一修复图像进行纹理修复的图像；将所述第二修复图像输入至色彩修复模型，进行色彩修复，获得第三修复图像，所述第三修复图像为对所述第二修复图像进行色彩修复的图像。
根据本公开的一个或多个实施例，示例5提供了示例1的方法，所述第二特征序列的长度为所述第一特征序列长度的四倍。
根据本公开的一个或多个实施例，示例6提供了示例3的方法，所述第一特征序列的长度为所述第四特征序列长度的四倍。
根据本公开的一个或多个实施例，示例7提供了示例1的方法，所述将所述第三特征序列与所述第二特征序列融合，包括：将所述第三特征序列与所述第二特征序列相加，并进行编码与解码，获得融合后的特征序列。
根据本公开的一个或多个实施例，示例8提供了示例1的方法，所述结构修复模型通过以下方式训练得到：获取训练图像，所述训练图像中包括掩码图像；通过所述结构修复模型的多个分支对所述掩码图像进行下采样，得到第一训练特征序列和第二训练特征序列，将所述第一训练特征序列转化为与所述第二训练特征序列长度相同的第三训练特征序列，将所述第三训练特征序列与所述第二训练特征序列融合，根据融合后的训练特征序列对所述掩码图像进行结构修复，获得第一训练修复图像；根据第一训练修复图像和掩码前的训练图像更新所述结构修复模型的参数。
根据本公开的一个或多个实施例，示例9提供了示例2的方法，所述将所述第一修复图像输入至纹理修复模型和/或色彩修复模型，进行纹理修复和/或色彩修复，获得第二修复图像，包括：将所述第一修复图像输入至纹理修复模型和/或色彩修复模型，通过所述纹理修复模型和/或所述色彩修复模型对所述第一修复图像进行下采样和编码获得第五特征序列，将所述第五特征序列进行反卷积获得特征图，根据所述特征图对所述第一修复图像进行纹理修复和/或色彩修复，获得第二修复图像。
根据本公开的一个或多个实施例，示例10提供了一种图像修复装置，所述装置包括：获取模块，用于获取待修复图像；结构修复模块，用于将所述待修复图像输入至结构修复模型，通过所述结构修复模型的多个分支对所述待修复图像进行下采样，得到第一特征序列和第二特征序列，将所述第一特征序列转化为与所述第二特征序列长度相同的第三特征序列，将所述第三特征序列与所述第二特征序列融合，根据融合后的特征序列对所述待修复图像进行结构修复，获得第一修复图像，所述第一修复图像为对所述待修复图像的结构进行修复的图像。
根据本公开的一个或多个实施例，示例11提供了示例10的装置，所述装置还包括：纹理修复模块和/或色彩修复模块，用于将所述第一修复图像输入至纹理修复模型和/或色彩修复模型，进行纹理修复和/或色彩修复，获得第二修复图像，所述第二修复图像为对所述第一修复图像进行纹理修复和/或色彩修复的图像。
根据本公开的一个或多个实施例，示例12提供了示例10的装置，所述结构修复模块还用于：通过所述结构修复模型的多个分支对所述待修复图像进行下采样得到第四特征序列；所述结构修复模块具体用于：将所述第四特征序列上采样并与所述第一特征序列融合， 获得与所述第二特征序列长度相同的第三特征序列。
根据本公开的一个或多个实施例，示例13提供了示例10的装置，所述装置还包括：纹理修复模块，用于将所述第一修复图像输入至纹理修复模型，进行纹理修复，获得第二修复图像，所述第二修复图像为对所述第一修复图像进行纹理修复的图像；色彩修复模块，用于将所述第二修复图像输入至色彩修复模型，进行色彩修复，获得第三修复图像，所述第三修复图像为对所述第二修复图像进行色彩修复的图像。
根据本公开的一个或多个实施例，示例14提供了示例10的装置，所述第二特征序列的长度为所述第一特征序列长度的四倍。
根据本公开的一个或多个实施例，示例15提供了示例12的装置，所述第一特征序列的长度为所述第四特征序列长度的四倍。
根据本公开的一个或多个实施例，示例16提供了示例10的装置，所述结构修复模块具体用于：将所述第三特征序列与所述第二特征序列相加，并进行编码与解码，获得融合后的特征序列。
根据本公开的一个或多个实施例，示例17提供了示例10的装置，所述结构修复模型通过以下方式训练得到：获取训练图像，所述训练图像中包括掩码图像；通过所述结构修复模型的多个分支对所述掩码图像进行下采样，得到第一训练特征序列和第二训练特征序列，将所述第一训练特征序列转化为与所述第二训练特征序列长度相同的第三训练特征序列，将所述第三训练特征序列与所述第二训练特征序列融合，根据融合后的训练特征序列对所述掩码图像进行结构修复，获得第一训练修复图像；根据第一训练修复图像和掩码前的训练图像更新所述结构修复模型的参数。
根据本公开的一个或多个实施例，示例18提供了示例11的装置，所述结构修复模块具体用于：将所述第一修复图像输入至纹理修复模型和/或色彩修复模型，通过所述纹理修复模型和/或所述色彩修复模型对所述第一修复图像进行下采样和编码获得第五特征序列，将所述第五特征序列进行反卷积获得特征图，根据所述特征图对所述第一修复图像进行纹理修复和/或色彩修复，获得第二修复图像。
以上描述仅为本公开的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本公开中所涉及的公开范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离上述公开构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本公开中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。
此外，虽然采用特定次序描绘了各操作，但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下，多任务和并行处理可能是有利的。同样地，虽然在上面论述中包含了若干具体实现细节，但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的某些特征还可以组合地实现在单个实施例中。相反地，在单个实施例的上下文中描述的各种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。
尽管已经采用特定于结构特征和/或方法逻辑动作的语言描述了本主题，但是应当理解 所附权利要求书中所限定的主题未必局限于上面描述的特定特征或动作。相反，上面所描述的特定特征和动作仅仅是实现权利要求书的示例形式。关于上述实施例中的装置，其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述，此处将不做详细阐述说明。

Claims (13)

  1. 一种图像修复方法，其特征在于，所述方法包括：
    获取待修复图像；
    将所述待修复图像输入至结构修复模型，通过所述结构修复模型的多个分支对所述待修复图像进行下采样，得到第一特征序列和第二特征序列，将所述第一特征序列转化为与所述第二特征序列长度相同的第三特征序列，将所述第三特征序列与所述第二特征序列融合，根据融合后的特征序列对所述待修复图像进行结构修复，获得第一修复图像，所述第一修复图像为对所述待修复图像的结构进行修复的图像。
  2. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    将所述第一修复图像输入至纹理修复模型和/或色彩修复模型，进行纹理修复和/或色彩修复，获得第二修复图像，所述第二修复图像为对所述第一修复图像进行纹理修复和/或色彩修复的图像。
  3. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    通过所述结构修复模型的多个分支对所述待修复图像进行下采样得到第四特征序列；
    所述将所述第一特征序列转化为与所述第二特征序列长度相同的第三特征序列，包括：
    将所述第四特征序列上采样并与所述第一特征序列融合，获得与所述第二特征序列长度相同的第三特征序列。
  4. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    将所述第一修复图像输入至纹理修复模型，进行纹理修复，获得第二修复图像，所述第二修复图像为对所述第一修复图像进行纹理修复的图像；
    将所述第二修复图像输入至色彩修复模型，进行色彩修复，获得第三修复图像，所述第三修复图像为对所述第二修复图像进行色彩修复的图像。
  5. 根据权利要求1所述的方法，其特征在于，所述第二特征序列的长度为所述第一特征序列长度的四倍。
  6. 根据权利要求3所述的方法，其特征在于，所述第一特征序列的长度为所述第四特征序列长度的四倍。
  7. 根据权利要求1所述的方法，其特征在于，所述将所述第三特征序列与所述第二特征序列融合，包括：
    将所述第三特征序列与所述第二特征序列相加，并进行编码与解码，获得融合后的特征序列。
  8. 根据权利要求1所述的方法，其特征在于，所述结构修复模型通过以下方式训练得到：
    获取训练图像，所述训练图像中包括掩码图像；
    通过所述结构修复模型的多个分支对所述掩码图像进行下采样，得到第一训练特征序列和第二训练特征序列，将所述第一训练特征序列转化为与所述第二训练特征序列长度相同的第三训练特征序列，将所述第三训练特征序列与所述第二训练特征序列融合，根据融合后的训练特征序列对所述掩码图像进行结构修复，获得第一训练修复图像；
    根据第一训练修复图像和掩码前的训练图像更新所述结构修复模型的参数。
  9. 根据权利要求2所述的方法，其特征在于，所述将所述第一修复图像输入至纹理修复模型和/或色彩修复模型，进行纹理修复和/或色彩修复，获得第二修复图像，包括：
    将所述第一修复图像输入至纹理修复模型和/或色彩修复模型，通过所述纹理修复模型和/或所述色彩修复模型对所述第一修复图像进行下采样和编码获得第五特征序列，将所述第五特征序列进行反卷积获得特征图，根据所述特征图对所述第一修复图像进行纹理修复和/或色彩修复，获得第二修复图像。
  10. 一种图像修复装置，其特征在于，所述装置包括：
    获取模块，用于获取待修复图像；
    结构修复模块，用于将所述待修复图像输入至结构修复模型，通过所述结构修复模型的多个分支对所述待修复图像进行下采样，得到第一特征序列和第二特征序列，将所述第一特征序列转化为与所述第二特征序列长度相同的第三特征序列，将所述第三特征序列与所述第二特征序列融合，根据融合后的特征序列对所述待修复图像进行结构修复，获得第一修复图像，所述第一修复图像为对所述待修复图像的结构进行修复的图像。
  11. 一种设备，其特征在于，所述设备包括处理器和存储器；
    所述处理器用于执行所述存储器中存储的指令，以使得所述设备执行如权利要求1至9中任一项所述的方法。
  12. 一种计算机可读存储介质，其特征在于，包括指令，所述指令指示设备执行如权利要求1至9中任一项所述的方法。
  13. 一种计算机程序产品，其特征在于，当所述计算机程序产品在计算机上运行时，使得计算机执行如权利要求1至9中任一项所述的方法。
PCT/CN2023/077871 2022-03-21 2023-02-23 图像修复方法、装置、设备、介质及产品 WO2023179291A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210278165.X 2022-03-21
CN202210278165.XA CN114627023A (zh) 2022-03-21 2022-03-21 图像修复方法、装置、设备、介质及产品

Publications (1)

Publication Number Publication Date
WO2023179291A1 true WO2023179291A1 (zh) 2023-09-28

Family

ID=81904359

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/077871 WO2023179291A1 (zh) 2022-03-21 2023-02-23 图像修复方法、装置、设备、介质及产品

Country Status (2)

Country Link
CN (1) CN114627023A (zh)
WO (1) WO2023179291A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114627023A (zh) * 2022-03-21 2022-06-14 北京有竹居网络技术有限公司 图像修复方法、装置、设备、介质及产品

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010123070A (ja) * 2008-11-21 2010-06-03 Nara Institute Of Science & Technology 三次元形状復元装置
CN110766623A (zh) * 2019-10-12 2020-02-07 北京工业大学 一种基于深度学习的立体图像修复方法
CN111861945A (zh) * 2020-09-21 2020-10-30 浙江大学 一种文本引导的图像修复方法和系统
CN113362239A (zh) * 2021-05-31 2021-09-07 西南科技大学 一种基于特征交互的深度学习图像修复方法
CN113744142A (zh) * 2021-08-05 2021-12-03 南方科技大学 图像修复方法、电子设备及存储介质
CN114627023A (zh) * 2022-03-21 2022-06-14 北京有竹居网络技术有限公司 图像修复方法、装置、设备、介质及产品

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010123070A (ja) * 2008-11-21 2010-06-03 Nara Institute Of Science & Technology 三次元形状復元装置
CN110766623A (zh) * 2019-10-12 2020-02-07 北京工业大学 一种基于深度学习的立体图像修复方法
CN111861945A (zh) * 2020-09-21 2020-10-30 浙江大学 一种文本引导的图像修复方法和系统
CN113362239A (zh) * 2021-05-31 2021-09-07 西南科技大学 一种基于特征交互的深度学习图像修复方法
CN113744142A (zh) * 2021-08-05 2021-12-03 南方科技大学 图像修复方法、电子设备及存储介质
CN114627023A (zh) * 2022-03-21 2022-06-14 北京有竹居网络技术有限公司 图像修复方法、装置、设备、介质及产品

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
LI SIYUAN; LU LU; ZHANG ZHIQIANG; CHENG XIN; XU KEPENG; YU WENXIN; HE GANG; ZHOU JINJIA; YANG ZHUO: "Interactive Separation Network For Image Inpainting", 2020 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 25 October 2020 (2020-10-25), pages 1008 - 1012, XP033869790, DOI: 10.1109/ICIP40778.2020.9191263 *

Also Published As

Publication number Publication date
CN114627023A (zh) 2022-06-14

Similar Documents

Publication Publication Date Title
CN110413812B (zh) 神经网络模型的训练方法、装置、电子设备及存储介质
CN110298851B (zh) 人体分割神经网络的训练方法及设备
CN112330788A (zh) 图像处理方法、装置、可读介质及电子设备
WO2023179291A1 (zh) 图像修复方法、装置、设备、介质及产品
CN115578570A (zh) 图像处理方法、装置、可读介质及电子设备
CN112418249A (zh) 掩膜图像生成方法、装置、电子设备和计算机可读介质
CN114581336A (zh) 图像修复方法、装置、设备、介质及产品
WO2024016923A1 (zh) 特效图的生成方法、装置、设备及存储介质
CN116823984A (zh) 元素布局信息生成方法、装置、设备、介质和程序产品
CN115760607A (zh) 图像修复方法、装置、可读介质以及电子设备
CN114998149A (zh) 图像修复模型的训练方法及图像修复方法、装置及设备
CN112070888B (zh) 图像生成方法、装置、设备和计算机可读介质
CN111680754B (zh) 图像分类方法、装置、电子设备及计算机可读存储介质
CN114004229A (zh) 文本识别方法、装置、可读介质及电子设备
CN110033413B (zh) 客户端的图像处理方法、装置、设备、计算机可读介质
CN110633595B (zh) 一种利用双线性插值的目标检测方法和装置
CN112488947A (zh) 模型训练和图像处理方法、装置、设备和计算机可读介质
CN112215774B (zh) 模型训练和图像去雾方法、装置、设备和计算机可读介质
CN111738899B (zh) 用于生成水印的方法、装置、设备和计算机可读介质
CN115170674B (zh) 基于单张图像的相机主点标定方法、装置、设备和介质
CN112070163B (zh) 图像分割模型训练和图像分割方法、装置、设备
CN116630436B (zh) 相机外参修正方法、装置、电子设备和计算机可读介质
CN111275813B (zh) 数据处理方法、装置和电子设备
CN115345931B (zh) 物体姿态关键点信息生成方法、装置、电子设备和介质
CN114998148A (zh) 一种图像修复方法、装置及设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23773542

Country of ref document: EP

Kind code of ref document: A1