WO2023179310A1 - 图像修复方法、装置、设备、介质及产品 - Google Patents

图像修复方法、装置、设备、介质及产品 Download PDF

Info

Publication number
WO2023179310A1
WO2023179310A1 PCT/CN2023/078345 CN2023078345W WO2023179310A1 WO 2023179310 A1 WO2023179310 A1 WO 2023179310A1 CN 2023078345 W CN2023078345 W CN 2023078345W WO 2023179310 A1 WO2023179310 A1 WO 2023179310A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
repaired
feature
training
similar
Prior art date
Application number
PCT/CN2023/078345
Other languages
English (en)
French (fr)
Inventor
毛晓飞
黄灿
Original Assignee
北京有竹居网络技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京有竹居网络技术有限公司 filed Critical 北京有竹居网络技术有限公司
Publication of WO2023179310A1 publication Critical patent/WO2023179310A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction

Definitions

  • the present disclosure belongs to the field of image processing technology, and specifically relates to an image repair method, device, equipment, computer-readable storage medium and computer program product.
  • Image restoration refers to repairing the missing parts of the image. Specifically, image restoration refers to restoring unknown information in an image based on known information in the image.
  • a typical image repair solution is to determine the repair area in the image to be repaired, determine a reference area for the repair area, and then predict the pixels in the repair area through a neural network model based on the pixel values of the reference area of the image. value, thereby achieving image restoration.
  • this image repair technology may cause distortions such as ripples and distortions in the repair area, which does not meet users' requirements for image repair realism.
  • the purpose of this disclosure is to provide an image repair method, device, equipment, computer-readable storage medium and computer program product, which can perform high-fidelity repair on the image to be repaired and improve the user experience.
  • the present disclosure provides an image repair method, which method includes:
  • the extracted feature maps are serialized and encoded to obtain the first encoding result.
  • the second branch of the image repair model is used to encode the set of sub-feature maps divided by the feature map according to different scales, and the sub-feature map sets are encoded separately.
  • the coding results of the sub-feature maps in the feature map set are fully connected to obtain a second coding result, and the repaired image is obtained according to the first coding result and the second coding result.
  • the present disclosure provides an image repair device, characterized in that the device includes:
  • a determination module configured to determine multiple similar images of the image to be repaired from the image library, where the multiple similar images include at least a first similar image and a second similar image;
  • a fusion module configured to fuse the image to be repaired, the first similar image, and the second similar image, and input the fused image into an image repair model, and use the first branch of the image repair model to
  • the feature map extracted from the fused image is serialized and encoded to obtain the first encoding result, and the set of sub-feature maps divided by the feature map according to different scales are encoded respectively through the second branch of the image repair model,
  • the coding results of the sub-feature maps in the sub-feature map set are fully connected to obtain a second coding result, and the repaired image is obtained according to the first coding result and the second coding result.
  • the present disclosure provides an electronic device, including: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the first aspect of the present disclosure. Method steps.
  • the present disclosure provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processing device, the steps of the method described in the first aspect of the present disclosure are implemented.
  • the present disclosure provides a computer program product containing instructions that, when run on a device, cause the device to perform the steps of the method described in the first aspect.
  • the electronic device acquires the image to be repaired, and then determines multiple similar images of the image to be repaired from the image library, including a first similar image and a second similar image. melt The image to be repaired, the first similar image and the second similar image are combined, and the fused image is input into the image repair model.
  • the feature map extracted from the fused image is serialized and encoded through the first branch of the image repair model to obtain a first encoding result.
  • the sub-feature map sets whose feature maps are divided according to different scales are separately encoded, and the encoding results of the sub-feature maps in the sub-feature map set are fully connected to obtain the second encoding result.
  • a repaired image is obtained according to the first encoding result and the second encoding result.
  • image repair can not only predict the area to be repaired based on the known area (reference area) of the image to be repaired, but also use similar images as known areas to predict the area to be repaired of the image to be repaired. In this way, reliable data for predicting the image of the area to be repaired is increased, thereby effectively improving the effect of image repair.
  • the two branches of the image repair model can repair the image to be repaired based on feature maps of different scales, making the image repair more accurate, improving the authenticity of the image repair, and improving the user experience.
  • Figure 1 is a schematic flow chart of an image repair method provided by an embodiment of the present disclosure
  • Figure 2 is a schematic diagram of image restoration provided by an embodiment of the present disclosure
  • Figure 3 is a schematic flowchart of obtaining similar images through a neural network model according to an embodiment of the present disclosure
  • Figure 4 is a schematic diagram of image repair using an image repair model provided by an embodiment of the present disclosure
  • Figure 5 is a schematic flow chart of another image repair method provided by an embodiment of the present disclosure.
  • Figure 6 is a schematic structural diagram of an image repair device provided by an embodiment of the present disclosure.
  • FIG. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • first and second in the embodiments of the present disclosure are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. thus, Features defined as “first” and “second” may explicitly or implicitly include one or more of these features.
  • Image processing technology generally refers to the processing of digital images, and specifically refers to the technology of analyzing and processing digital images through computers. Based on image processing technology, various types of processing can be performed on images, such as repairing images with missing parts, that is, image repair technology.
  • Image repair technology refers to determining the repair area and reference area in the image to be repaired, and repairing the repair area based on the reference area.
  • the image to be repaired may be an image with part of the pattern missing, or may be an image whose clarity does not meet the user's needs.
  • the image repair technology usually used is to predict the pixel values of the repair area through a neural network model based on the pixel values of the reference area in the image to be repaired, thereby achieving image repair.
  • This image repair method only repairs the image from the perspective of pixels, which may cause distortions such as ripples and distortions in the repaired area.
  • this method cannot accurately determine the missing content and does not meet the user's requirements for the authenticity of the image repair.
  • Electronic equipment refers to equipment with data processing capabilities, such as servers or terminals.
  • terminals include but are not limited to smartphones, tablets, laptops, personal digital assistants (personal digital assistants, PDAs) or smart wearable devices.
  • the server may be a cloud server, such as a central server in a central cloud computing cluster, or an edge server in an edge cloud computing cluster.
  • the server can also be a server in a local data center.
  • a local data center refers to a data center directly controlled by the user.
  • the electronic device acquires the image to be repaired, then determines multiple similar images of the image to be repaired from the image library, and fuses the image to be repaired, the first similar image, and the second similar image.
  • the fused image is input into the image repair model, and the feature map extracted from the fused image is serialized and encoded through the first branch of the image repair model to obtain the first encoding result.
  • the feature map is segmented according to different scales into a set of sub-feature maps. Encoding is performed separately, and the encoding results of the sub-feature maps in the sub-feature map set are fully connected to obtain the second encoding result.
  • a repaired image is obtained according to the first encoding result and the second encoding result.
  • image repair can not only predict the area to be repaired based on the known area (reference area) of the image to be repaired, but also use similar images as known areas to predict the area to be repaired of the image to be repaired. In this way, reliable data for predicting the image of the area to be repaired is increased, thereby effectively improving the effect of image repair.
  • the two branches of the image repair model can repair the image to be repaired based on feature maps of different scales, making the image repair more accurate, improving the authenticity of the image repair, and improving the user experience.
  • the following takes an electronic device as a terminal as an example, as shown in Figure 1, to introduce the image repair method provided by the embodiment of the present disclosure.
  • the method includes the following steps:
  • S102 The terminal obtains the image to be repaired.
  • the image to be repaired can be an image with parts missing, or an image whose clarity does not meet the user's requirements.
  • Figure 2 is a schematic diagram of image restoration provided by an embodiment of the present disclosure.
  • the image to be repaired is an image with partial missing parts as an example.
  • a in Figure 2 there is a blank (missing) part in the upper left part of the figure.
  • the terminal can obtain the image to be repaired through multiple ways.
  • the terminal can capture the image to be repaired through a camera. For example, if the user has a paper photo that is partially missing, the user can capture the paper photo with the camera and obtain the image to be repaired in digital format.
  • the terminal can obtain the image to be repaired through the user's input operation on the corresponding page. For example, the user saves an image to be repaired in digital format in another terminal, so the image to be repaired can be transmitted to the terminal through any transmission method, so that the terminal can obtain the image to be repaired.
  • S104 The terminal determines multiple similar images of the image to be repaired from the image library.
  • the terminal can search and obtain multiple similar images similar to the image to be repaired in the image library based on the image to be repaired.
  • the multiple similar images include a first similar image and a second similar image.
  • the multiple similar images and the image to be repaired are images of the same subject at different times or different angles.
  • the image to be repaired is a photographed image of a certain building.
  • the terminal can obtain other similar images whose main image is the building in the image library.
  • the similar image and the image to be repaired may only differ in shooting angle, shooting distance, light, etc., and therefore can be used as reference data for repairing the image to be repaired.
  • the multiple similar images and the image to be repaired are images taken at the same location.
  • the image A to be repaired is an image of a building
  • the multiple similar images B and C determined by it can be images of the building at different times and under light intensities.
  • the terminal can determine multiple similar images of the image to be repaired from the image library in various ways.
  • the parameters of the image include the shooting location of the image.
  • the terminal can obtain the image taken at the same location from the image library based on the shooting location, and then determine whether the image taken at the same location is a similar image by comparing the similarity between the image taken at the same location and the image to be repaired.
  • the image to be restored is an image of a famous building taken in front of the famous building. Therefore, other images that were also taken at the location can be obtained in the image library, and then compared to see if they are similar to the image to be repaired. If they are similar, they can be determined as similar images to the image to be repaired.
  • the parameters of the image include the shooting time of the image.
  • the terminal can obtain images taken at the same time from the image library based on the shooting time, and then determine whether the images taken at the same time are similar images by comparing the similarity between the images taken at the same time and the image to be repaired.
  • the image to be repaired is an image of the sky taken during a meteor shower. Therefore, you can obtain other images of the sky that were also taken at that moment in the picture library, and then compare whether they are similar to the image to be repaired. If they are similar, they can be determined to be similar to the image to be repaired.
  • the terminal can also determine multiple similar images of the image to be repaired from the image library through the neural network model.
  • Figure 3 is a schematic flowchart of obtaining similar images through a neural network model according to an embodiment of the present disclosure. As shown in Figure 3, the method of determining multiple similar images of the image to be repaired from the image library through a neural network model includes the following steps:
  • S302 The terminal compares the features of the image to be repaired to the model and obtains the search features of the image to be repaired.
  • the feature comparison model is used to determine search features from the image to be repaired for searching in the image library.
  • the search features are features of the image to be repaired that are needed to find similar images.
  • the feature comparison model is a trained neural network model. By searching the search features output by the feature comparison model in the image feature database, similar features corresponding to similar images that are similar to the image to be repaired can be obtained, thereby determining similar images.
  • the feature comparison model can use convolutional neural networks (CNN) to 4 times downsample the image to be repaired (subsampled) to obtain a feature map that is 1/16 of the original image size. After flattening the feature map into a one-dimensional sequence, it is sent to the N-layer encoder for encoding (encoder) to obtain the encoded one-dimensional sequence, and then the encoded one-dimensional sequence is converted into two-dimensional features. , obtain the search features of the image to be repaired.
  • CNN convolutional neural networks
  • S304 The terminal obtains multiple similar features similar to the search feature from the image feature database according to the search feature.
  • the image feature library is a feature library that has a one-to-one correspondence with the image library.
  • the images in the image library have a one-to-one correspondence with the features in the image feature library.
  • the feature comparison model is a feature comparison model obtained by the neural network model through training a large number of similar images. Through the search features output by the feature comparison model, multiple similar features similar to the search features can be obtained in the image feature library.
  • S306 The terminal obtains multiple similar images of the image to be repaired from the image library based on multiple similar features.
  • the terminal can determine multiple similar images of the image to be repaired from the image database based on similar features.
  • the terminal can determine multiple similar images of the image to be repaired from the image library through the neural network model, thereby obtaining more reference data for image repair.
  • image restoration only uses the reference area in the image as known data to determine the unknown data of the repair area.
  • similar images of the image to be repaired are determined, and similar images are also used as known data to determine unknown data in the repair area, which effectively increases the amount of known data, so a better repair effect can be obtained.
  • the terminal fuses the image to be repaired, the first similar image, and the second similar image, inputs the fused image into the image repair model, and obtains the repaired image through the image repair model.
  • the terminal fuses (concat) the image to be repaired and multiple similar images to obtain a fused image.
  • the plurality of similar images may include a first similar image and a second similar image.
  • Figure 4 is a schematic diagram of image repair using an image repair model provided by an embodiment of the present disclosure. picture. In this embodiment, multiple similar images are used as the first similar image and the second similar image as an example for introduction, as shown in FIG. 4 .
  • the terminal uses the image repair model to first downsample the fused image 8 times through a 4-layer convolutional neural network (Convolutional Neural Networks, CNN), and then downsample it 2 times through the downsample layer. , obtain a feature map with a size of 1/256 of the original image.
  • a 4-layer convolutional neural network Convolutional Neural Networks, CNN
  • the image repair model includes two branches.
  • the first branch flattens the feature map to obtain a one-dimensional sequence, and then inputs the one-dimensional sequence into the N-layer encoder for encoding to obtain the first encoding result.
  • the second branch performs segmentation on the feature map at different scales. For example, windows of 1/4, 1/16, and 1/64 of the feature map are used to segment the feature map and generate a set of sub-feature maps. Then the feature maps in the sub-feature set are flattened to obtain corresponding one-dimensional sequences.
  • These one-dimensional sequences are input into the N-layer encoder for encoding, and then the sequences of different scales output by the encoder are mapped back to the same length through the fully connected (FC) layer to obtain the second encoding result.
  • the first encoding result and the second encoding result are fused through the FC layer to obtain the final output, thereby realizing the repair of the image to be repaired.
  • the image repair model has two branches, and the feature maps are segmented at different scales in the second branch, the image to be repaired can be repaired based on multiple feature maps of different scales, making the image repair more accurate and improving the image repair The authenticity improves the user experience.
  • the image repair model can be obtained by training images including multiple similar images.
  • the multiple similar images include a mask image, and the mask image is an image obtained by masking the similar images.
  • the training process of the image restoration model specifically includes:
  • the terminal obtains training feature maps from training images including multiple similar images. Specifically, the terminal fuses multiple similar images, and then the fused image is first down-sampled 8 times through a 4-layer convolutional neural network, and then down-sampled 2 times through the down-sampling layer to obtain the original image. Training feature map with image size 1/256.
  • the one-dimensional sequence is obtained by flattening the training feature map through the first branch of the image repair model, and then the one-dimensional sequence is input to the N-layer encoder for encoding, and the first training encoding result is obtained.
  • the second branch of the image repair model adopts the training feature map 1/4, 1/16 and 1/64 windows are used to segment the training feature map at different scales to generate a set of training sub-feature maps. Then, the training feature maps in the training sub-feature set are flattened to obtain corresponding one-dimensional sequences.
  • These one-dimensional sequences are respectively input into the N-layer encoder for encoding, and the sequences of different scales output by the encoder are mapped back to the same length through the FC layer to obtain the second training encoding result.
  • the first training coding result and the second training coding result are fused through the FC layer to obtain the final output, thereby realizing the repair of the mask image.
  • the terminal can compare the repaired mask image with the training image before the mask, and update the parameters of the image repair model.
  • the terminal that executes the image repair method in this embodiment and the terminal that performs model training can be the same terminal, or they can be different terminals.
  • a terminal can transmit its trained image repair model to multiple other terminals, so that multiple other terminals can directly use the image repair model to implement the image repair method in the present disclosure.
  • the present disclosure provides an image repair method.
  • the terminal obtains the image to be repaired, and then determines multiple similar images of the image to be repaired from the image library. Fuse the image to be repaired and multiple similar images, and input the fused image into the image repair model.
  • the feature map extracted from the fused image is serialized and encoded through the first branch of the image repair model to obtain a first encoding result.
  • the sub-feature map sets whose feature maps are divided according to different scales are separately encoded, and the encoding results of the sub-feature maps in the sub-feature map set are fully connected to obtain the second encoding result.
  • a repaired image is obtained according to the first encoding result and the second encoding result.
  • the terminal can not only predict the area to be repaired based on the known area (reference area) of the image to be repaired, but also use similar images as known areas to predict the area to be repaired of the image to be repaired.
  • This increases reliable data for predicting the image of the area to be repaired, thereby effectively improving the effect of image repair.
  • the two branches of the image repair model can repair the image to be repaired based on feature maps of different scales, making the image repair more accurate, improving the authenticity of the image repair, and improving the user experience.
  • the subject of the image to be repaired is a famous building, as shown in Figure 2, where the upper left part of the image is missing.
  • Figure 5 is provided by an embodiment of the present disclosure. Flowchart of another image restoration method. As shown in Figure 5, the repair of the image to be repaired includes the following steps:
  • S502 The terminal obtains the image to be repaired.
  • the image to be repaired in this embodiment is shown in Figure 2.
  • the terminal can convert the paper image to be repaired into a digital image to be repaired by taking a photo, or it can directly obtain the digital image to be repaired.
  • S504 The terminal inputs the features of the image to be repaired into the model and obtains the search features of the image to be repaired.
  • the feature comparison model can use a convolutional neural network to downsample the image to be repaired 4 times to obtain a feature map that is 1/16 of the original image size. Then, the feature map is flattened and converted into a one-dimensional sequence, and then sent to the N-layer encoder for encoding to obtain the encoded one-dimensional sequence. Convert the encoded one-dimensional sequence into two-dimensional features to obtain the search features of the image to be repaired.
  • S506 The terminal obtains similar features similar to the search feature from the image feature library based on the search feature.
  • the terminal obtains multiple similar features that are similar to the search feature from an image feature database that has a one-to-one correspondence with the image database based on the search feature.
  • S508 The terminal obtains multiple similar images of the image to be repaired from the image library based on multiple similar features.
  • the terminal can determine similar images corresponding to multiple similar features in the image database based on multiple similar features, so that multiple images to be repaired can be obtained. similar images.
  • S510 The terminal fuses the image to be repaired and multiple similar images, inputs the fused image into the image repair model, and obtains the repaired image through the image repair model.
  • the terminal fuses the image to be repaired and multiple similar images to obtain a fused image. Then, through the image repair model, the fused image is first down-sampled by 8 times through a 4-layer convolutional neural network, and then down-sampled by 2 times through the down-sampling layer to obtain features of 1/256 of the original image size. picture.
  • the terminal obtains a one-dimensional sequence by flattening the feature map through the first branch of the image repair model, and then inputs the one-dimensional sequence into the N-layer encoder for encoding, and obtains the first encoding result.
  • the terminal uses the second branch of the image repair model to segment the feature map at different scales using windows of 1/4, 1/16, and 1/64 of the feature map to generate a set of sub-feature maps. Then, the feature maps in the sub-feature set are flattened to obtain corresponding one-dimensional sequences. These one-dimensional sequences are respectively input to the N-layer encoder for encoding, and the sequences of different scales output by the encoder are mapped back to the same length through the fully connected layer to obtain the second encoding result.
  • the separate processing of the feature map through the first branch and the second branch of the model can be performed simultaneously.
  • the terminal fuses the first encoding result and the second encoding result through the FC layer to obtain the final output, thereby realizing the repair of the image to be repaired.
  • the image to be repaired can be shown as A in Figure 2
  • similar images of the image to be repaired can be shown as B and C in Figure 2
  • B is the first similar image
  • C is The second similar image
  • D in Figure 2 is the repaired image. Since the reference data in the repair process not only includes the known parts in A, but also includes similar images B and C, there is more reference data, which enables more accurate repair.
  • Figure 6 is a schematic diagram of an image repair device according to an exemplary disclosed embodiment. As shown in Figure 6, the image repair device 600 includes:
  • Acquisition module 602 used to obtain the image to be repaired
  • Determining module 604 configured to determine multiple similar images of the image to be repaired from the image library, where the multiple similar images include at least a first similar image and a second similar image;
  • the fusion module 606 is used to fuse the image to be repaired, the first similar image, and the second similar image, and input the fused image into an image repair model, and use the first branch of the image repair model to
  • the feature map extracted from the fused image is serialized and encoded to obtain a first encoding result, and the set of sub-feature maps divided by the feature map according to different scales are separately encoded through the second branch of the image repair model. , and fully connect the coding results of the sub-feature maps in the sub-feature map set to obtain a second coding result, and obtain a repaired image according to the first coding result and the second coding result.
  • the image repair model is obtained by training on training images
  • the training images include multiple similar images
  • the multiple similar images include a mask image
  • the mask image is obtained by masking the similar images. code obtained.
  • the image repair model is trained in the following manner:
  • the second branch of the image repair model encodes the training sub-feature map sets divided into different scales by using the second branch of the image repair model, and fully connects the coding results of the training sub-feature maps in the training sub-feature set. , obtain the second training encoding result;
  • the image repair model parameters are updated according to the repaired mask image and the training image before the mask.
  • the determining module 604 can be used to:
  • multiple similar images of the image to be repaired are obtained from an image database, and the images in the image database correspond to the features in the image feature database one-to-one.
  • the determining module 604 is specifically used to:
  • the image to be repaired is input into a feature comparison model, the image to be repaired is downsampled through the feature comparison model to obtain a feature map of the image to be repaired, and the feature map is encoded to obtain the Search features for images to be repaired.
  • the multiple similar images and the image to be repaired are images of the same subject at different times or different angles.
  • the multiple similar images and the image to be repaired are images taken at the same location.
  • Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (tablet computers), etc. mobile terminals such as computers), PMP (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs, desktop computers, etc.
  • the electronic device shown in FIG. 7 is only an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.
  • the electronic device 700 may include a processing device (eg, central processing unit, graphics processor, etc.) 701 that may be loaded into a random access device according to a program stored in a read-only memory (ROM) 702 or from a storage device 708 .
  • the program in the memory (RAM) 703 executes various appropriate actions and processes.
  • various programs and data required for the operation of the electronic device 700 are also stored.
  • the processing device 701, the ROM 702 and the RAM 703 are connected to each other via a bus 704.
  • An input/output (I/O) interface 705 is also connected to bus 704.
  • the following devices may be connected to the I/O interface 705: input devices 706 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 707 such as a computer; a storage device 708 including a magnetic tape, a hard disk, etc.; and a communication device 709. Communication device 709 may allow electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data.
  • FIG. 7 illustrates an electronic device 700 having various means, it should be understood that implementation or availability of all illustrated means is not required. More or fewer means may alternatively be implemented or provided.
  • embodiments of the present disclosure include a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart.
  • the computer program may be downloaded and installed from the network via communication device 709, or from storage device 708, or from ROM 702.
  • the processing device 701 the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
  • the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
  • the computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer Computer disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage components, magnetic storage devices, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above.
  • a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device .
  • Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wire, optical cable, RF (radio frequency), etc., or any suitable combination of the above.
  • the client and server can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can communicate with digital data in any form or medium.
  • Communications e.g., communications network
  • communications networks include local area networks (“LAN”), wide area networks (“WAN”), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or developed in the future network of.
  • the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; it may also exist independently without being assembled into the electronic device.
  • the computer-readable medium carries one or more programs.
  • the electronic device When the one or more programs are executed by the electronic device, the electronic device: performs text detection on the image, obtains the text area in the image, and the text
  • the region includes a plurality of text lines; a graph network model is constructed according to the text region, and each text behavior in the text region is a node of the graph network model; the nodes in the graph network model are processed through a node classification model Classify, and classify edges between nodes in the graph network model through an edge classification model; obtain at least one key value in the image based on the classification results of the nodes and the classification results of the edges. right.
  • Programs for executing the present disclosure may be written in one or more programming languages, or a combination thereof
  • the above programming languages include but are not limited to object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as "C" language or similar programming languages .
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider). connected via the Internet).
  • LAN local area network
  • WAN wide area network
  • Internet service provider such as an Internet service provider
  • each block in the flowchart or block diagram may represent a module, segment, or portion of code that contains one or more logic functions that implement the specified executable instructions.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved.
  • each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration can be implemented by special purpose hardware-based systems that perform the specified functions or operations. , or can be implemented using a combination of specialized hardware and computer instructions.
  • the modules involved in the embodiments of the present disclosure can be implemented in software or hardware. Among them, the name of the module does not constitute a limitation on the module itself under certain circumstances.
  • FPGAs Field Programmable Gate Arrays
  • ASICs Application Specific Integrated Circuits
  • ASSPs Application Specific Standard Products
  • SOCs Systems on Chips
  • CPLD Complex Programmable Logical device
  • a machine-readable medium may be a tangible medium that may contain or be stored for use by or in connection with an instruction execution system, apparatus, or device. programs used in conjunction with the equipment.
  • the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any suitable combination of the foregoing.
  • machine-readable storage media would include one or more wire-based electrical connections, laptop disks, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • RAM random access memory
  • ROM read only memory
  • EPROM or flash memory erasable programmable read only memory
  • CD-ROM portable compact disk read-only memory
  • magnetic storage device or any suitable combination of the above.
  • Example 1 provides an image repair method.
  • the method includes: acquiring an image to be repaired; determining multiple similar images of the image to be repaired from an image library, and the multiple similar images of the image to be repaired are determined.
  • a similar image includes at least a first similar image and a second similar image; fuse the image to be repaired with the first similar image and the second similar image, and input the fused image into the image repair model, through the
  • the first branch of the image repair model serializes and encodes the feature map extracted from the fused image to obtain a first encoding result, and the second branch of the image repair model sorts the feature map according to different scales.
  • the divided sub-feature map sets are encoded respectively, and the encoding results of the sub-feature maps in the sub-feature map set are fully connected to obtain a second encoding result. According to the first encoding result and the second encoding result Get the repaired image.
  • Example 2 provides the method of Example 1.
  • the image repair model is trained through training images.
  • the training images include multiple similar images, and the multiple similar images include masks. code image, and the mask image is obtained by masking the similar image.
  • Example 3 provides the method of Example 2.
  • the image repair model is trained in the following manner: extracting a training feature map from the training image; One branch serializes and encodes the training feature map to obtain the first training encoding result; the second branch of the image repair model encodes the training sub-feature map sets divided by the training feature map according to different scales respectively. , and fully connect the coding results of the training sub-feature maps in the training sub-feature set to obtain the second training coding result; repair the mask according to the first training coding result and the second training coding result image; update the image repair model parameters according to the repaired mask image and the training image before the mask.
  • Example 5 provides the method of Example 4, and inputting the image to be repaired into a feature comparison model to obtain the search features of the image to be repaired includes: The repaired image inputs a feature comparison model, uses the feature comparison model to downsample the image to be repaired to obtain the feature map of the image to be repaired, and encodes the feature map to obtain the feature map of the image to be repaired. Search features.
  • Example 6 provides the method described in any one of Examples 1 to 5.
  • the multiple similar images and the image to be repaired are the same subject at different times or different angles. Image.
  • Example 7 provides the method described in any one of Examples 1 to 5.
  • the multiple similar images and the image to be repaired are images taken at the same location.
  • Example 8 provides an image repair device.
  • the device includes: an acquisition module, configured to acquire an image to be repaired; and a determination module, configured to determine the image to be repaired from an image library.
  • a plurality of similar images of an image the plurality of similar images including at least a first similar image and a second similar image; a fusion module for fusing the image to be repaired with the first similar image and the second similar image , and input the fused image into the image repair model, serialize and encode the feature map extracted from the fused image through the first branch of the image repair model, and obtain the first encoding result, through the
  • the second branch of the image repair model separately encodes the set of sub-feature maps divided by the feature map according to different scales, and fully connects the coding results of the sub-feature maps in the set of sub-feature maps to obtain the second coding result. , obtaining the repaired image according to the first encoding result and the second encoding result.
  • Example 9 provides the device of Example 8.
  • the image repair model is trained through training images.
  • the training images include multiple similar images, and the multiple similar images include masks. code image, and the mask image is obtained by masking the similar image.
  • Example 10 provides the device of Example 9.
  • the image repair model is trained in the following manner: extracting a training feature map from the training image; One branch serializes and encodes the training feature map to obtain the first training encoding result; the second branch of the image repair model encodes the training sub-feature map sets divided by the training feature map according to different scales respectively. , and fully connect the coding results of the training sub-feature maps in the training sub-feature set to obtain the second training coding result; repair the mask according to the first training coding result and the second training coding result image; update the image repair model parameters according to the repaired mask image and the training image before the mask.
  • Example 11 provides the device of Example 8, and the determination module is specifically configured to: input the feature comparison model of the image to be repaired, and obtain the search features of the image to be repaired; According to the search features, multiple similar features similar to the search features are obtained from the image feature library; according to the multiple similar features, multiple similar images of the image to be repaired are obtained from the image library, and the image The images in the library correspond to the features in the image feature library one-to-one.
  • Example 12 provides the device of Example 11, and the determination module is specifically configured to: input the image to be repaired into a feature comparison model, and use the feature comparison model to compare the The image to be repaired is downsampled to obtain the feature map of the image to be repaired, and the feature map is encoded to obtain the search features of the image to be repaired.
  • Example 13 provides the device described in any one of Examples 8 to 12, and the multiple similar images and the image to be repaired are of the same subject at different times or different angles. Image.
  • Example 14 provides the device described in any one of Examples 8 to 12, and the multiple similar images and the image to be repaired are images taken at the same location.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Processing (AREA)

Abstract

本公开提供了一种图像修复方法、装置、设备、介质以及产品，该方法包括：获取待修复图像，然后从图像库中确定待修复图像的多张相似图像，其中包括第一相似图像和第二相似图像。融合待修复图像和第一相似图像与第二相似图像，并将融合后的图像输入图像修复模型。通过图像修复模型的第一分支对从融合后的图像提取的特征图进行序列化和编码，得到第一编码结果。通过图像修复模型的第二分支对特征图按照不同尺度分割的子特征图集合进行分别编码，并将子特征图集合中的子特征图的编码结果进行全连接，获得第二编码结果。根据第一编码结果和第二编码结果获得修复后的图像。如此，提高了图像修复的真实性，提高了用户的使用体验。

Description

图像修复方法、装置、设备、介质及产品
本公开要求于2022年3月21日提交中国国家知识产权局、申请号为202210278129.3、发明名称为“图像修复方法、装置、设备、介质及产品”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本公开属于图像处理技术领域，具体涉及一种图像修复方法、装置、设备、计算机可读存储介质以及计算机程序产品。
背景技术
随着图像处理技术的不断成熟，用户对于通过图像处理技术进行图像修复的需求逐渐增强。图像修复是指对于图像中的缺失部分进行修复。具体地，图像修复是指基于图像中的已知信息去还原图像中的未知信息。
一种典型的图像修复方案是，针对待修复图像，确定待修复图像中的修复区域，为该修复区域确定参考区域，然后基于图像的参考区域的像素值，通过神经网络模型预测修复区域的像素值，从而实现图像修复。但是这种图像修复技术可能导致修复区域存在波纹、扭曲等失真情况，不满足用户对于图像修复真实度的要求。
如何提高图像修复的真实性成为亟需解决的问题。
发明内容
本公开的目的在于：提供了一种图像修复方法、装置、设备、计算机可读存储介质以及计算机程序产品，能够对于待修复图像进行高真实度的修复，提高用户的使用体验。
第一方面，本公开提供了一种图像修复方法，所述方法包括：
获取待修复图像；
从图像库中确定所述待修复图像的多张相似图像，所述多张相似图像至少包括第一相似图像和第二相似图像；
融合所述待修复图像和所述第一相似图像、所述第二相似图像，并将融合后的图像输入图像修复模型，通过所述图像修复模型的第一分支对从所述融合后的图像中提取的特征图进行序列化和编码，得到第一编码结果，通过所述图像修复模型的第二分支对所述特征图按照不同尺度分割的子特征图集合分别进行编码，并将所述子特征图集合中的子特征图的编码结果进行全连接，得到第二编码结果，根据所述第一编码结果和所述第二编码结果获得修复后的图像。
第二方面，本公开提供了一种图像修复装置，其特征在于，所述装置包括：
获取模块，用于获取待修复图像；
确定模块，用于从图像库中确定所述待修复图像的多张相似图像，所述多张相似图像至少包括第一相似图像和第二相似图像；
融合模块，用于融合所述待修复图像和所述第一相似图像、所述第二相似图像，并将融合后的图像输入图像修复模型，通过所述图像修复模型的第一分支对从所述融合后的图像中提取的特征图进行序列化和编码，得到第一编码结果，通过所述图像修复模型的第二分支对所述特征图按照不同尺度分割的子特征图集合分别进行编码，并将所述子特征图集合中的子特征图的编码结果进行全连接，得到第二编码结果，根据所述第一编码结果和所述第二编码结果获得修复后的图像。
第三方面，本公开提供一种电子设备，包括：存储装置，其上存储有计算机程序；处理装置，用于执行所述存储装置中的所述计算机程序，以实现本公开第一方面所述方法的步骤。
第四方面，本公开提供一种计算机可读介质，其上存储有计算机程序，该程序被处理装置执行时实现本公开第一方面所述方法的步骤。
第五方面，本公开提供了一种包含指令的计算机程序产品，当其在设备上运行时，使得设备执行上述第一方面所述方法的步骤。
从以上技术方案可以看出，本公开至少具有如下优点：
在上述技术方案中，电子设备获取待修复图像，然后从图像库中确定待修复图像的多张相似图像，其中包括第一相似图像和第二相似图像。融 合待修复图像、第一相似图像与第二相似图像，并将融合后的图像输入图像修复模型。通过图像修复模型的第一分支对从融合后的图像提取的特征图进行序列化和编码，得到第一编码结果。通过图像修复模型的第二分支对特征图按照不同尺度分割的子特征图集合进行分别编码，并将子特征图集合中的子特征图的编码结果进行全连接，获得第二编码结果。根据第一编码结果和第二编码结果获得修复后的图像。
一方面，对于图像的修复不仅可以基于待修复图像的已知区域(参考区域)对待修复区域进行预测，而且可以将相似图像作为已知区域对待修复图像的待修复区域进行预测。如此，增加了对待修复区域图像进行预测的可靠数据，从而有效的提高了图像修复的效果。另一方面，图像修复模型的两个分支可以基于不同尺度的特征图对待修复图像进行修复，使图像修复更加准确，提高了图像修复的真实性，提高了用户的使用体验。
本公开的其他特征和优点将在随后的具体实施方式部分予以详细说明。
附图说明
附图用来提供对本发明的进一步理解，并且构成说明书的一部分，与本发明实施例一起用于解释本发明，并不构成对本发明的限制。在附图中：
图1为本公开实施例提供的一种图像修复方法的流程示意图；
图2为本公开实施例提供的一种图像修复的示意图；
图3为本公开实施例提供的一种通过神经网络模型获取相似图像的流程示意图；
图4为本公开实施例提供的一种通过图像修复模型进行图像修复的示意图；
图5为本公开实施例提供的另一种图像修复方法的流程示意图；
图6为本公开实施例提供的一种图像修复装置的结构示意图；
图7为本公开实施例提供的一种电子设备的结构示意图。
具体实施方式
本公开实施例中的术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此， 限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。
首先对本公开实施例中所涉及到的一些技术术语进行介绍。
图像处理(image processing)技术一般是对数字图像进行处理，具体是指通过计算机对于数字图像进行分析处理的技术。基于图像处理技术，可以对图像进行多种类型的处理，例如对存在缺失部分的图像进行修复，即图像修复技术。
图像修复技术是指针对于待修复图像，确定待修复图像中的修复区域以及参考区域，基于参考区域对修复区域进行修复。其中，待修复图像可以为存在部分图案缺失的图像，也可以为清晰度不满足用户需求的图像。
通常情况下所采用的图像修复技术是根据待修复图像中参考区域的像素值，通过神经网络模型预测修复区域的像素值，从而实现图像修复。这种图像修复方法仅从像素的角度对图像进行修复，可能导致修复后区域存在波纹、扭曲等失真的情况。并且，当待修复图像中存在较大部分图案缺失时，这种方法无法准确确定出缺失内容，不满足用户对于图像修复真实度的要求。
有鉴于此，本公开提供一种图像修复方法，该方法应用于电子设备。电子设备是指具有数据处理能力的设备，例如可以是服务器，或者是终端。其中，终端包括但不限于智能手机、平板电脑、笔记本电脑、个人数字助理(personal digital assistant，PDA)或者智能穿戴设备等。服务器可以是云服务器，例如是中心云计算集群中的中心服务器，或者是边缘云计算集群中的边缘服务器。当然，服务器也可以是本地数据中心中的服务器。本地数据中心是指用户直接控制的数据中心。
具体地，电子设备获取待修复图像，然后从图像库中确定待修复图像的多张相似图像，融合待修复图像、第一相似图像与第二相似图像。将融合后的图像输入图像修复模型，通过图像修复模型的第一分支对从融合后的图像中提取的特征图进行序列化和编码，获得第一编码结果。然后，通过该图像修复模型的第二分支对特征图按照不同尺度分割的子特征图集合 分别进行编码，并将子特征图集合中的子特征图的编码结果进行全连接，获得第二编码结果。根据第一编码结果和第二编码结果获得修复后的图像。
一方面，对于图像的修复不仅可以基于待修复图像的已知区域(参考区域)对待修复区域进行预测，而且可以将相似图像作为已知区域对待修复图像的待修复区域进行预测。如此，增加了对待修复区域图像进行预测的可靠数据，从而有效的提高了图像修复的效果。另一方面，图像修复模型的两个分支可以基于不同尺度的特征图对待修复图像进行修复，使图像修复更加准确，提高了图像修复的真实性，提高了用户的使用体验。
为了使得本公开的技术方案更加清楚、易于理解，下面从电子设备为终端为例，如图1所示，对本公开实施例提供的图像修复方法进行介绍，该方法包括以下步骤：
S102：终端获取待修复图像。
待修复图像可以为存在部分缺失的图像，也可以为清晰度不满足用户要求的图像。参见图2，图2为本公开实施例提供的一种图像修复的示意图。本实施例中，以待修复图像为存在部分缺失的图像为例进行介绍，如图2中A所示，该图中左上部分存在空白(缺失)。
终端可以通过多种途径获取待修复图像，在一些可能的实现方式中，终端可以通过摄像头捕获待修复图像。例如，用户有一张纸质照片存在部分缺失，用户可以通过摄像头拍摄该纸质照片，获得数字格式的待修复图像。在另一些可能的实现方式中，终端可以通过用户在对应页面的输入操作获取待修复图像。例如，用户在其他终端中保存有数字格式的待修复图像，因此可以将该待修复图像通过任意传输方式传输至该终端，以使终端获取待修复图像。
S104：终端从图像库中确定待修复图像的多张相似图像。
其中，图像库中保存有大量的图像。终端可以根据待修复图像，在图像库中查找获得与该待修复图像相似的多张相似图像。多张相似图像中包括第一相似图像和第二相似图像。
在一些可能的实现方式中，多张相似图像与待修复图像均为同一主体在不同时刻或不同角度的图像。例如，待修复图像为某一建筑的拍摄图像， 终端可以在图像库中获取图像主体为该建筑的其他相似图像。其中，相似图像与待修复图像可能仅存在拍摄角度、拍摄距离、光线等方面的不同，因此可以作为对该待修复图像进行修复的参考数据。在另一些可能的实现方式中，多张相似图像与待修复图像为拍摄地点相同的图像。
如图2所示，待修复图像A为一建筑的图像，其确定的多张相似图像B和C可以为该建筑在不同时间、光照强度下的图像。
终端可以通过多种方式从图像库中确定待修复图像的多张相似图像。在一些可能的实现方式中，图像的参数中包括该图像的拍摄地点。终端可以根据该拍摄地点从图像库中获取在同一个地点所拍摄的图像，然后通过对比在同一个地点拍摄的图像与待修复图像的相似度，确定在同一个地点拍摄的图像是否为相似图像。例如，待修复图像为在某一著名建筑前拍摄的该著名建筑的图像。因此，可以在图片库中获得同样在该地点拍摄的其他图像，然后对比其是否与待修复图像相似，当相似时可以将其确定为待修复图像的相似图像。
在另一些可能的实现方式中，图像的参数中包括该图像的拍摄时间。终端可以根据该拍摄时间从图像库中获取同一时间拍摄的图像，然后通过对比在同一个时间拍摄的图像与待修复图像的相似度，确定在同一个时间拍摄的图像是否为相似图像。例如，待修复图像为流星雨时所拍摄的天空的图像。因此，可以在图片库中获得同样在该时刻拍摄天空的其他图像，然后对比其是否与待修复图像相似，当相似时可以将其确定为待修复图像的相似图像。
终端也可以通过神经网络模型从图像库中确定待修复图像的多张相似图像。参见图3，图3为本公开实施例提供的一种通过神经网络模型获取相似图像的流程示意图。如图3所示，该通过神经网络模型从图像库中确定待修复图像的多张相似图像的方法包括以下步骤：
S302：终端将待修复图像输入特征比对模型，获得待修复图像的搜索特征。
特征比对模型用于从待修复图像中确定出用于在图像库中进行搜索的搜索特征。搜索特征为用于寻找相似图像所需要的该待修复图像的特征。 其中，特征比对模型为经过训练的神经网络模型。通过特征比对模型所输出的搜索特征在图像特征库中进行搜索，可以获取与待修复图像相似的相似图像对应的相似特征，从而确定出相似图像。
具体地，特征比对模型可以采用卷积神经网络(convolutional neural networks,CNN)对待修复图像进行4倍下采样(subsampled)，获得原图尺寸的1/16的特征图。将特征图压平(flatten)转化为一维序列后，送入N层的编码器中进行编码(encoder)，获得编码后的一维序列，然后将编码后的一维序列转化为二维特征，获得待修复图像的搜索特征。
S304：终端根据搜索特征从图像特征库中获取与该搜索特征相似的多个相似特征。
图像特征库是与图像库存在一一对应关系的特征库，图像库中的图像与图像特征库中的特征一一对应。特征比对模型是神经网络模型通过大量相似图像训练获得的特征比对模型，通过特征比对模型所输出的搜索特征可以在图像特征库中获得与该搜索特征相似的多个相似特征。
S306：终端根据多个相似特征，从图像库中获取待修复图像的多张相似图像。
其中，图像库与图像特征库中存在一一对应关系，因此终端可以根据相似特征，从图像库中确定待修复图像的多张相似图像。
如此，终端可以通过神经网络模型从图像库中确定待修复图像的多张相似图像，从而获得进行图像修复的更多参考数据。
通常情况下，图像修复仅将图像中的参考区域作为已知数据来确定修复区域的未知数据。而本公开中则确定出待修复图像的相似图像，将相似图像也作为已知数据来确定修复区域的未知数据，有效增加了已知数据的数据量，因此可以获得较好的修复效果。
S106：终端融合待修复图像、第一相似图像和第二相似图像，并将融合后的图像输入图像修复模型，通过图像修复模型获得修复后的图像。
终端融合(concat)待修复图像与多张相似图像，获得融合后的图像。其中，多张相似图像可以包括第一相似图像和第二相似图像。参见图4，图4为本公开实施例提供的一种通过图像修复模型进行图像修复的示意 图。本实施例中，以多张相似图像为第一相似图像和第二相似图像为例进行介绍，如图4所示。
终端通过图像修复模型，将融合后的图像首先通过4层的卷积神经网络(Convolutional Neural Networks,CNN)进行8倍的下采样，然后再通过下采样(downsample)层再进行2倍的下采样，获得原图尺寸1/256的特征图。
图像修复模型包括两个分支，其中第一分支对特征图进行压平后获得一维序列，然后将一维序列输入至N层的编码器中进行编码，获得第一编码结果。第二分支对特征图进行不同尺度的分割。例如，采用特征图1/4、1/16、1/64的窗口对特征图进行分割，生成子特征图集合。然后对子特征集合中的特征图分别进行压平后获得分别对应的一维序列。将这些一维序列分别输入N层的编码器中进行编码，然后将编码器输出的不同尺度的序列经过全连接(Fully connected，FC)层映射回相同的长度，获得第二编码结果。最后将第一编码结果和第二编码结果通过FC层进行融合，获得最终的输出，从而实现对于待修复图像的修复。
由于图像修复模型具有两个分支，并且第二分支中对特征图进行不同尺度的分割，因此可以基于多个不同尺度的特征图对待修复图像进行修复，从而是图像修复更加准确，提高了图像修复的真实性，提高了用户的使用体验。
其中，图像修复模型可以通过包括多张相似图像的训练图像训练获得。其中，多张相似图像中包括掩码图像，掩码图像为对相似图像进行掩码处理后的图像。该图像修复模型的训练过程具体包括：
终端从包括多张相似图像的训练图像中获取训练特征图。具体地，终端将多张相似图像进行融合，然后将融合后的图像首先通过4层的卷积神经网络进行8倍的下采样，然后再通过下采样层再进行2倍的下采样，获得原图尺寸1/256的训练特征图。
通过图像修复模型的第一分支对训练特征图进行压平后获得一维序列，然后将一维序列输入至N层的编码器中进行编码，获得第一训练编码结果。通过图像修复模型的第二分支对训练特征图采用训练特征图1/4、 1/16、1/64的窗口对训练特征图进行不同尺度的分割，生成训练子特征图集合。然后，对于训练子特征集合中的训练特征图分别进行压平后获得分别对应的一维序列。将这些一维序列分别输入N层的编码器中进行编码，将编码器输出的不同尺度的序列经过FC层映射回相同的长度，获得第二训练编码结果。最后将第一训练编码结果和第二训练编码结果进通过FC层进行融合，获得最终的输出，从而实现对于掩码图像的修复。终端可以对比修复后的掩码图像和掩码前的训练图像，更新图像修复模型的参数。
其中，执行本实施例中图像修复方法的终端和进行模型训练的终端可以为同一终端，也可以为不同终端。在一些可能的实现方式中，终端可以将其训练完成的图像修复模型传输至多个其他终端，以使多个其他终端可以直接使用该图像修复模型，实现本公开中的图像修复方法。
基于以上内容的描述，本公开提供了一种图像修复方法。终端获取待修复图像，然后从图像库中确定待修复图像的多张相似图像。融合待修复图像和多张相似图像，并将融合后的图像输入图像修复模型。通过图像修复模型的第一分支对从融合后的图像提取的特征图进行序列化和编码，得到第一编码结果。通过图像修复模型的第二分支对特征图按照不同尺度分割的子特征图集合进行分别编码，并将子特征图集合中的子特征图的编码结果进行全连接，获得第二编码结果。根据第一编码结果和第二编码结果获得修复后的图像。
如此，一方面，终端对于图像的修复不仅可以基于待修复图像的已知区域(参考区域)对待修复区域进行预测，而且可以将相似图像作为已知区域对待修复图像的待修复区域进行预测。这增加了对于待修复区域图像进行预测的可靠数据，从而有效的提高了图像修复的效果。另一方面，图像修复模型的两个分支可以基于不同尺度的特征图对待修复图像进行修复，使图像修复更加准确，提高了图像修复的真实性，提高了用户的使用体验。
在一些可能的实现方式中，待修复图像的主体为某一著名建筑，如图2所示，该图中左上部分存在缺失。参见图5，图5为本公开实施例提供的 另一种图像修复方法的流程示意图。如图5所示，对于该待修复图像的修复包括以下步骤：
S502：终端获取待修复图像。
本实施例中的待修复图像如图2所示，终端可以通过拍摄将纸质的待修复图像转换为数字的待修复图像，也可以直接获取数字的待修复图像。
S504：终端将待修复图像输入特征比对模型，获得待修复图像的搜索特征。
特征比对模型可以采用卷积神经网络对待修复图像进行4倍下采样，获得原图尺寸的1/16的特征图。然后，将特征图压平转化为一维序列后，送入N层的编码器中进行编码，获得编码后的一维序列。将编码后的一维序列转化为二维特征，获得待修复图像的搜索特征。
S506：终端根据搜索特征从图像特征库中获取与该搜索特征相似的相似特征。
终端根据搜索特征从与图像库具有一一对应关系的图像特征库中获取与该搜索特征相似的多个相似特征。
S508：终端根据多个相似特征，从图像库中获取待修复图像的多张相似图像。
由于图像特征库中的特征与图像库中的图像一一对应，因此终端可以根据多个相似特征，在图像库中确定与多个相似特征分别对应的相似图像，如此可以获取待修复图像的多张相似图像。
S510：终端融合待修复图像与多张相似图像，将融合后的图像输入至图像修复模型，通过图像修复模型获得修复后的图像。
具体地，终端融合待修复图像与多张相似图像，获得融合后的图像。然后通过图像修复模型，将融合后的图像首先通过4层的卷积神经网络进行8倍的下采样，然后再通过下采样层再进行2倍的下采样，获得原图尺寸1/256的特征图。
进一步地，终端通过该图像修复模型的第一分支将特征图进行压平后获得一维序列，然后将一维序列输入至N层的编码器中进行编码，获得第一编码结果。
并且，终端通过图像修复模型的第二分支对特征图采用特征图1/4、1/16、1/64的窗口对特征图进行不同尺度的分割，生成子特征图集合。然后，对于子特征集合中的特征图分别进行压平后获得分别对应的一维序列。将这些一维序列分别输入至N层的编码器中进行编码，将编码器输出的不同尺度的序列经过全连接层映射回相同的长度，获得第二编码结果。其中，通过模型的第一分支和第二分支对特征图进行的分别处理可以同时进行。
最后，终端将第一编码结果和第二编码结果进通过FC层进行融合，获得最终的输出，从而实现对于待修复图像的修复。
在一些可能的实现方式中，待修复图像可以如图2中的A所示，待修复图像的相似图像可以如图2中的B和C所示，其中，B为第一相似图像，C为第二相似图像，图2中D为修复后的图像。由于修复过程中的参考数据不仅包括A中的已知部分，还包括相似图像B和C，因此具有更多的参考数据，从而能够更准确的进行修复。
图6是根据一示例性公开实施例示出的一种图像修复装置的示意图，如图6所示，所述图像修复装置600包括：
获取模块602，用于获取待修复图像；
确定模块604，用于从图像库中确定所述待修复图像的多张相似图像，所述多张相似图像至少包括第一相似图像和第二相似图像；
融合模块606，用于融合所述待修复图像和所述第一相似图像、所述第二相似图像，并将融合后的图像输入图像修复模型，通过所述图像修复模型的第一分支对从所述融合后的图像中提取的特征图进行序列化和编码，得到第一编码结果，通过所述图像修复模型的第二分支对所述特征图按照不同尺度分割的子特征图集合分别进行编码，并将所述子特征图集合中的子特征图的编码结果进行全连接，得到第二编码结果，根据所述第一编码结果和所述第二编码结果获得修复后的图像。
可选地，所述图像修复模型通过训练图像训练得到，所述训练图像包括多张相似图像，所述多张相似图像中包括掩码图像，所述掩码图像通过对所述相似图像进行掩码获得。
可选地，所述图像修复模型通过以下方式训练得到：
从所述训练图像中提取训练特征图；
通过所述图像修复模型的第一分支对所述训练特征图进行序列化和编码，获得第一训练编码结果；
通过所述图像修复模型的第二分支对所述训练特征图按照不同尺度分割的训练子特征图集合分别进行编码，并将所述训练子特征集合中的训练子特征图的编码结果进行全连接，获得第二训练编码结果；
根据所述第一训练编码结果、所述第二训练编码结果修复所述掩码图像；
根据修复后的掩码图像与掩码前的训练图像更新所述图像修复模型参数。
可选地，所述确定模块604可以用于：
将所述待修复图像输入特征比对模型，获得所述待修复图像的搜索特征；
根据所述搜索特征从图像特征库中获取与所述搜索特征相似的多个相似特征；
根据所述多个相似特征，从图像库中获取所述待修复图像的多张相似图像，所述图像库中的图像与所述图像特征库中的特征一一对应。
可选地，所述确定模块604具体用于：
将所述待修复图像输入特征比对模型，通过所述特征比对模型对所述待修复图像进行下采样获得所述待修复图像的特征图，并对所述特征图进行编码，获得所述待修复图像的搜索特征。
可选地，所述多张相似图像与所述待修复图像均为同一主体在不同时刻或不同角度的图像。
可选地，所述多张相似图像与所述待修复图像为拍摄地点相同的图像。
上述各模块的功能在上一实施例中的方法步骤中已详细阐述，在此不做赘述。
下面参考图7，其示出了适于用来实现本公开实施例的电子设备700的结构示意图。本公开实施例中的终端设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、PDA(个人数字助理)、PAD(平板电 脑)、PMP(便携式多媒体播放器)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字TV、台式计算机等等的固定终端。图7示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图7所示，电子设备700可以包括处理装置(例如中央处理器、图形处理器等)701，其可以根据存储在只读存储器(ROM)702中的程序或者从存储装置708加载到随机访问存储器(RAM)703中的程序而执行各种适当的动作和处理。在RAM703中，还存储有电子设备700操作所需的各种程序和数据。处理装置701、ROM702以及RAM703通过总线704彼此相连。输入/输出(I/O)接口705也连接至总线704。
通常，以下装置可以连接至I/O接口705：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置706；包括例如液晶显示器(LCD)、扬声器、振动器等的输出装置707；包括例如磁带、硬盘等的存储装置708；以及通信装置709。通信装置709可以允许电子设备700与其他设备进行无线或有线通信以交换数据。虽然图7示出了具有各种装置的电子设备700，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在非暂态计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置709从网络上被下载和安装，或者从存储装置708被安装，或者从ROM 702被安装。在该计算机程序被处理装置701执行时，执行本公开实施例的方法中限定的上述功能。
需要说明的是，本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计 算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、RF(射频)等等，或者上述的任意合适的组合。
在一些实施方式中，客户端、服务器可以利用诸如HTTP(HyperText Transfer Protocol，超文本传输协议)之类的任何当前已知或未来研发的网络协议进行通信，并且可以与任意形式或介质的数字数据通信(例如，通信网络)互连。通信网络的示例包括局域网(“LAN”)，广域网(“WAN”)，网际网(例如，互联网)以及端对端网络(例如，ad hoc端对端网络)，以及任何当前已知或未来研发的网络。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备：对图像进行文本检测，获得所述图像中的文本区域，所述文本区域包括多个文本行；根据所述文本区域构建图网络模型，所述文本区域中的每个文本行为所述图网络模型的一个节点；通过节点分类模型对所述图网络模型中的节点进行分类，以及通过边分类模型对所述图网络模型中的节点之间的边进行分类；根据对所述节点的分类结果以及对所述边的分类结果，获得所述图像中的至少一个键值对。可以以一种或多种程序设计语言或其组合来编写用于执行本公开的 操作的计算机程序代码，上述程序设计语言包括但不限于面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言——诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)——连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的模块可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，模块的名称在某种情况下并不构成对该模块本身的限定。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如，非限制性地，可以使用的示范类型的硬件逻辑部件包括：现场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、片上系统(SOC)、复杂可编程逻辑设备(CPLD)等等。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设 备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
根据本公开的一个或多个实施例，示例1提供了一种图像修复方法，所述方法包括：获取待修复图像；从图像库中确定所述待修复图像的多张相似图像，所述多张相似图像至少包括第一相似图像和第二相似图像；融合所述待修复图像和所述第一相似图像、所述第二相似图像，并将融合后的图像输入图像修复模型，通过所述图像修复模型的第一分支对从所述融合后的图像中提取的特征图进行序列化和编码，得到第一编码结果，通过所述图像修复模型的第二分支对所述特征图按照不同尺度分割的子特征图集合分别进行编码，并将所述子特征图集合中的子特征图的编码结果进行全连接，得到第二编码结果，根据所述第一编码结果和所述第二编码结果获得修复后的图像。
根据本公开的一个或多个实施例，示例2提供了示例1的方法，所述图像修复模型通过训练图像训练得到，所述训练图像包括多张相似图像，所述多张相似图像中包括掩码图像，所述掩码图像通过对所述相似图像进行掩码获得。
根据本公开的一个或多个实施例，示例3提供了示例2的方法，所述图像修复模型通过以下方式训练得到：从所述训练图像中提取训练特征图；通过所述图像修复模型的第一分支对所述训练特征图进行序列化和编码，获得第一训练编码结果；通过所述图像修复模型的第二分支对所述训练特征图按照不同尺度分割的训练子特征图集合分别进行编码，并将所述训练子特征集合中的训练子特征图的编码结果进行全连接，获得第二训练编码结果；根据所述第一训练编码结果、所述第二训练编码结果修复所述掩码 图像；根据修复后的掩码图像与掩码前的训练图像更新所述图像修复模型参数。
根据本公开的一个或多个实施例，示例4提供了示例1的方法，所述从图像库中确定所述待修复图像的多张相似图像，包括：将所述待修复图像输入特征比对模型，获得所述待修复图像的搜索特征；根据所述搜索特征从图像特征库中获取与所述搜索特征相似的多个相似特征；根据所述多个相似特征，从图像库中获取所述待修复图像的多张相似图像，所述图像库中的图像与所述图像特征库中的特征一一对应。
根据本公开的一个或多个实施例，示例5提供了示例4的方法，所述将所述待修复图像输入特征比对模型，获得所述待修复图像的搜索特征，包括：将所述待修复图像输入特征比对模型，通过所述特征比对模型对所述待修复图像进行下采样获得所述待修复图像的特征图，并对所述特征图进行编码，获得所述待修复图像的搜索特征。
根据本公开的一个或多个实施例，示例6提供了示例1至示例5任意一项所述的方法，所述多张相似图像与所述待修复图像均为同一主体在不同时刻或不同角度的图像。
根据本公开的一个或多个实施例，示例7提供了示例1至示例5任意一项所述的方法所述多张相似图像与所述待修复图像为拍摄地点相同的图像。
根据本公开的一个或多个实施例，示例8提供了一种图像修复装置，所述装置包括：获取模块，用于获取待修复图像；确定模块，用于从图像库中确定所述待修复图像的多张相似图像，所述多张相似图像至少包括第一相似图像和第二相似图像；融合模块，用于融合所述待修复图像和所述第一相似图像、所述第二相似图像，并将融合后的图像输入图像修复模型，通过所述图像修复模型的第一分支对从所述融合后的图像中提取的特征图进行序列化和编码，得到第一编码结果，通过所述图像修复模型的第二分支对所述特征图按照不同尺度分割的子特征图集合分别进行编码，并将所述子特征图集合中的子特征图的编码结果进行全连接，得到第二编码结果，根据所述第一编码结果和所述第二编码结果获得修复后的图像。
根据本公开的一个或多个实施例，示例9提供了示例8的装置，所述图像修复模型通过训练图像训练得到，所述训练图像包括多张相似图像，所述多张相似图像中包括掩码图像，所述掩码图像通过对所述相似图像进行掩码获得。
根据本公开的一个或多个实施例，示例10提供了示例9的装置，所述图像修复模型通过以下方式训练得到：从所述训练图像中提取训练特征图；通过所述图像修复模型的第一分支对所述训练特征图进行序列化和编码，获得第一训练编码结果；通过所述图像修复模型的第二分支对所述训练特征图按照不同尺度分割的训练子特征图集合分别进行编码，并将所述训练子特征集合中的训练子特征图的编码结果进行全连接，获得第二训练编码结果；根据所述第一训练编码结果、所述第二训练编码结果修复所述掩码图像；根据修复后的掩码图像与掩码前的训练图像更新所述图像修复模型参数。
根据本公开的一个或多个实施例，示例11提供了示例8的装置，所述确定模块具体用于：将所述待修复图像输入特征比对模型，获得所述待修复图像的搜索特征；根据所述搜索特征从图像特征库中获取与所述搜索特征相似的多个相似特征；根据所述多个相似特征，从图像库中获取所述待修复图像的多张相似图像，所述图像库中的图像与所述图像特征库中的特征一一对应。
根据本公开的一个或多个实施例，示例12提供了示例11的装置，所述确定模块具体用于：将所述待修复图像输入特征比对模型，通过所述特征比对模型对所述待修复图像进行下采样获得所述待修复图像的特征图，并对所述特征图进行编码，获得所述待修复图像的搜索特征。
根据本公开的一个或多个实施例，示例13提供了示例8至示例12任意一项所述的装置，所述多张相似图像与所述待修复图像均为同一主体在不同时刻或不同角度的图像。
根据本公开的一个或多个实施例，示例14提供了示例8至示例12任意一项所述的装置，所述多张相似图像与所述待修复图像为拍摄地点相同的图像。
以上描述仅为本公开的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本公开中所涉及的公开范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离上述公开构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本公开中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。
此外，虽然采用特定次序描绘了各操作，但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下，多任务和并行处理可能是有利的。同样地，虽然在上面论述中包含了若干具体实现细节，但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的某些特征还可以组合地实现在单个实施例中。相反地，在单个实施例的上下文中描述的各种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。
尽管已经采用特定于结构特征和/或方法逻辑动作的语言描述了本主题，但是应当理解所附权利要求书中所限定的主题未必局限于上面描述的特定特征或动作。相反，上面所描述的特定特征和动作仅仅是实现权利要求书的示例形式。关于上述实施例中的装置，其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述，此处将不做详细阐述说明。

Claims (17)

  1. 一种图像修复方法，其特征在于，所述方法包括：
    获取待修复图像；
    从图像库中确定所述待修复图像的多张相似图像，所述多张相似图像至少包括第一相似图像和第二相似图像；
    融合所述待修复图像和所述第一相似图像、所述第二相似图像，并将融合后的图像输入图像修复模型，通过所述图像修复模型的第一分支对从所述融合后的图像中提取的特征图进行序列化和编码，得到第一编码结果，通过所述图像修复模型的第二分支对所述特征图按照不同尺度分割的子特征图集合分别进行编码，并将所述子特征图集合中的子特征图的编码结果进行全连接，得到第二编码结果，根据所述第一编码结果和所述第二编码结果获得修复后的图像。
  2. 根据权利要求1所述的方法，其特征在于，所述图像修复模型通过训练图像训练得到，所述训练图像包括多张相似图像，所述多张相似图像中包括掩码图像，所述掩码图像通过对所述相似图像进行掩码获得。
  3. 根据权利要求2所述的方法，其特征在于，所述图像修复模型通过以下方式训练得到：
    从所述训练图像中提取训练特征图；
    通过所述图像修复模型的第一分支对所述训练特征图进行序列化和编码，获得第一训练编码结果；
    通过所述图像修复模型的第二分支对所述训练特征图按照不同尺度分割的训练子特征图集合分别进行编码，并将所述训练子特征集合中的训练子特征图的编码结果进行全连接，获得第二训练编码结果；
    根据所述第一训练编码结果、所述第二训练编码结果修复所述掩码图像；
    根据修复后的掩码图像与掩码前的训练图像更新所述图像修复模型的参数。
  4. 根据权利要求1所述的方法，其特征在于，所述从图像库中确定所述待修复图像的多张相似图像，包括：
    将所述待修复图像输入特征比对模型，获得所述待修复图像的搜索特征；
    根据所述搜索特征从图像特征库中获取与所述搜索特征相似的多个相似特征；
    根据所述多个相似特征，从图像库中获取所述待修复图像的多张相似图像，所述图像库中的图像与所述图像特征库中的特征一一对应。
  5. 根据权利要求4所述的方法，其特征在于，所述将所述待修复图像输入特征比对模型，获得所述待修复图像的搜索特征，包括：
    将所述待修复图像输入特征比对模型，通过所述特征比对模型对所述待修复图像进行下采样获得所述待修复图像的特征图，并对所述待修复图像的特征图进行编码，获得所述待修复图像的搜索特征。
  6. 根据权利要求1至5任意一项所述的方法，其特征在于，所述多张相似图像与所述待修复图像均为同一主体在不同时刻或不同角度的图像。
  7. 根据权利要求1至5任意一项所述的方法，其特征在于，所述多张相似图像与所述待修复图像为拍摄地点相同的图像。
  8. 一种图像修复装置，其特征在于，所述装置包括：
    获取模块，用于获取待修复图像；
    确定模块，用于从图像库中确定所述待修复图像的多张相似图像，所述多张相似图像至少包括第一相似图像和第二相似图像；
    融合模块，用于融合所述待修复图像和所述第一相似图像、所述第二相似图像，并将融合后的图像输入图像修复模型，通过所述图像修复模型的第一分支对从所述融合后的图像中提取的特征图进行序列化和编码，得到第一编码结果，通过所述图像修复模型的第二分支对所述特征图按照不同尺度分割的子特征图集合分别进行编码，并将所述子特征图集合中的子特征图的编码结果进行全连接，得到第二编码结果，根据所述第一编码结果和所述第二编码结果获得修复后的图像。
  9. 根据权利要求8所述的装置，其特征在于，所述图像修复模型通过训练图像训练得到，所述训练图像包括多张相似图像，所述多张相似图像中包括掩码图像，所述掩码图像通过对所述相似图像进行掩码获得。
  10. 根据权利要求9所述的装置，其特征在于，所述图像修复模型通过以下方式训练得到：
    从所述训练图像中提取训练特征图；
    通过所述图像修复模型的第一分支对所述训练特征图进行序列化和编码，获得第一训练编码结果；
    通过所述图像修复模型的第二分支对所述训练特征图按照不同尺度分割的训练子特征图集合分别进行编码，并将所述训练子特征集合中的训练子特征图的编码结果进行全连接，获得第二训练编码结果；
    根据所述第一训练编码结果、所述第二训练编码结果修复所述掩码图像；
    根据修复后的掩码图像与掩码前的训练图像更新所述图像修复模型参数。
  11. 根据权利要求8所述的装置，其特征在于，所述确定模块具体用于：
    将所述待修复图像输入特征比对模型，获得所述待修复图像的搜索特征；
    根据所述搜索特征从图像特征库中获取与所述搜索特征相似的多个相似特征；
    根据所述多个相似特征，从图像库中获取所述待修复图像的多张相似图像，所述图像库中的图像与所述图像特征库中的特征一一对应。
  12. 根据权利要求11所述的装置，其特征在于，所述确定模块具体用于：
    将所述待修复图像输入特征比对模型，通过所述特征比对模型对所述待修复图像进行下采样获得所述待修复图像的特征图，并对所述待修复图像的特征图进行编码，获得所述待修复图像的搜索特征。
  13. 根据权利要求8至12任意一项所述的装置，其特征在于，所述多张相似图像与所述待修复图像均为同一主体在不同时刻或不同角度的图像。
  14. 根据权利要求8至12任意一项所述的装置，其特征在于，所述多张相似图像与所述待修复图像为拍摄地点相同的图像。
  15. 一种设备，其特征在于，所述设备包括处理器和存储器；
    所述处理器用于执行所述存储器中存储的指令，以使得所述设备执行如权利要求1至7中任一项所述的方法。
  16. 一种计算机可读存储介质，其特征在于，包括指令，所述指令指示设备执行如权利要求1至7中任一项所述的方法。
  17. 一种计算机程序产品，其特征在于，当所述计算机程序产品在计算机上运行时，使得计算机执行如权利要求1至7中任一项所述的方法。
PCT/CN2023/078345 2022-03-21 2023-02-27 图像修复方法、装置、设备、介质及产品 WO2023179310A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210278129.3 2022-03-21
CN202210278129.3A CN114581336A (zh) 2022-03-21 2022-03-21 图像修复方法、装置、设备、介质及产品

Publications (1)

Publication Number Publication Date
WO2023179310A1 true WO2023179310A1 (zh) 2023-09-28

Family

ID=81781733

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/078345 WO2023179310A1 (zh) 2022-03-21 2023-02-27 图像修复方法、装置、设备、介质及产品

Country Status (2)

Country Link
CN (1) CN114581336A (zh)
WO (1) WO2023179310A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114581336A (zh) * 2022-03-21 2022-06-03 北京有竹居网络技术有限公司 图像修复方法、装置、设备、介质及产品
CN114943241B (zh) * 2022-07-26 2022-10-25 北京紫光青藤微系统有限公司 用于修复条码码图的方法、电子设备、存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150130799A1 (en) * 2013-11-12 2015-05-14 Fyusion, Inc. Analysis and manipulation of images and video for generation of surround views
CN111325704A (zh) * 2020-02-07 2020-06-23 北京字节跳动网络技术有限公司 图像修复方法、装置、电子设备及计算机可读存储介质
CN113724143A (zh) * 2020-09-27 2021-11-30 天翼智慧家庭科技有限公司 用于图像修复的方法和装置
CN114581336A (zh) * 2022-03-21 2022-06-03 北京有竹居网络技术有限公司 图像修复方法、装置、设备、介质及产品

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110223256A (zh) * 2019-06-10 2019-09-10 北京大学深圳研究生院 一种逆色调映射方法、装置及电子设备
CN113592724A (zh) * 2020-04-30 2021-11-02 北京金山云网络技术有限公司 目标人脸图像的修复方法和装置
CN113870254B (zh) * 2021-11-30 2022-03-15 中国科学院自动化研究所 目标对象的检测方法、装置、电子设备及存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150130799A1 (en) * 2013-11-12 2015-05-14 Fyusion, Inc. Analysis and manipulation of images and video for generation of surround views
CN111325704A (zh) * 2020-02-07 2020-06-23 北京字节跳动网络技术有限公司 图像修复方法、装置、电子设备及计算机可读存储介质
CN113724143A (zh) * 2020-09-27 2021-11-30 天翼智慧家庭科技有限公司 用于图像修复的方法和装置
CN114581336A (zh) * 2022-03-21 2022-06-03 北京有竹居网络技术有限公司 图像修复方法、装置、设备、介质及产品

Also Published As

Publication number Publication date
CN114581336A (zh) 2022-06-03

Similar Documents

Publication Publication Date Title
US11367313B2 (en) Method and apparatus for recognizing body movement
WO2023179310A1 (zh) 图像修复方法、装置、设备、介质及产品
WO2022257578A1 (zh) 用于识别文本的方法和装置
CN111414879B (zh) 人脸遮挡程度识别方法、装置、电子设备及可读存储介质
WO2022012179A1 (zh) 生成特征提取网络的方法、装置、设备和计算机可读介质
CN112668588B (zh) 车位信息生成方法、装置、设备和计算机可读介质
WO2022171036A1 (zh) 视频目标追踪方法、视频目标追踪装置、存储介质及电子设备
CN111784712B (zh) 图像处理方法、装置、设备和计算机可读介质
US20240221126A1 (en) Image splicing method and apparatus, and device and medium
WO2022105622A1 (zh) 图像分割方法、装置、可读介质及电子设备
CN112418249A (zh) 掩膜图像生成方法、装置、电子设备和计算机可读介质
CN112330788A (zh) 图像处理方法、装置、可读介质及电子设备
WO2023072015A1 (zh) 人物风格形象图的生成方法、装置、设备及存储介质
CN115578570A (zh) 图像处理方法、装置、可读介质及电子设备
CN112598673A (zh) 全景分割方法、装置、电子设备和计算机可读介质
CN113610034B (zh) 识别视频中人物实体的方法、装置、存储介质及电子设备
WO2022012178A1 (zh) 用于生成目标函数的方法、装置、电子设备和计算机可读介质
CN112257598B (zh) 图像中四边形的识别方法、装置、可读介质和电子设备
CN111862351B (zh) 定位模型优化方法、定位方法和定位设备
WO2023179291A1 (zh) 图像修复方法、装置、设备、介质及产品
WO2023202543A1 (zh) 文字处理方法、装置、电子设备及存储介质
CN114627353B (zh) 一种图像描述生成方法、装置、设备、介质及产品
CN114742707B (zh) 多源遥感图像拼接方法、装置、电子设备和可读介质
WO2022052889A1 (zh) 图像识别方法、装置、电子设备和计算机可读介质
CN113807056B (zh) 一种文档名称序号纠错方法、装置和设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23773561

Country of ref document: EP

Kind code of ref document: A1