WO2024065701A1 - 图像修复方法、装置、设备和非瞬态计算机存储介质 - Google Patents

图像修复方法、装置、设备和非瞬态计算机存储介质 Download PDF

Info

Publication number
WO2024065701A1
WO2024065701A1 PCT/CN2022/123334 CN2022123334W WO2024065701A1 WO 2024065701 A1 WO2024065701 A1 WO 2024065701A1 CN 2022123334 W CN2022123334 W CN 2022123334W WO 2024065701 A1 WO2024065701 A1 WO 2024065701A1
Authority
WO
WIPO (PCT)
Prior art keywords
scratch
image
feature map
original
network
Prior art date
Application number
PCT/CN2022/123334
Other languages
English (en)
French (fr)
Inventor
孙梦笛
Original Assignee
京东方科技集团股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司 filed Critical 京东方科技集团股份有限公司
Priority to CN202280003405.3A priority Critical patent/CN118140247A/zh
Priority to PCT/CN2022/123334 priority patent/WO2024065701A1/zh
Publication of WO2024065701A1 publication Critical patent/WO2024065701A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration

Definitions

  • the present application relates to the field of image technology, and in particular to an image restoration method, device, equipment and non-transient computer storage medium.
  • Image restoration refers to an image processing technique that reconstructs lost or damaged parts of an image, such as repairing scratches or blemishes on old photos.
  • the original image to be restored is first obtained, and then the original scratches in the original image to be restored and the original image are marked to obtain a marked intermediate feature map, and the intermediate feature map is input into an image restoration network to perform feature fusion on the original scratches and the original image to obtain a restored image.
  • the present application provides an image restoration method, device, equipment and non-transient computer storage medium.
  • the technical solution is as follows:
  • an image restoration method comprising:
  • the scratch enhancement feature map is input into an image restoration network to repair the dilated scratches in the scratch enhancement feature map, thereby obtaining a restored image corresponding to the original image to be restored.
  • processing the original scratches in the original scratch feature map into expanded scratches to obtain the expanded scratch feature map includes:
  • Target area includes pixel points of the original scratch in the original scratch feature map
  • the detection frame is a square detection frame, and a moving step length of the detection frame during traversal is equal to a side length.
  • the scratch enhancement feature map includes an expanded scratch feature and original image features except for an area where the expanded scratch feature is located;
  • the step of inputting the scratch enhancement feature map into an image restoration network to restore the dilated scratches in the scratch enhancement feature map to obtain a restored image corresponding to the original image to be restored includes:
  • the original image features in the scratch enhancement feature map are fused with the dilated scratch features through the image restoration network to restore the dilated scratches in the scratch enhancement feature map to obtain a restored feature map.
  • the restoration feature map is transformed in dimension to obtain a restoration image corresponding to the original image to be restored.
  • the image restoration network includes multiple feature fusion networks
  • the method of fusing the original image features in the scratch enhancement feature map with the dilated scratch features through the image restoration network to restore the dilated scratches in the scratch enhancement feature map to obtain a restored feature map includes:
  • the original image features and the dilated scratch features in the global feature map are fused through the multiple feature fusion networks to obtain a local repair feature map;
  • the global feature map and the local restoration feature map are merged to obtain the restoration feature map.
  • the multiple feature fusion networks include a first feature fusion network, multiple second feature fusion networks and a third feature fusion network, and the first feature fusion network, the multiple second feature fusion networks and the third feature fusion network each include multiple attention networks;
  • the fusing the original image features and the dilated scratch features in the global feature map through the multiple feature fusion networks to obtain a local repair feature map includes:
  • the sixth feature map is upsampled to obtain the local repair feature map.
  • the attention network includes a first image enhancement network and a second image enhancement network; the method further includes:
  • the image to be processed is input into the first image enhancement network to obtain a first enhanced image
  • the image to be processed is passed through the second image enhancement network to obtain a second enhanced image
  • the first image enhancement network includes a plurality of convolutional layers, a plurality of depth-separable convolutional layers and a pooling layer;
  • the second image enhancement network includes a convolutional layer and a depth-separable convolutional layer
  • the first enhanced image network and the second image enhancement network are used to expand the receptive field of the image to be processed, wherein the receptive field expansion capability of the first image enhancement network is greater than that of the second image enhancement network.
  • the training method of the image restoration network includes:
  • the training data set includes an original image to be repaired and a training sample, wherein the original image to be repaired includes scratches, and the training sample is an image with complete image content;
  • the image restoration network to be trained is determined as the image restoration network.
  • the expanded scratch sample includes a plurality of the expanded scratches, and any two of the plurality of expanded scratches have different sizes.
  • the loss function of the image restoration network includes:
  • Loss is the contrast difference
  • I gt is the training sample
  • the preset value is 0;
  • the step of overlaying the expanded scratch in the expanded scratch feature map at the location of the original scratch in the original image to be repaired to obtain a scratch enhancement feature map includes:
  • the dilated scratch feature map is multiplied by the original image to be repaired to obtain the scratch enhancement feature map, wherein the pixel value of the pixel point of the dilated scratch in the obtained scratch enhancement feature map is 0.
  • an image restoration device comprising:
  • a first acquisition module is used to acquire the original image to be restored
  • a detection module used for performing scratch detection on the original image to be repaired
  • a second acquisition module configured to obtain an original scratch feature map of the original image to be repaired in response to detecting the original scratch in the original image to be repaired;
  • an expansion module configured to process an original scratch in the original scratch feature map into an expanded scratch to obtain an expanded scratch feature map, wherein the expanded scratch includes at least one scratch unit, and the original scratch is located in a region where the at least one scratch unit is located;
  • An enhancement module used for overlaying the expanded scratch in the expanded scratch feature map at the location of the original scratch in the original image to be repaired, to obtain a scratch enhancement feature map
  • the repair module is used to input the scratch enhancement feature map into the image repair network to repair the dilated scratches in the scratch enhancement feature map to obtain a repaired image corresponding to the original image to be repaired.
  • the expansion module comprises:
  • a third acquisition module is used to acquire a target area, where the target area includes pixel points of the original scratch in the original scratch feature map;
  • a traversal module used for traversing the target area through a detection frame
  • a pixel processing module is used for processing a plurality of pixel points in the detection frame into the scratch unit in response to detecting the presence of pixel points of the original scratch in the detection frame, wherein the pixel values of the pixel points in the scratch unit are preset values, so as to obtain the expanded scratch feature map.
  • the repair module is used to:
  • the original image features in the scratch enhancement feature map are fused with the dilated scratch features through the image restoration network to restore the dilated scratches in the scratch enhancement feature map to obtain a restored feature map.
  • an image restoration device which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the image restoration method as described above.
  • a non-volatile computer storage medium in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the image restoration method as described above.
  • a computer program product or computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium.
  • a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned image restoration method.
  • An image restoration method which processes the original scratches in the original image to be restored into dilated scratches with a relatively regular shape, and covers the dilated scratches at the positions of the original scratches in the original image to be restored, thereby obtaining a scratch enhancement feature map, and then uses an image restoration network to restore the dilated scratches in the scratch enhancement feature map, so as to obtain a restored image corresponding to the original image to be restored. Since the shape of the dilated scratches is relatively regular, the restoration effect of the scratches in the original image can be improved, which solves the problem of poor restoration effect of the restored image in the related art and improves the restoration effect of the restored image.
  • FIG1 is a flow chart of an image restoration method provided by an embodiment of the present application.
  • FIG2 is a flow chart of another image restoration method provided by an embodiment of the present application.
  • FIG3 is a schematic diagram of a network structure of a scratch detection network provided in an embodiment of the present application.
  • FIG4 is a schematic diagram of the network structure of some modules in the scratch detection network shown in FIG3 ;
  • FIG5 is a schematic diagram of a structure of a detection frame traversing a target area provided by an embodiment of the present application
  • FIG6 is a network architecture diagram for processing an original scratch into an expanded scratch provided by an embodiment of the present application.
  • FIG7 is a network architecture diagram of an image restoration provided by an embodiment of the present application.
  • FIG8 is a flow chart of obtaining a repair feature map provided by an embodiment of the present application.
  • FIG9 is a schematic diagram of a network structure of an image restoration network provided in an embodiment of the present application.
  • FIG10 is a schematic diagram of a network structure of a feature fusion network provided in an embodiment of the present application.
  • FIG11 is a schematic diagram of a network structure of an attention network provided in an embodiment of the present application.
  • FIG12 is a schematic diagram of an image restoration process shown in an embodiment of the present application.
  • FIG13 is a network architecture diagram of an image restoration network training provided by an embodiment of the present application.
  • FIG14 is a training flowchart of an image restoration network provided in an embodiment of the present application.
  • FIG15 is a structural block diagram of an image restoration device provided in an embodiment of the present application.
  • Image restoration technology refers to an image processing technology that restores the pixel features of the damaged part of the incomplete image to obtain a complete image.
  • image restoration technology can be used to remove unnecessary objects in the image, restore the damaged part of the image, etc.
  • Image restoration technology can include image restoration methods based on machine learning.
  • Machine learning is a branch of artificial intelligence.
  • Machine learning refers to using computers as tools to learn from big data the representation of various things in the real world that can be directly used for computer calculations.
  • machine learning can be used for target detection, image generation, and image segmentation.
  • Artificial intelligence refers to a technology that studies the design principles and implementation methods of various intelligent machines to enable machines to have perception, reasoning, and decision-making functions.
  • photos may be damaged, causing older photos to become worn and blurred, such as multiple scratches on the photos, causing the photos to be incomplete and unclear.
  • Image processing technology can repair damaged photos to obtain clearer and more complete photos.
  • the implementation environment may include a photo, a shooting component, a server, and a display terminal.
  • the shooting component may include a camera, which may be used to shoot a real photo into a digital image.
  • the server includes a processor, and the server may establish a wired or wireless connection with the shooting component to generate a repaired image based on the image captured by the shooting component, and display the repaired image on the display terminal.
  • FIG1 is a flow chart of an image restoration method provided by an embodiment of the present application.
  • the method can be applied to the server in the above implementation environment.
  • the method can include the following steps:
  • Step 101 Obtain an original image to be restored.
  • Step 102 Perform scratch detection on the original image to be repaired.
  • Step 103 In response to detecting the original scratch in the original image to be repaired, an original scratch feature map of the original image to be repaired is obtained.
  • Step 104 Process the original scratches in the original scratch feature map into dilated scratches to obtain an dilated scratch feature map.
  • the expanded scratch includes at least one scratch unit, and the original scratch is located in the area where the at least one scratch unit is located.
  • Step 105 Overlay the expanded scratch in the expanded scratch feature map on the location of the original scratch in the original image to be repaired, to obtain a scratch enhancement feature map.
  • Step 106 Input the scratch enhancement feature map into the image restoration network to repair the dilated scratches in the scratch enhancement feature map, and obtain a restored image corresponding to the original image to be restored.
  • the embodiment of the present application provides an image restoration method, which processes the original scratches in the original image to be restored into dilated scratches with a relatively regular shape, and covers the dilated scratches at the positions of the original scratches in the original image to be restored, obtains a scratch enhancement feature map, and then uses an image restoration network to restore the dilated scratches in the scratch enhancement feature map to obtain a restored image corresponding to the original image to be restored. Since the shape of the dilated scratches is relatively regular, the restoration effect of the scratches in the original image can be improved, which solves the problem of poor restoration effect of the restored image in the related art and improves the restoration effect of the restored image.
  • FIG2 is a flow chart of another image restoration method provided by an embodiment of the present application.
  • the method can be applied to the server in the above implementation environment.
  • the method can include the following steps:
  • Step 201 Obtain an original image to be restored.
  • the original image to be repaired may include an image with image repair requirements, that is, the original image to be repaired may include the original image and a lost or damaged portion (such as an original scratch).
  • the original image to be repaired may also include an image with relatively complete content that does not require image repair.
  • the original image to be repaired may be an image obtained by taking a photo with a camera of the terminal device; or, the original image to be repaired may also be an image obtained from inside the terminal device.
  • the original image to be repaired may be an image stored in an album of the terminal device, or an image obtained by the terminal device from the cloud.
  • Step 202 Perform scratch detection on the original image to be repaired.
  • the scratch detection network can be used to detect scratches on the original image to be repaired to identify whether there are original scratches in the original image to be repaired. If there are original scratches in the original image to be repaired, the scratch detection network can be used to detect the position and shape of the original scratches in the original image. If there are no original scratches in the original image to be repaired, it indicates that the clarity and integrity of the original image to be repaired are good, and the original image to be repaired can be directly output as a repaired image.
  • the scratch detection network may include a YOLOv5 detection network. Since the YOLO series algorithm has a relatively simple structure and a fast computing speed, it can be widely used in detection-related image processing processes. As shown in Figures 3 and 4, Figure 3 is a schematic diagram of a network structure of a scratch detection network provided by an embodiment of the present application, and Figure 4 is a schematic diagram of the network structure of some modules in the scratch detection network shown in Figure 3. As can be seen from Figure 3, the YOLOv5 detection network may include an input end, a backbone network (Backbone), a neck network (Neck) and an output end.
  • Backbone backbone
  • Neck neck network
  • Foc represents the focus network, which can be used to reduce the amount of calculation of the YOLOv5 detection network and increase the calculation speed
  • Conv represents the convolution layer
  • Con (Concat) represents the merging or permutation operation of the matrix
  • Lr (Leaky relu) represents the activation function
  • CBL consists of a convolution layer (Conv), a batch normalization process (BN) and an activation function (Leaky relu), where the batch normalization process (BN) can also be called a normalized batch process.
  • SPP consists of multiple pooling layers (maxpool) and Concat.
  • CSP1-x represents the first detection network, where x can be 1, 2 or 3, etc.
  • CSP2-x represents the second detection network, where x can be 1, 2 or 3, etc.
  • Res unit represents the residual component
  • a (add) represents the addition operation of the matrix
  • sl (slice) represents the data slice.
  • the Focus layer in FIG4 can split a higher-resolution image (feature map) into multiple lower-resolution images (feature maps) through a data slicing operation, that is, the higher-resolution image is sampled in alternate columns and then spliced.
  • the original 640 ⁇ 640 ⁇ 3 image is input into the Focus layer, and the image is divided into a total of 4 parts through data slicing, that is, the image is processed into a 320 ⁇ 320 ⁇ 12 feature map, and the feature map is concatenated (Concat), and then subjected to a convolution (CBL) process, and finally becomes a 320 ⁇ 320 ⁇ 64 feature map.
  • Processing the feature map through the Focus layer can reduce information loss.
  • the SPP layer in Figure 4 performs maximum pooling on the same image through multiple pooling kernels to obtain multiple feature maps, and then fuses the multiple feature maps and the feature maps that have not been processed by maximum pooling, wherein the sizes of any two pooling kernels in the multiple pooling kernels are different.
  • the SPP layer fuses features of different sizes so that the scratch detection network can be applied to the situation where the scratch sizes in the original image to be repaired are greatly different.
  • the scratch detection network can be a pre-trained neural network; the training data can include sample images and scratch images, and the scratch image refers to an image with scratches covered on the sample image.
  • the sample image can refer to an image with high clarity and relatively complete image content in the sample data set, and the scratch image can refer to an image with scratches of different shapes covered on the sample image.
  • the scratch detection network in the embodiment of the present application can also be other detection networks such as the YOLOv4 detection network.
  • Step 203 In response to detecting the original scratch in the original image to be repaired, an original scratch feature map of the original image to be repaired is obtained.
  • the scratch detection network can output an original scratch feature map. Specifically, the pixel values of the pixels in the original scratch (Mask) area in the original image to be repaired can be set to 0, and the pixel values of the pixels outside the original scratch (Mask) area in the original image to be repaired can be set to 1. The area outside the original scratch (Mask) area can be called the original image area to obtain the original scratch feature map.
  • the original scratch feature map can be a matrix including 0 and 1.
  • Step 204 Acquire the target area.
  • the target area may include pixel points of the original scratch in the original scratch feature map.
  • the original scratch feature map may have multiple original scratches, and the area where the pixel points of each original scratch in the original scratch feature map are located can be obtained respectively.
  • a coordinate system can be established based on the original scratch feature map to obtain the coordinates of the pixel points of multiple original scratches in the original scratch feature map, and then obtain the target area where the pixel points of the multiple original scratches are located.
  • FIG5 is a schematic diagram of a structure of a detection frame traversing a target area provided by an embodiment of the present application.
  • the original scratch feature map can be a rectangle, and a two-dimensional coordinate system (x, y) is established with a vertex of the original scratch feature map as the point (0, 0), and the coordinates of multiple pixel points in the original scratch R1 are obtained to obtain the first pixel point c1, the second pixel point c2, and the third pixel point c3 in the original scratch R1.
  • the x coordinate value of the first pixel point c1 and the x coordinate value of the second pixel point c2 are respectively the maximum and minimum values of the x coordinate values of the multiple pixel points in the original scratch
  • the y coordinate value of the third pixel point c3 and the y coordinate value of the fourth pixel point c4 are respectively the maximum and minimum values of the y coordinate values of the multiple pixel points in the original scratch R1. That is, by obtaining the four vertices of the original scratch R1 in the x direction and the y direction, and based on these four vertices, the target area where the pixel points of the original scratch R1 in the original scratch feature map are located is obtained.
  • Step 205 traverse the target area through the detection frame.
  • the detection frame can be used to detect whether there are pixels of the original scratch in the detection frame during the process of traversing the target area.
  • the detection frame can be a rectangular detection frame.
  • the detection frame a1 is a square detection frame, and the moving step length of the detection frame a1 during traversal is equal to the side length. In this way, the detection efficiency of the detection frame can be improved.
  • the side length of the detection frame can be 4, and the moving step length during traversal is also 4.
  • Step 206 In response to detecting that there are pixels of original scratches in the detection frame, multiple pixels in the detection frame are processed into scratch units, and the pixel values of the pixels in the scratch units are preset values to obtain an expanded scratch feature map.
  • FIG6 is a network architecture diagram for processing an original scratch into an expanded scratch provided by an embodiment of the present application, please refer to FIG6.
  • the original scratch in the original scratch feature map can be processed into an expanded scratch
  • the irregular original scratch can be processed into a more regular expanded scratch to obtain an expanded scratch feature map.
  • the expanded scratch includes at least one scratch unit, and the original scratch is located in the area where the at least one scratch unit is located.
  • the scratch unit can be a rectangular scratch unit, which can improve the regularity of the shape of the expanded scratch.
  • the preset value can be 0. If the pixel point of the original scratch is detected in the detection frame, the pixel values of multiple pixel points in the detection frame are assigned to 0, that is, the pixel value of the pixel point in the scratch unit is 0, so as to obtain the expanded scratch R2 corresponding to the original scratch R1, and then obtain the expanded scratch feature map.
  • Step 207 Overlay the expanded scratch in the expanded scratch feature map on the location of the original scratch in the original image to be repaired, to obtain a scratch enhancement feature map.
  • the dilated scratch feature map and the original image to be repaired can be multiplied to obtain a scratch enhancement feature map, wherein, since the above preset value can be 0, the pixel value of the pixel point in the dilated scratch in the dilated scratch feature map is also 0, and therefore, the pixel value of the pixel point of the dilated scratch in the scratch enhancement feature map is 0.
  • the expanded scratch feature map can be a matrix including 0 and 1
  • the original image to be repaired can be a matrix including multiple pixel values (such as 5, 8, 20, etc.).
  • the pixel value of the pixel point multiplied by 0 in the original image to be repaired is 0, and the pixel value of the pixel point multiplied by 1 in the original image to be repaired remains unchanged, which is equivalent to covering the expanded scratch in the expanded scratch feature map at the position of the original scratch in the original image to be repaired, and the pixel values of the pixel points at the position of the expanded scratch are all 0.
  • Step 208 Obtain an image restoration network.
  • the image restoration network may be a trained fusion network, which may include multiple convolutional layers and multiple feature fusion networks.
  • the multiple feature fusion networks may be used to fuse original image features with dilated scratch features to fill in the dilated scratches in the scratch enhancement feature map.
  • the original image features in the scratch enhancement feature map and the dilated scratch features can be fused through the image restoration network to restore the dilated scratches in the scratch enhancement feature map to obtain a restoration feature map.
  • the multiple feature fusion networks can also be trained networks.
  • Step 209 input the scratch enhancement feature map into the image restoration network, and fuse the original image features in the scratch enhancement feature map with the dilated scratch features through the image restoration network to repair the dilated scratches in the scratch enhancement feature map, thereby obtaining a repair feature map.
  • FIG7 is a network architecture diagram of an image restoration provided by an embodiment of the present application, please refer to FIG7.
  • the image restoration network is a network trained based on a sample scratch feature map including dilated scratches.
  • the scratch enhancement feature map includes dilated scratch features and original image features except for the area where the dilated scratch features are located.
  • the image restoration network in the related art Since the texture features of the original scratches in the original scratch feature map are refined, complex and irregular, the image restoration network in the related art has low restoration accuracy for complex original scratches, and the filling content of the generated original scratches is inaccurate, resulting in incomplete restoration of some original scratches with small size or complex texture.
  • the image restoration network is trained based on the sample scratch images including the original scratches, and since the texture of the original scratches is complex and changeable, the training of the image restoration network is difficult.
  • the image restoration network is trained by a sample scratch feature map including dilated scratches.
  • the original scratches in the original scratch feature map are first processed into dilated scratches by scratch detection and scratch dilation to obtain the dilated scratch feature map, and then the dilated scratches in the dilated scratch feature map are covered at the location of the original scratches in the original image to be repaired to obtain the scratch enhancement feature map, so that the dilated scratches in the scratch enhancement feature map have simpler and more regular texture features than the original scratches, which can reduce the difficulty of image restoration.
  • the image restoration network in the embodiment of the present application is a network trained based on the sample scratch feature map including dilated scratches, and the texture of the dilated scratches is relatively simple, which can make the training difficulty of the image restoration network relatively small. Furthermore, by processing the original scratches into dilated scratches, the adaptability of the original image to be repaired and the image restoration network can be improved, and at the same time, the universality of the image restoration network can also be improved.
  • step 209 may include the following three sub-steps:
  • Sub-step 2091 extract features from the scratch enhancement feature map to obtain a global feature map.
  • the feature extraction process may be performed by downsampling or the like.
  • the feature extraction process may be performed by downsampling or the like.
  • Downsampling the scratch enhancement feature map can reduce the dimension of the scratch enhancement feature map and retain valid information, thereby avoiding overfitting.
  • Sub-step 2092 fusing the original image features and the dilated scratch features in the global feature map through multiple feature fusion networks to obtain a local repair feature map.
  • multiple feature fusion networks may include a first feature fusion network (FFG1), multiple second feature fusion networks (FFG2) and a third feature fusion network (FFG3), and the first feature fusion network (FFG1), multiple second feature fusion networks (FFG2) and the third feature fusion network (FFG3) all include multiple attention networks (MAB).
  • the attention network (MAB) can also be a trained network.
  • Conv represents the convolution layer
  • SConv represents the strided convolution
  • the step size of the strided convolution here is 2, that is, the image feature map is reduced by 2 times after the strided convolution.
  • up-sampling (Up-sample) can be used to restore the image feature map.
  • FIG10 is a schematic diagram of the network structure of a feature fusion network provided in an embodiment of the present application, please refer to FIG10 .
  • the structures of the first feature fusion network, multiple second feature fusion networks, and the third feature fusion network may be the same, and may all be referred to as feature fusion networks (English: Feature Fusion Group; abbreviated: FFG).
  • the feature fusion network may include multiple attention networks (English: Multi-attention Block; abbreviated: MAB).
  • Conv represents a convolutional layer
  • Con represents a matrix merge or permutation operation
  • C Shu represents Channel Shuffle, which is used to mix the information between the connection channels
  • the convolutional layer may be a 1 ⁇ 1 convolutional layer
  • the 1x1 convolutional layer may be used to reduce the channels of the feature map.
  • FIG 11 is a schematic diagram of the network structure of an attention network provided in an embodiment of the present application.
  • Conv represents a convolutional layer
  • SConv represents a strided convolution
  • Concat represents a matrix merge or permutation operation
  • DConv represents a dilated convolution
  • DWConv represents a deep convolution
  • S represents a Sigmoid activation function.
  • the Sigmoid activation function is a logical activation function, also known as an S-shaped growth curve.
  • the Sigmoid function can be used as an activation function of a neural network to map variables between [0, 1]. After the initial extraction of the feature map, it is input into the attention network and divided into three branches for different processing.
  • the upper branch passes through a 1x1 convolution, a 5x5 depth separable convolution, and a Sigmoid activation function.
  • the middle branch first passes through two 3x3 convolutions and a 1x1 convolution, a 3x3 convolution with a step size of 2, and maximum pooling. It is followed by two parallel 3x3 depth separable convolutions, which are added and then upsampled. Then, through a 1x1 convolution and a Sigmoid activation function, the lower branch is actually a jump connection to multiply the output of the middle branch, and then the result is multiplied again with the output of the upper branch to obtain the final output.
  • the Sigmoid activation function can map variables to the interval of (0,1), and the data is not easy to diverge during the transmission process.
  • the hollow convolution can expand the receptive field and help restore the image details missing caused by the oversaturated area and motion dislocation.
  • the structure of the attention network in the embodiment of the present application can be an attention network as shown in Figure 11, or it can be an attention network of other structures, and the embodiment of the present application is not limited to this.
  • the attention network is introduced into the image restoration work.
  • the attention network can not only utilize the information around the area where the dilated scratch is located, but also enhance and utilize the features in the global information of the entire image that are beneficial to restoration, so that the structure and texture in the restored image are clearer and more coherent.
  • the attention network may include a first image enhancement network and a second image enhancement network. Processing an image through the attention network may include the following steps:
  • the image to be processed can be an image feature input into the attention network during the process of processing the image through the feature network in the embodiment of the present application. Since the image restoration network includes multiple attention networks, the image to be processed can be different image features and does not specifically refer to the image to be processed in a certain attention network.
  • the attention network can include a first image enhancement network and a second image enhancement network.
  • the image to be processed is input into the first image enhancement network to obtain a first enhanced image.
  • the first image enhancement network may include multiple convolutional layers, multiple depth-separable convolutional layers, and pooling layers.
  • the first enhanced image network is used to expand the receptive field of the image to be processed. After the image to be processed is input into the attention network, the image to be processed can be input into the first image enhancement network in the attention network to process the image through the first image enhancement network to obtain a first enhanced image.
  • the image to be processed is passed through a second image enhancement network to obtain a second enhanced image.
  • the second image enhancement network may include a convolution layer and a depth-separable convolution layer.
  • the second image enhancement network is used to expand the receptive field of the image to be processed, wherein the receptive field expansion capability of the first image enhancement network is greater than that of the second image enhancement network.
  • the attention network can fuse the first enhanced image obtained by the first image enhancement network with the image to be processed that has not been processed by the attention network, so that the details of the first intermediate image are richer.
  • the attention network can also fuse the second enhanced image obtained by the second image enhancement network with the first intermediate image to make the first intermediate image more detailed. In this way, the information from different convolutional layers can be fully utilized, so that more details are retained in the obtained feature enhanced image, which helps to restore the details of the scratch area in the image to be processed.
  • the above-mentioned method of repairing an image through an attention network can be applied to the process of fusing the original image features and the dilated scratch features in the scratch enhancement feature map in the embodiment of the present application, that is, it can be applied to each feature fusion network of multiple feature fusion networks.
  • the feature enhanced image can be the output result of any attention network in the feature fusion network.
  • fusing the original image features and the dilated scratch features in the global feature map through multiple feature fusion networks to obtain a local repair feature map may include the following steps:
  • Features in the global feature map that are beneficial to repairing the dilated scratches may be extracted to obtain a first feature map.
  • the first feature fusion network can perform a first fusion of the dilated scratch feature and the original image feature, and the multiple feature fusion networks can have different focuses when fusing the dilated scratch feature and the original image feature. For example, the first feature fusion network focuses on repairing the color in the area where the dilated scratch is located.
  • the dilated scratch features and the original image features in the third feature map are fused multiple times through multiple second feature fusion networks to obtain a fourth feature map.
  • the plurality of second feature fusion networks can perform a second fusion on the dilated scratch features and the original image features.
  • the second feature fusion network focuses on repairing the texture in the area where the dilated scratch is located.
  • Upsampling can include image processing methods such as pixel shuffer.
  • the first feature map and the fifth feature map are merged and input into the third feature fusion network to obtain a sixth feature map.
  • the image feature maps fused by fusion networks with different focuses can be merged, and the dilated scratch features and the original image features can be fused again through a third feature fusion network to make the details in the fused image features richer.
  • the first characteristic map and the fifth characteristic map in the embodiment of the present application are essentially a matrix.
  • the merger of the first characteristic map and the fifth characteristic map is essentially the merger of two matrices, which is a process of arranging or merging two matrices without changing the order of the two matrices themselves.
  • Sub-step 2093 merge the global feature map and the local repair feature map to obtain a repair feature map.
  • the global feature map and the local restoration feature map can be merged to make the restoration feature map have more image details.
  • Such an image restoration network can pay attention to both local and global features in the original image to be restored, which can improve the restoration performance of the network.
  • Step 210 perform dimension transformation on the repair feature map to obtain a repaired image corresponding to the original image to be repaired.
  • the dimensional transformation may be performed by upsampling or downsampling.
  • upsampling or downsampling.
  • FIG. 12 is a schematic diagram of an image repair process shown in an embodiment of the present application.
  • the original scratches in the original image to be repaired are processed into dilated scratches, and the dilated scratches are covered on the positions of the original scratches in the original image to be repaired, so as to obtain a scratch enhancement feature map, and then the image repair network is used to repair the dilated scratches in the scratch enhancement feature map to obtain a repaired image corresponding to the original image to be repaired.
  • an image repair network with strong repair ability and good universality can be obtained. The image is repaired using the image repair network with high repair efficiency and good repair effect.
  • the embodiment of the present application provides an image restoration method, which processes the original scratches in the original image to be restored into dilated scratches with a relatively regular shape, and covers the dilated scratches at the positions of the original scratches in the original image to be restored, obtains a scratch enhancement feature map, and then uses an image restoration network to restore the dilated scratches in the scratch enhancement feature map to obtain a restored image corresponding to the original image to be restored. Since the shape of the dilated scratches is relatively regular, the restoration effect of the scratches in the original image can be improved, which solves the problem of poor restoration effect of the restored image in the related art and improves the restoration effect of the restored image.
  • the image restoration network in step 208 may be a pre-trained image restoration network, or the image restoration network may be trained in step 208.
  • each network (scratch detection network, first feature fusion network, multiple second feature fusion networks, third feature fusion network, and attention network) used in the embodiments of the present application is a trained network structure, and each network can be trained by deep learning, which is a method of machine learning. The embodiments of the present application do not limit the training methods of these networks.
  • FIG13 is a network architecture diagram of an image restoration network training provided by an embodiment of the present application, please refer to FIG13.
  • the training of the image restoration network can be input by multiplying the sample scratch feature map and the training sample, and the training sample is used as the true value.
  • the training sample can be randomly extracted from the sample library and input into the fusion network for training.
  • the network optimizer can include an Adam optimizer, and the initial learning rate is 1e -4 .
  • the loss function of the image restoration network includes:
  • Loss is the contrast difference, is the repaired image corresponding to the training sample, and I gt is the training sample.
  • the training method of the image restoration network may include the following steps:
  • Step 301 Obtain a training data set, where the training data set includes an original image to be repaired and training samples.
  • the original image to be repaired includes scratches, and the training samples are images with complete image content.
  • the training data set may include multiple training samples, and the training samples may be images with high definition and relatively complete image content.
  • the expanded scratch sample includes a plurality of expanded scratches, and any two of the plurality of expanded scratches have different sizes.
  • the plurality of expanded scratches in the expanded scratch sample may be randomly generated rectangular expanded scratches. Since the original scratches in the image to be repaired are usually linear or strip-shaped, the rectangular expanded scratches can have a higher degree of matching with the original scratches, and at the same time, compared with the irregular original scratches, the rectangular expanded scratches are less difficult to repair.
  • Step 302 Obtain an original scratch feature map according to the original image to be repaired.
  • the original scratch feature map can be output through the scratch detection network.
  • Step 303 Process the original scratches in the original scratch feature map into dilated scratches.
  • the target area may be traversed through the detection frame to obtain the dilated scratch, and the target area may include pixel points of the original scratch in the original scratch feature map.
  • Step 304 Cover the expanded scratches on a part of the area in the training sample to obtain a sample scratch enhancement feature map.
  • the sample scratch feature map can be multiplied by the training sample to obtain the sample scratch enhancement feature map.
  • the pixel value of the pixel point of the dilated scratch in the dilated scratch sample is 0, and the pixel value of the pixel point of the dilated scratch in the sample scratch enhancement feature map is 0.
  • the sample scratch feature map can be a matrix including 0 and 1
  • the training sample can be a matrix including multiple pixel values (such as 5, 8, 20, etc.).
  • the sample scratch feature map is multiplied by the training sample, the pixel value of the pixel point multiplied by 0 in the training sample is 0, and the pixel value of the pixel point multiplied by 1 in the training sample remains unchanged, which is equivalent to covering the expanded scratch in the sample scratch feature map at the position of the original scratch in the training sample, and the pixel values of the pixels at the position of the expanded scratch are all 0.
  • Step 305 Input the sample scratch enhancement feature map into the image restoration network to be trained to repair the dilated scratches in the sample scratch enhancement feature map, and obtain a restoration image corresponding to the training sample.
  • the image restoration network is trained by using a sample scratch feature map including dilated scratches.
  • the texture features of the dilated scratches are relatively simple and regular, which can reduce the difficulty of image restoration.
  • the image restoration network in the embodiment of the present application is a network trained based on the sample scratch feature map including dilated scratches.
  • the texture of the dilated scratches is relatively simple, which can make the training difficulty of the image restoration network relatively small.
  • the original scratches can be processed as dilated scratches to improve the adaptability of the original image to be restored and the image restoration network, and at the same time, the universality of the image restoration network can also be improved.
  • Step 306 Compare the repaired image corresponding to the training sample with the training sample to obtain a comparison difference.
  • the contrast difference can be used to indicate the difference between the restored image and the training sample.
  • Step 307 In response to the contrast difference being greater than a preset result, adjusting the image restoration network to be trained based on the contrast difference, and executing the step of processing the original scratch in the original scratch feature map into an expanded scratch.
  • step 307 If the acquired inpainted image is significantly different from the training sample, it means that the parameters in the image inpainting network are not accurate enough, and the parameters in the image inpainting network can be further adjusted through multiple trainings. That is, after step 307, step 301, 302 or 303 is performed.
  • Step 308 In response to the comparison difference being less than or equal to a preset result, the image restoration network to be trained is determined as the image restoration network.
  • the parameters in the image repair network are relatively accurate, and the training of the image repair network can be terminated.
  • FIG. 15 is a structural block diagram of an image restoration device provided in an embodiment of the present application.
  • the image restoration device 1400 includes:
  • the first acquisition module 1410 is used to acquire the original image to be restored.
  • the detection module 1420 is used to perform scratch detection on the original image to be repaired.
  • the second acquisition module 1430 is configured to obtain an original scratch feature map of the original image to be repaired in response to detecting the original scratch in the original image to be repaired.
  • the expansion module 1440 is used to process the original scratch in the original scratch feature map into an expanded scratch to obtain an expanded scratch feature map, where the expanded scratch includes at least one scratch unit, and the original scratch is located in a region where the at least one scratch unit is located.
  • the enhancement module 1450 is used to overlay the expanded scratch in the expanded scratch feature map on the location of the original scratch in the original image to be repaired, so as to obtain a scratch enhancement feature map.
  • the repair module 1460 is used to input the scratch enhancement feature map into the image repair network to repair the dilated scratches in the scratch enhancement feature map to obtain a repaired image corresponding to the original image to be repaired.
  • the image repair network is a network trained based on the sample scratch feature map including the dilated scratches.
  • the expansion module comprises:
  • the third acquisition module is used to acquire a target area, where the target area includes the pixel points of the original scratch in the original scratch feature map.
  • the traversal module is used to traverse the target area through the detection box.
  • the pixel processing module is used to process multiple pixel points in the detection frame into scratch units in response to detecting pixel points with original scratches in the detection frame, and the pixel values of the pixel points in the scratch units are preset values to obtain an expanded scratch feature map.
  • the repair module is used to:
  • the original image features in the scratch enhancement feature map are fused with the dilated scratch features through the image restoration network to repair the dilated scratches in the scratch enhancement feature map and obtain a restoration feature map.
  • the embodiment of the present application provides an image restoration device, which processes the original scratches in the original image to be restored into dilated scratches with a relatively regular shape, and covers the dilated scratches at the positions of the original scratches in the original image to be restored, obtains a scratch enhancement feature map, and then uses an image restoration network to restore the dilated scratches in the scratch enhancement feature map to obtain a restored image corresponding to the original image to be restored. Since the shape of the dilated scratches is relatively regular, the restoration effect of the scratches in the original image can be improved, which solves the problem of poor restoration effect of the restored image in the related art and improves the restoration effect of the restored image.
  • an embodiment of the present application also provides a structural schematic diagram of an electronic device.
  • the electronic device includes one or more processors, a camera component, a memory, and a terminal.
  • the memory may include a random access memory (RAM) and a read-only memory (ROM), and the camera component may be an integrated structure with the terminal.
  • RAM random access memory
  • ROM read-only memory
  • the part of the above-mentioned image acquisition method related to network training can be applied to the server, and the other parts related to image processing except network training can be applied to both the server and the terminal.
  • an embodiment of the present application also provides an image restoration device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement an image restoration method as in any of the above embodiments.
  • an embodiment of the present application also provides a non-volatile computer storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the image restoration method in any of the above embodiments.
  • the embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium.
  • the processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the image restoration method in any of the above embodiments.
  • the disclosed devices and methods can be implemented in other ways.
  • the device embodiments described above are only schematic.
  • the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
  • Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

本申请公开了一种图像修复方法、装置、设备和非瞬态计算机存储介质，属于图像技术领域。所述方法通过将原始待修复图像中的原始划痕处理为形状较为规则的扩张划痕，并将扩张划痕覆盖在原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图，再利用图像修复网络对划痕增强特征图中的扩张划痕进行修复，以得到原始待修复图像对应的修复图像。由于扩张划痕的形状较为规则，进而能够提升对于原始图像中划痕的修复效果，解决了相关技术中修复图像的修复效果较差的问题，提升了修复图像的修复效果。

Description

图像修复方法、装置、设备和非瞬态计算机存储介质 技术领域
本申请涉及图像技术领域，特别涉及一种图像修复方法、装置、设备和非瞬态计算机存储介质。
背景技术
图像修复是指重建图像中丢失或损坏的部分的一种图像处理技术，如，对老照片上的划痕或者瑕疵进行修复。
一种图像修复方法中，先获取原始待修复图像，再标记出原始待修复图像中的原始划痕以及原始图像，得到标记后的中间特征图，并将中间特征图输入图像修复网络，以对原始划痕以及原始图像进行特征融合，得到修复后的图像。
但是，上述方法中，由于照片上的划痕的样式存在较大的差异，进而导致修复图像的修复效果较差。
发明内容
本申请实施例提供了一种图像修复方法、装置、设备和非瞬态计算机存储介质。所述技术方案如下：
根据本申请的一方面，提供了一种图像修复方法，所述方法包括：
获取原始待修复图像；
对所述原始待修复图像进行划痕检测；
响应于检测到所述原始待修复图像中的原始划痕，得到所述原始待修复图像的原始划痕特征图；
将所述原始划痕特征图中的原始划痕处理为扩张划痕，以得到扩张划痕特征图，所述扩张划痕包括至少一个划痕单元，所述原始划痕位于所述至少一个划痕单元所在的区域中；
将所述扩张划痕特征图中的扩张划痕覆盖在所述原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图；
将所述划痕增强特征图输入图像修复网络，以对所述划痕增强特征图中的 扩张划痕进行修复，得到所述原始待修复图像对应的修复图像。
可选地，所述将所述原始划痕特征图中的原始划痕处理为扩张划痕，以得到扩张划痕特征图，包括：
获取目标区域，所述目标区域包括原始划痕特征图中原始划痕的像素点；
通过检测框遍历所述目标区域；
响应于检测到所述检测框中存在所述原始划痕的像素点，将所述检测框中的多个像素点处理为所述划痕单元，所述划痕单元中像素点的像素值为预设值，以得到所述扩张划痕特征图。
可选地，所述检测框为正方形检测框，所述检测框在遍历时的移动步长与边长相等。
可选地，所述划痕增强特征图中包括扩张划痕特征和除所述扩张划痕特征所在区域外的原始图像特征；
所述将所述划痕增强特征图输入图像修复网络，以对所述划痕增强特征图中的扩张划痕进行修复，得到所述原始待修复图像对应的修复图像，包括：
通过所述图像修复网络将所述划痕增强特征图中的原始图像特征与扩张划痕特征进行融合，以对所述划痕增强特征图中的扩张划痕进行修复，得到修复特征图。
可选地，得到所述修复特征图之后，包括：
对所述修复特征图进行维度变换，得到所述原始待修复图像对应的修复图像。
可选地，所述图像修复网络中包括多个特征融合网络；
所述通过所述图像修复网络将所述划痕增强特征图中的原始图像特征与扩张划痕特征进行融合，以对所述划痕增强特征图中的扩张划痕进行修复，得到修复特征图，包括：
对所述划痕增强特征图进行特征提取，得到全局特征图；
通过所述多个特征融合网络对所述全局特征图中的原始图像特征和扩张划痕特征进行融合，得到局部修复特征图；
将所述全局特征图和所述局部修复特征图进行合并，得到所述修复特征图。
可选地，所述多个特征融合网络包括第一特征融合网络、多个第二特征融合网络以及第三特征融合网络，所述第一特征融合网络、所述多个第二特征融合网络以及所述第三特征融合网络中均包括多个注意力网络；
所述通过所述多个特征融合网络对所述全局特征图中的原始图像特征和扩张划痕特征进行融合，得到局部修复特征图，包括：
对所述全局特征图进行下采样处理，得到第一特征图；
通过所述第一特征融合网络对所述第一特征图中的扩张划痕特征和原始图像特征进行融合，以得到第二特征图；
对所述第二特征图进行下采样处理，得到第三特征图；
通过所述多个第二特征融合网络对所述第三特征图中的扩张划痕特征和原始图像特征进行多次融合，以得到第四特征图；
对所述第四特征图进行上采样处理，得到第五特征图；
将所述第一特征图和所述第五特征图进行合并，并输入所述第三特征融合网络，以得到第六特征图；
对所述第六特征图进行上采样处理，得到所述局部修复特征图。
可选地，所述注意力网络包括第一图像增强网络以及第二图像增强网络；所述方法还包括：
将待处理图像输入所述注意力网络；
所述待处理图像输入所述第一图像增强网络得到第一增强图像；
所述待处理图像经过所述第二图像增强网络得到第二增强图像；
将所述第一增强图像与待处理图像进行特征融合获得第一中间图像；
将所述第一中间图像与所述第二增强图像进行特征融合获得特征增强图像；
所述第一图像增强网络包括多个卷积层、多个深度可分离卷积层以及池化层；
所述第二图像增强网络包括一个卷积层以及一个深度可分离卷积层；
所述第一增强图像网络以及所述第二图像增强网络用于扩大待处理图像的感受野，其中，所述第一图像增强网络的感受野扩大能力大于所述第二图像增强网络。
可选地，所述图像修复网络的训练方法，包括：
获取训练数据集，所述训练数据集包括原始待修复图像和训练样本，所述原始待修复图像包括划痕，所述训练样本为图像内容完整的图像；
将所述原始划痕特征图中的原始划痕处理为扩张划痕；
将所述扩张划痕覆盖在所述训练样本中的部分区域处，得到样本划痕增强特征图；
将所述样本划痕增强特征图输入待训练的图像修复网络，以对样本划痕增强特征图中的扩张划痕进行修复，得到所述训练样本对应的修复图像；
将所述训练样本对应的修复图像与所述训练样本进行对比，得到对比差值；
响应于所述对比差值大于预设结果，基于所述对比差值对所述待训练的图像修复网络进行调整，并执行所述将所述原始划痕特征图中的原始划痕处理为扩张划痕的步骤；
响应于所述对比差值小于或等于预设结果，将所述待训练的图像修复网络确定为所述图像修复网络。
可选地，所述扩张划痕样本中包括多个所述扩张划痕，所述多个扩张划痕中任意两个扩张划痕的尺寸不同。
可选地，所述图像修复网络的损失函数包括：
Figure PCTCN2022123334-appb-000001
其中，Loss为所述对比差值，
Figure PCTCN2022123334-appb-000002
为所述训练样本对应的修复图像，I gt为训练样本。
可选地，所述预设值为0；
所述将所述扩张划痕特征图中的扩张划痕覆盖在所述原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图，包括：
将所述扩张划痕特征图和所述原始待修复图像相乘，以得到所述划痕增强特征图，所述得到划痕增强特征图中扩张划痕的像素点的像素值为0。
根据本申请的另一方面，提供了一种图像修复装置，所述装置包括：
第一获取模块，用于获取原始待修复图像；
检测模块，用于对所述原始待修复图像进行划痕检测；
第二获取模块，用于响应于检测到所述原始待修复图像中的原始划痕，得到所述原始待修复图像的原始划痕特征图；
扩张模块，用于将所述原始划痕特征图中的原始划痕处理为扩张划痕，以得到扩张划痕特征图，所述扩张划痕包括至少一个划痕单元，所述原始划痕位于所述至少一个划痕单元所在的区域中；
增强模块，用于将所述扩张划痕特征图中的扩张划痕覆盖在所述原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图；
修复模块，用于将所述划痕增强特征图输入图像修复网络，以对所述划痕增强特征图中的扩张划痕进行修复，得到所述原始待修复图像对应的修复图像。
可选地，所述扩张模块，包括：
第三获取模块，用于获取目标区域，所述目标区域包括原始划痕特征图中原始划痕的像素点；
遍历模块，用于通过检测框遍历所述目标区域；
像素处理模块，用于响应于检测到所述检测框中存在所述原始划痕的像素点，将所述检测框中的多个像素点处理为所述划痕单元，所述划痕单元中像素点的像素值为预设值，以得到所述扩张划痕特征图。
可选地，所述修复模块用于：
通过所述图像修复网络将所述划痕增强特征图中的原始图像特征与扩张划痕特征进行融合，以对所述划痕增强特征图中的扩张划痕进行修复，得到修复特征图。
根据本申请的另一方面，提供了一种图像修复设备，所述图像修复设备包括处理器和存储器，所述存储器中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或指令集由所述处理器加载并执行以实现如上述的图像修复方法。
根据本申请的另一方面，提供了一种非瞬态计算机存储介质，所述非瞬态计算机存储介质中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或指令集由处理器加载并执行以实现如上述的图像修复方法。
根据本申请的另一方面，提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行上述的图像修复方法。
本申请实施例提供的技术方案带来的有益效果至少包括：
提供了一种图像修复方法，通过将原始待修复图像中的原始划痕处理为形状较为规则的扩张划痕，并将扩张划痕覆盖在原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图，再利用图像修复网络对划痕增强特征图中的扩张划痕进行修复，以得到原始待修复图像对应的修复图像。由于扩张划痕的形状较为规则，进而能够提升对于原始图像中划痕的修复效果，解决了相关技术中修复图像的修复效果较差的问题，提升了修复图像的修复效果。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请实施例提供的一种图像修复方法的流程图；
图2是本申请实施例提供的另一种图像修复方法的流程图；
图3是本申请实施例提供的一种划痕检测网络的网络结构示意图；
图4是图3所示的划痕检测网络中的部分模块的网络结构示意图；
图5为本申请实施例提供的一种检测框遍历目标区域的结构示意图；
图6是本申请实施例提供的一种将原始划痕处理为扩张划痕的网络架构图；
图7是本申请实施例提供的一种图像修复的网络架构图；
图8是本申请实施例提供的一种获取修复特征图的流程图；
图9是本申请实施例提供的一种图像修复网络的网络结构示意图；
图10是本申请实施例提供的一种特征融合网络的网络结构示意图；
图11是本申请实施例提供的一种注意力网络的网络结构示意图；
图12是本申请实施例示出的一种图像修复过程示意图；
图13是本申请实施例提供的一种图像修复网络训练的网络架构图；
图14是本申请实施例提供的一种图像修复网络的训练流程图；
图15是本申请实施例提供的一种图像修复装置的结构框图。
通过上述附图，已示出本申请明确的实施例，后文中将有更详细的描述。这些附图和文字描述并不是为了通过任何方式限制本申请构思的范围，而是通过参考特定实施例为本领域技术人员说明本申请的概念。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施方式作进一步地详细描述。
首先对本申请实施例所涉及的应用场景进行介绍。
图像修复技术是指通过恢复残缺图像中损坏部分的像素特征，来获得完整的图像的一种图像处理技术。如，图像修复技术可以用于去除图像中不需要的 目标、恢复图像中破损的部分等。图像修复技术可以包括基于机器学习的图像修复方法。
机器学习是人工智能的一个分支，机器学习是指使用计算机作为工具，并从大数据中学习现实世界中各类事物能直接用于计算机计算的表示形式。在图像技术领域，机器学习可以用于目标检测、图像生成和图像分割等方面。人工智能是指研究各种智能机器的设计原理与实现方法，使机器具有感知、推理与决策功能的一种技术。
示例性的，随着时间流逝，照片可能会受到损坏，导致年限较长的照片变得破旧、模糊，如，照片上存在多个划痕，导致照片不够完整清晰。图像处理技术可以对受到损坏的照片进行修复，以得到较为清晰完整的照片。
需要说明的是，本申请实施例描述的应用场景是为了更加清楚地说明本申请实施例的技术方案，并不构成对于本申请实施例提供的技术方案的限定，本领域普通技术人员可知，本申请实施例提供的技术方案对于类似的技术问题，同样适用。
该实施环境可以包括照片、拍摄组件、服务器以及显示终端。拍摄组件可以包括相机，该相机可以用于将真实的照片拍摄为数字化图像。服务器包括处理器，服务器可以与拍摄组件之间建立有线或无线的连接，以根据拍摄组件采集的图像生成修复图像，并在显示终端上显示该修复图像。
图1是本申请实施例提供的一种图像修复方法的流程图。该方法可以应用于上述实施环境的服务器中。该方法可以包括下面几个步骤：
步骤101、获取原始待修复图像。
步骤102、对原始待修复图像进行划痕检测。
步骤103、响应于检测到原始待修复图像中的原始划痕，得到原始待修复图像的原始划痕特征图。
步骤104、将原始划痕特征图中的原始划痕处理为扩张划痕，以得到扩张划痕特征图。
其中，扩张划痕包括至少一个划痕单元，原始划痕位于至少一个划痕单元所在的区域中。
步骤105、将扩张划痕特征图中的扩张划痕覆盖在原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图。
步骤106、将划痕增强特征图输入图像修复网络，以对划痕增强特征图中的扩张划痕进行修复，得到原始待修复图像对应的修复图像。
综上所述，本申请实施例提供了一种图像修复方法，通过将原始待修复图像中的原始划痕处理为形状较为规则的扩张划痕，并将扩张划痕覆盖在原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图，再利用图像修复网络对划痕增强特征图中的扩张划痕进行修复，以得到原始待修复图像对应的修复图像。由于扩张划痕的形状较为规则，进而能够提升对于原始图像中划痕的修复效果，解决了相关技术中修复图像的修复效果较差的问题，提升了修复图像的修复效果。
图2是本申请实施例提供的另一种图像修复方法的流程图。该方法可以应用于上述实施环境的服务器中。该方法可以包括下面几个步骤：
步骤201、获取原始待修复图像。
该原始待修复图像可以包括具有图像修复需求的图像，即原始待修复图像中可以包括原始图像以及丢失或者损坏的部分(如，原始划痕)。该原始待修复图像也可以包括不需要图像修复的内容较为完整的图像。
在一种可能的实现方式中，原始待修复图像可以是终端设备通过摄像头拍摄到照片得到的图像；或者，原始待修复图像还可以是从终端设备内部获得的图像，示例性的，该原始待修复图像可以为终端设备的相册中存储的图像，或者，终端设备从云端获取的图像。
步骤202、对原始待修复图像进行划痕检测。
可以通过划痕检测网络对原始待修复图像进行划痕检测，以辨别上述原始待修复图像中是否存在原始划痕，若原始待修复图中存在原始划痕，可以通过划痕检测网络检测出原始图像中的原始划痕的位置以及形状等信息。若原始待修复图中不存在原始划痕，表明该原始待修复图像的清晰度和完整程度较好，则可以直接输出该原始待修复图像为修复图像。
该划痕检测网络可以包括YOLOv5检测网络，由于YOLO系列算法具有比较简洁的结构，且计算处理速度较快，因此能够广泛应用于检测相关的图像处理过程中。如图3和图4所示，图3是本申请实施例提供的一种划痕检测网络的网络结构示意图，图4是图3所示的划痕检测网络中的部分模块的网络结构示意图。从图3中可以看出，YOLOv5检测网络可以包括输入端、主干网络 (Backbone)、脖子网络(Neck)以及输出端。
其中，Foc(Focus)表示聚焦网络，可以用于减少YOLOv5检测网络的计算量，增加计算速度；Conv表示卷积层；Con(Concat)表示矩阵的合并或排列运算；Lr(Leaky relu)表示激活函数；CBL由卷积层(Conv)、批量标准化处理(BN)和激活函数(Leaky relu)组成，其中，批量标准化处理(BN)也可以称为归一化批处理。SPP由多个池化层(maxpool)和Concat组成。CSP1-x表示第一检测网络，其中，x可以为1、2或者3等；CSP2-x表示第二检测网络，其中，x可以为1、2或者3等。Res unit表示残差组件，a(add)表示矩阵的相加运算，sl(slice)表示数据切片。
示例性的，图4中的Focus层可以通过数据切片(slice)操作将较高分辨率的图像(特征图)拆分成多个较低分辨率的图像(特征图)，即就是，将较高分辨率的图像进行隔列采样后，再进行拼接。如，将原始的640×640×3的图像输入Focus层，通过数据切片(slice)处理，将图像分成共4部分，即将图像处理为320×320×12的特征图，并将该特征图进行拼接(Concat)，再经过一次卷积(CBL)处理，最终变成320×320×64的特征图。通过Focus层处理特征图，可以减少信息损失。
图4中的SPP层，通过对同一个图像通过多个池化核进行最大池化处理，得到多个特征图，再将该多个特征图以及未经最大池化处理的特征图进行融合，其中，该多个池化核中任意两个池化核的大小不同。SPP层通过将不同大小特征进行融合，以使得划痕检测网络能够适用于原始待修复图像中划痕大小差异较大的情况。
划痕检测网络可以是一个预先训练的神经网络；训练数据可以包括样本图像以及划痕图像，划痕图像是指在样本图像上覆盖划痕的图像。示例性的，样本图像可以指样本数据集中清晰度较高、图像内容较为完整的图像，划痕图像可以是指在样本图像上覆盖不同形状的划痕的图像。需要说明的是，本申请实施例中的划痕检测网络也可以为YOLOv4检测网络等其他检测网络。
步骤203、响应于检测到原始待修复图像中的原始划痕，得到原始待修复图像的原始划痕特征图。
当原始待修复图像中具有原始划痕时，可以通过划痕检测网络输出原始划痕特征图。具体地，可以将原始待修复图像中原始划痕(Mask)区域内的像素点的像素值置为0，并将原始待修复图像中原始划痕(Mask)区域外的像素点 的像素值置为1，该原始划痕(Mask)区域外的区域可以称作原始图像区域，以得到原始划痕特征图，原始划痕特征图可以为一个矩阵，该矩阵中包括0和1。
步骤204、获取目标区域。
其中，目标区域可以包括原始划痕特征图中原始划痕的像素点。
原始划痕特征图中可以具有多个原始划痕，可以分别获取原始划痕特征图中的每一个原始划痕的像素点所在的区域。示例性的，可以基于原始划痕特征图建立坐标系，获取原始划痕特征图中的多个原始划痕的像素点的坐标，进而获取多个原始划痕的像素点所在的目标区域。
示例性的，如图5所示，图5为本申请实施例提供的一种检测框遍历目标区域的结构示意图。该原始划痕特征图可以为矩形，以原始划痕特征图的一个顶点为(0，0)点建立二维坐标系(x，y)，获取原始划痕R1中的多个像素点的坐标，以得到原始划痕R1中的第一像素点c1、第二像素点c2、第三像素点c3和，该第一像素点c1的x坐标值和第二像素点c2的x坐标值分别为原始划痕中的多个像素点的x坐标值的最大值和最小值，第三像素点c3的y坐标值和第四像素点c4的y坐标值分别为原始划痕R1中的多个像素点的y坐标值的最大值和最小值。即就是，通过获取原始划痕R1在x方向和y方向上的四个顶点，并基于这四个顶点，获取原始划痕特征图中原始划痕R1的像素点所在的目标区域。
步骤205、通过检测框遍历目标区域。
该检测框可以用于在遍历目标区域的过程中检测检测框中是否存在原始划痕的像素点。可选地，该检测框可以为矩形检测框。
可选地，检测框a1为正方形检测框，检测框a1在遍历时的移动步长与边长相等。如此，可以提高检测框的检测效率。示例性的，参考图5，该检测框的边长可以为4，在遍历时的移动步长也为4。
步骤206、响应于检测到检测框中存在原始划痕的像素点，将检测框中的多个像素点处理为划痕单元，划痕单元中像素点的像素值为预设值，以得到扩张划痕特征图。
图6是本申请实施例提供的一种将原始划痕处理为扩张划痕的网络架构图，请参考图6。如此，可以将原始划痕特征图中的原始划痕处理为扩张划痕，能够将不规则的原始划痕处理为较为规则的扩张划痕，以得到扩张划痕特征图。其中，扩张划痕包括至少一个划痕单元，原始划痕位于至少一个划痕单元所在的 区域中。可选地，该划痕单元可以为矩形的划痕单元，可以提高扩张划痕的形状的规则程度。
示例性的，请参考图5，该预设值可以为0，若检测到检测框中存在原始划痕的像素点，将检测框中的多个像素点的像素值均赋值为0，即就是，划痕单元中像素点的像素值为0，以得到原始划痕R1对应的扩张划痕R2，进而得到扩张划痕特征图。
步骤207、将扩张划痕特征图中的扩张划痕覆盖在原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图。
可以将扩张划痕特征图和原始待修复图像相乘，以得到划痕增强特征图，其中，由于上述预设值可以为0，则扩张划痕特征图中的扩张划痕中的像素点的像素值也为0，因此，得到划痕增强特征图中扩张划痕的像素点的像素值为0。
示例性的，扩张划痕特征图可以为一个包括0和1的矩阵，原始待修复图像可以为一个包括多个像素值(如，5、8、20等)的矩阵，当扩张划痕特征图和原始待修复图像相乘后，原始待修复图像中与0相乘的像素点的像素值为0，原始待修复图像中与1相乘的像素点的像素值不变，相当于将扩张划痕特征图中的扩张划痕覆盖在原始待修复图像中的原始划痕所在位置处，扩张划痕所在位置处的像素点的像素值均为0。
步骤208、获取图像修复网络。
图像修复网络可以为经过训练后的融合网络，该图像修复网络可以包括多个卷积层以及多个特征融合网络，该多个特征融合网络可以用于原始图像特征与扩张划痕特征融合，以对划痕增强特征图中的扩张划痕进行填充。
即就是，可以通过图像修复网络将划痕增强特征图中的原始图像特征与扩张划痕特征进行融合，以对划痕增强特征图中的扩张划痕进行修复，得到修复特征图。其中，多个特征融合网络也可以是经过训练得到的网络。
步骤209、将划痕增强特征图输入图像修复网络，通过图像修复网络将划痕增强特征图中的原始图像特征与扩张划痕特征进行融合，以对划痕增强特征图中的扩张划痕进行修复，得到修复特征图。
图7是本申请实施例提供的一种图像修复的网络架构图，请参考图7。其中，图像修复网络是根据包括扩张划痕的样本划痕特征图训练得到的网络。可选地，划痕增强特征图中包括扩张划痕特征和除扩张划痕特征所在区域外的原始图像特征。
由于原始划痕特征图中的原始划痕的纹理特征表现为精细化、复杂化且不规则，相关技术中的图像修复网络对复杂的原始划痕的修复精度较低，生成的原始划痕的填充内容不精确，致使部分尺寸较小或者纹理复杂的原始划痕的修复不够完整。并且，相关技术中根据包括原始划痕的样本划痕图像，对图像修复网络进行训练，由于原始划痕的纹理复杂多变，导致图像修复网络的训练难度较大。
本申请实施例中，通过包括扩张划痕的样本划痕特征图对图像修复网络进行训练，在修复原始待修复图像的过程中，首先通过划痕检测和划痕扩张，将原始划痕特征图中的原始划痕处理为扩张划痕，以得到扩张划痕特征图，然后将扩张划痕特征图中的扩张划痕覆盖在原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图，以使得划痕增强特征图中的扩张划痕，相较于原始划痕，纹理特征较为简单和规则，可以降低图像修复难度。并且，本申请实施例中的图像修复网络是根据包括扩张划痕的样本划痕特征图训练得到的网络，扩张划痕的纹理较为简单，可以使得图像修复网络的训练难度较小。进一步的，可以通过将原始划痕处理为扩张划痕，提高待修复原始图像与图像修复网络的适配性，同时，还可以提高图像修复网络的普适性。
如图8所示，步骤209可以包括以下3个子步骤：
子步骤2091、对划痕增强特征图进行特征提取，得到全局特征图。
所述特征提取处理可以为下采样等方式，在一可选实施例中：
对划痕增强特征图进行下采样处理，可以降低划痕增强特征图的维度并保留有效信息，可以避免过拟合现象。
子步骤2092、通过多个特征融合网络对全局特征图中的原始图像特征和扩张划痕特征进行融合，得到局部修复特征图。
图9是本申请实施例提供的一种图像修复网络的网络结构示意图，请参考图9。可选地，多个特征融合网络可以包括第一特征融合网络(FFG1)、多个第二特征融合网络(FFG2)以及第三特征融合网络(FFG3)，第一特征融合网络(FFG1)、多个第二特征融合网络(FFG2)以及第三特征融合网络(FFG3)中均包括多个注意力网络(MAB)。其中，注意力网络(MAB)也可以是经过训练得到的网络。其中，Conv表示卷积层，SConv表示跨步卷积，此处跨步卷积的步长为2，即就是，图像特征图经过跨步卷积后缩小了2倍，在后续的图像处理过程中，可以通过上采样(Up-sample)处理，以恢复图像特征图。
图10是本申请实施例提供的一种特征融合网络的网络结构示意图，请参考图10。第一特征融合网络、多个第二特征融合网络以及第三特征融合网络的结构可以相同，均可以称为特征融合网络(英文：Feature Fusion Group；简写：FFG)。特征融合网络中可以包括多个注意力网络(英文：Multi-attention Block；简写：MAB)。其中，Conv表示卷积层，Con表示矩阵的合并或排列运算，C Shu表示Channel Shuffle，用于来混合连接通道之间的信息，卷积层可以为1×1的卷积层，1x1卷积层可以用于对特征图的通道进行缩减。
图11是本申请实施例提供的一种注意力网络的网络结构示意图，请参考图11，其中，Conv表示卷积层，SConv表示跨步卷积，Concat表示矩阵的合并或排列运算，DConv表示空洞卷积，DWConv表示深度卷积，S表示Sigmoid激活函数，Sigmoid激活函数是一种逻辑激活函数，也称为S型生长曲线，Sigmoid函数可以被用作神经网络的激活函数，以将变量映射到[0，1]之间。经过初步提取的特征图输入进注意力网络中，分三个支路进行不同处理，上分支经过一个1x1的卷积，一个5x5的深度可分离卷积，以及一个Sigmoid激活函数，中间分支首先经过两个3x3的卷积以及一个1x1的卷积，一个步长为2的3x3卷积，最大池化，后接并行的两个3x3深度可分离卷积，相加后进行上采样，再通过1x1的卷积以及一个Sigmoid激活函数，下分支其实就是一个跳连接到与中间分支的输出相乘，随后结果与上分支的输出再次相乘，得到最后的输出。Sigmoid激活函数可以将变量映射到(0,1)的区间，数据在传递的过程中不容易发散。空洞卷积可以扩大感受野，有助于恢复过饱和区域和运动错位导致的图像细节缺失。需要说明的是，本申请实施例中的注意力网络的结构可以为如图11所示的注意力网络，还可以为其他结构的注意力网络，本申请实施例对此不进行限制。
在划痕增强特征图中的原始图像特征与扩张划痕特征融合的过程中，将注意力网络引入图像修复的工作中，注意力网络可以不仅利用扩张划痕所在的区域周围的信息，还能增强并利用整张图片的全局信息中对修复有利的特征，以使得修复后的图像中的结构和纹理更加清晰连贯。
在一种示例性的实施方式中，请参考图11，注意力网络可以包括第一图像增强网络以及第二图像增强网络，通过注意力网络对图像进行处理可以包括以下几个步骤：
1)将待处理图像输入注意力网络。
该待处理图像可以为本申请实施例中在通过特征网络处理图像的过程中，输入注意力网络中的图像特征，由于图像修复网络中包括多个注意力网络，因此还待处理图像可以为不同的图像特征，并不特指某一个注意力网络中待处理图像。该注意力网络可以包括第一图像增强网络以及第二图像增强网络。
2)待处理图像输入第一图像增强网络得到第一增强图像。
第一图像增强网络可以包括多个卷积层、多个深度可分离卷积层以及池化层。第一增强图像网络用于扩大待处理图像的感受野。在待处理图像输入注意力网络之后，该待处理图像可以输入注意力网络中的第一图像增强网络，以通过第一图像增强网络对图像进行处理，得到第一增强图像。
3)待处理图像经过第二图像增强网络得到第二增强图像。
第二图像增强网络可以包括一个卷积层以及一个深度可分离卷积层。第二图像增强网络用于扩大待处理图像的感受野，其中，第一图像增强网络的感受野扩大能力大于第二图像增强网络。在待处理图像输入注意力网络之后，该待处理图像可以输入注意力网络中的第二图像增强网络，以通过第二图像增强网络对图像进行处理，得到第一增强图像。
4)将第一增强图像与待处理图像进行特征融合获得第一中间图像。
注意力网络可以将通过第一图像增强网络得到第一增强图像，和未经注意力网络处理的待处理图像进行融合，以使得第一中间图像的细节较为丰富。
5)将第一中间图像与第二增强图像进行特征融合获得特征增强图像。
注意力网络还可以将通过第二图像增强网络得到第二增强图像，和第一中间图像进行融合，以使得第一中间图像的细节较为丰富。如此，可以充分利用来自不同卷积层的信息，从而使获得的特征增强图像中保留更多的细节，有助于恢复待处理图像中的划痕区域的细节。
上述通过注意力网络对图像进行修复的方法可以应用于本申请实施例中的划痕增强特征图中的原始图像特征与扩张划痕特征融合的过程中，即可以应用于多个特征融合网络的每一个特征融合网络中，如图10所示，特征增强图像可以为特征融合网络中的任意一个注意力网络的输出结果。
进一步的，在本申请的一个实施例中：通过多个特征融合网络对全局特征图中的原始图像特征和扩张划痕特征进行融合，得到局部修复特征图，可以包括以下几个步骤：
1)对全局特征图进行下采样处理，得到第一特征图。
可以对全局特征图中对修复扩张划痕有利的特征进行提取，以得到第一特征图。
2)通过第一特征融合网络对第一特征图中的扩张划痕特征和原始图像特征进行融合，以得到第二特征图。
第一特征融合网络可以对扩张划痕特征以及原始图像特征进行第一次融合，多个特征融合网络在对扩张划痕特征以及原始图像特征进行融合时的侧重可以不同。示例性的，第一特征融合网络侧重于对扩张划痕所在的区域中的颜色进行修复。
3)对第二特征图进行下采样处理，得到第三特征图。
4)通过多个第二特征融合网络对第三特征图中的扩张划痕特征和原始图像特征进行多次融合，以得到第四特征图。
多个第二特征融合网络可以对扩张划痕特征以及原始图像特征进行第二次融合。示例性的，第二特征融合网络侧重于对扩张划痕所在的区域中的纹理进行修复。
5)对第四特征图进行上采样处理，得到第五特征图。
上采样可以包括重组像素(pixel shuffer)的图像处理方式。
6)将第一特征图和第五特征图进行合并，并输入第三特征融合网络，以得到第六特征图。
可以将经过不同的侧重的融合网络融合后的图像特征图进行合并，并通过第三特征融合网络再次对扩张划痕特征以及原始图像特征进行融合，以使得融合后的图像特征中的细节较为丰富。
需要说明的是，本申请实施例中的第一特征图和第五特征图的本质都是一个矩阵，在本申请的实施例中，第一特征图和第五特征图的合并，本质上是两个矩阵的合并，是在不改变两个矩阵本身顺序的前提下，将两个矩阵进行排列或合并的过程。
7)对第六特征图进行上采样处理，得到局部修复特征图。
子步骤2093、将全局特征图和局部修复特征图进行合并，得到修复特征图。
可以将上述的全局特征图和局部修复特征图进行合并，以使得修复特征图具有较多的图像细节。这样的图像修复网络能够对原始待修复图像中的局部和全局特征都能有所注意，可以提升网络的修复性能。
步骤210、对修复特征图进行维度变换，得到原始待修复图像对应的修复图 像。
所述维度变换可以为上采样或者下采样等方式，在一可选实施例中：
上述修复特征图中可能存在冗余信息，可以采用两个3x3的卷积与激活函数的组合，对重要特征图进行降维处理，以得到原始待修复图像对应的修复图像。如图12所示，图12是本申请实施例示出的一种图像修复过程示意图。本申请实施例中，通过将原始待修复图像中的原始划痕处理为扩张划痕，并将扩张划痕覆盖在原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图，再利用图像修复网络对划痕增强特征图中的扩张划痕进行修复，以得到原始待修复图像对应的修复图像。并且，基于扩张划痕的样本划痕特征图对图像修复网络进行训练，可得到修复能力较强，普适性较好的图像修复网络，使用该图像修复网络对图像进行修复，修复效率较高且修复的效果较好。
需要说明的是，上述步骤仅用于对本申请实施例的一种解释性说明，本领域技术人员可以对上述步骤进行删减，应注意，上述方法中各个操作的序号仅作为该操作的表示以便描述，而不应被看作表示该各个操作的执行顺序。除非明确指出，否则该方法不需要完全按照所示顺序来执行。
综上所述，本申请实施例提供了一种图像修复方法，通过将原始待修复图像中的原始划痕处理为形状较为规则的扩张划痕，并将扩张划痕覆盖在原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图，再利用图像修复网络对划痕增强特征图中的扩张划痕进行修复，以得到原始待修复图像对应的修复图像。由于扩张划痕的形状较为规则，进而能够提升对于原始图像中划痕的修复效果，解决了相关技术中修复图像的修复效果较差的问题，提升了修复图像的修复效果。
可选地，上述步骤208中的图像修复网络可以为提前训练好的图像修复网络，也可以在步骤208中对图像修复网络进行训练。需要说明的是，本申请实施例中所应用的各个网络(划痕检测网络、第一特征融合网络、多个第二特征融合网络以及第三特征融合网络以及注意力网络)均是经过训练的网络结构，可以通过深度学习对各个网络进行训练，深度学习为机器学习的一种方法。本申请实施例不限制这些网络的训练方式。
图13是本申请实施例提供的一种图像修复网络训练的网络架构图，请参考图13。图像修复网络的训练可以样本划痕特征图和训练样本相乘后作为输入， 以训练样本作为真值，在训练的过程中，可以随机从样本库中抽取训练样本输入融合网络进行训练，网络优化器可以包括Adam优化器，初始学习率为1e -4，图像修复网络的损失函数包括：
Figure PCTCN2022123334-appb-000003
其中，Loss为对比差值，
Figure PCTCN2022123334-appb-000004
为训练样本对应的修复图像，I gt为训练样本。
如图14所示，本申请实施例中，图像修复网络的训练方法可以包括下面几个步骤：
步骤301、获取训练数据集，训练数据集包括原始待修复图像和训练样本，原始待修复图像包括划痕，训练样本为图像内容完整的图像。
其中，训练数据集中可以包括多个训练样本，该训练样本可以为清晰度较高，图像内容较为完整的图像。
可选地，扩张划痕样本中包括多个扩张划痕，多个扩张划痕中任意两个扩张划痕的尺寸不同。扩张划痕样本中的多个扩张划痕可以为随机生成的矩形的扩张划痕。由于待修复图像中的原始划痕通常为线状或者条状，因此，矩形的扩张划痕可以与原始划痕的匹配度较高，同时，相较于不规则的原始划痕，矩形的扩张划痕的修复难度较低。
步骤302、根据原始待修复图像，获得原始划痕特征图。
当原始待修复图像中具有原始划痕时，可以通过划痕检测网络输出原始划痕特征图。
步骤303、将原始划痕特征图中的原始划痕处理为扩张划痕。
可以通过检测框遍历目标区域，以获取扩张划痕，该目标区域可以包括原始划痕特征图中原始划痕的像素点。
步骤304、将扩张划痕覆盖在训练样本中的部分区域处，得到样本划痕增强特征图。
可以将样本划痕特征图与训练样本相乘，以得到样本划痕增强特征图。扩张划痕样本中扩张划痕的像素点的像素值为0，得到样本划痕增强特征图中扩张划痕的像素点的像素值为0。
示例性的，样本划痕特征图可以为一个包括0和1的矩阵，训练样本可以为一个包括多个像素值(如，5、8、20等)的矩阵，当样本划痕特征图与训练样本相乘后，训练样本中与0相乘的像素点的像素值为0，训练样本中与1相乘的像素点的像素值不变，相当于将样本划痕特征图中的扩张划痕覆盖在训练样本 中的原始划痕所在位置处，扩张划痕所在位置处的像素点的像素值均为0。
步骤305、将样本划痕增强特征图输入待训练的图像修复网络，以对样本划痕增强特征图中的扩张划痕进行修复，得到训练样本对应的修复图像。
通过包括扩张划痕的样本划痕特征图对图像修复网络进行训练，在训练样本的过程中，相较于相关技术中的原始划痕，扩张划痕的纹理特征较为简单和规则，可以降低图像修复难度。并且，本申请实施例中的图像修复网络是根据包括扩张划痕的样本划痕特征图训练得到的网络，扩张划痕的纹理较为简单，可以使得图像修复网络的训练难度较小。进一步的，在后续的图像修复过程中，可以通过将原始划痕处理为扩张划痕，提高待修复原始图像与图像修复网络的适配性，同时，还可以提高图像修复网络的普适性。
步骤306、将训练样本对应的修复图像与训练样本进行对比，得到对比差值。
该对比差值可以用于表示修复图像和训练样本的差异程度。
步骤307、响应于对比差值大于预设结果，基于对比差值对待训练图像修复网络进行调整，并执行将原始划痕特征图中的原始划痕处理为扩张划痕的步骤。
若获取的修复图像与训练样本相比，差异较大，则可以说明该图像修复网络中的参数不够准确，可以通过多次训练，进一步调整图像修复网络中的参数。即在步骤307后执行步骤301、302或者303。
步骤308、响应于对比差值小于或等于预设结果，将待训练的图像修复网络确定为图像修复网络。
若获取的修复图像与训练样本相比，差异较小，则可以说明该图像修复网络中的参数较为准确，可以结束图像修复网络的训练。
图15是本申请实施例提供的一种图像修复装置的结构框图，该图像修复装置1400包括：
第一获取模块1410，用于获取原始待修复图像。
检测模块1420，用于对原始待修复图像进行划痕检测。
第二获取模块1430，用于响应于检测到原始待修复图像中的原始划痕，得到原始待修复图像的原始划痕特征图。
扩张模块1440，用于将原始划痕特征图中的原始划痕处理为扩张划痕，以得到扩张划痕特征图，扩张划痕包括至少一个划痕单元，原始划痕位于至少一个划痕单元所在的区域中。
增强模块1450，用于将扩张划痕特征图中的扩张划痕覆盖在原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图。
修复模块1460，用于将划痕增强特征图输入图像修复网络，以对划痕增强特征图中的扩张划痕进行修复，得到原始待修复图像对应的修复图像，图像修复网络是根据包括扩张划痕的样本划痕特征图训练得到的网络。
可选地，扩张模块，包括：
第三获取模块，用于获取目标区域，目标区域包括原始划痕特征图中原始划痕的像素点。
遍历模块，用于通过检测框遍历目标区域。
像素处理模块，用于响应于检测到检测框中存在原始划痕的像素点，将检测框中的多个像素点处理为划痕单元，划痕单元中像素点的像素值为预设值，以得到扩张划痕特征图。
可选地，修复模块用于：
通过图像修复网络将划痕增强特征图中的原始图像特征与扩张划痕特征进行融合，以对划痕增强特征图中的扩张划痕进行修复，得到修复特征图。
综上所述，本申请实施例提供了一种图像修复装置，通过将原始待修复图像中的原始划痕处理为形状较为规则的扩张划痕，并将扩张划痕覆盖在原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图，再利用图像修复网络对划痕增强特征图中的扩张划痕进行修复，以得到原始待修复图像对应的修复图像。由于扩张划痕的形状较为规则，进而能够提升对于原始图像中划痕的修复效果，解决了相关技术中修复图像的修复效果较差的问题，提升了修复图像的修复效果。
此外，本申请实施例还提供了一种电子设备的结构示意图。该电子设备包括一个或多个处理器、摄像组件、存储器以及终端。存储器可以包括随机存取存储器(random access memory，RAM)和只读存储器(read only memory，ROM)，摄像组件可以和终端为一体结构。上述图像获取方法中关于网络训练的部分可以应用于服务器，除网络训练的其他关于图像处理的部分既可以应用于服务器，也可以应用于终端。
此外，本申请实施例还提供了一种图像修复设备，图像修复设备包括处理 器和存储器，存储器中存储有至少一条指令、至少一段程序、代码集或指令集，至少一条指令、至少一段程序、代码集或指令集由处理器加载并执行以实现如上述任一实施例中的图像修复方法。
此外，本申请实施例还提供了一种非瞬态计算机存储介质，非瞬态计算机存储介质中存储有至少一条指令、至少一段程序、代码集或指令集，至少一条指令、至少一段程序、代码集或指令集由处理器加载并执行以实现上述任一实施例中的图像修复方法。
此外，本申请实施例还提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行上述任一实施例中的图像修复方法。
在本申请中，术语“第一”、“第二”、“第三”和“第四”仅用于描述目的，而不能理解为指示或暗示相对重要性。术语“多个”指两个或两个以上，除非另有明确的限定。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
本领域普通技术人员可以理解实现上述实施例的全部或部分步骤可以通过 硬件来完成，也可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，上述提到的存储介质可以是只读存储器，磁盘或光盘等。
以上所述仅为本申请的可选实施例，并不用以限制本申请，凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (17)

  1. 一种图像修复方法，其特征在于，所述方法包括：
    获取原始待修复图像；
    对所述原始待修复图像进行划痕检测；
    响应于检测到所述原始待修复图像中的原始划痕，得到所述原始待修复图像的原始划痕特征图；
    将所述原始划痕特征图中的原始划痕处理为扩张划痕，以得到扩张划痕特征图，所述扩张划痕包括至少一个划痕单元，所述原始划痕位于所述至少一个划痕单元所在的区域中；
    将所述扩张划痕特征图中的扩张划痕覆盖在所述原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图；
    将所述划痕增强特征图输入图像修复网络，以对所述划痕增强特征图中的扩张划痕进行修复，得到所述原始待修复图像对应的修复图像。
  2. 根据权利要求1所述的方法，其特征在于，所述将所述原始划痕特征图中的原始划痕处理为扩张划痕，以得到扩张划痕特征图，包括：
    获取目标区域，所述目标区域包括原始划痕特征图中原始划痕的像素点；
    通过检测框遍历所述目标区域；
    响应于检测到所述检测框中存在所述原始划痕的像素点，将所述检测框中的多个像素点处理为所述划痕单元，所述划痕单元中像素点的像素值为预设值，以得到所述扩张划痕特征图。
  3. 根据权利要求2所述的方法，其特征在于，所述检测框为正方形检测框，所述检测框在遍历时的移动步长与边长相等。
  4. 根据权利要求1所述的方法，其特征在于，所述划痕增强特征图中包括扩张划痕特征和除所述扩张划痕特征所在区域外的原始图像特征；
    所述将所述划痕增强特征图输入图像修复网络，以对所述划痕增强特征图中的扩张划痕进行修复，得到所述原始待修复图像对应的修复图像，包括：
    通过所述图像修复网络将所述划痕增强特征图中的原始图像特征与扩张划 痕特征进行融合，以对所述划痕增强特征图中的扩张划痕进行修复，得到修复特征图。
  5. 根据权利要求4所述的方法，其特征在于，得到所述修复特征图之后，包括：
    对所述修复特征图进行维度变换，得到所述原始待修复图像对应的修复图像。
  6. 根据权利要求4所述的方法，其特征在于，所述图像修复网络中包括多个特征融合网络；
    所述通过所述图像修复网络将所述划痕增强特征图中的原始图像特征与扩张划痕特征进行融合，以对所述划痕增强特征图中的扩张划痕进行修复，得到修复特征图，包括：
    对所述划痕增强特征图进行特征提取，得到全局特征图；
    通过所述多个特征融合网络对所述全局特征图中的原始图像特征和扩张划痕特征进行融合，得到局部修复特征图；
    将所述全局特征图和所述局部修复特征图进行合并，得到所述修复特征图。
  7. 根据权利要求6所述的方法，其特征在于，所述多个特征融合网络包括第一特征融合网络、多个第二特征融合网络以及第三特征融合网络，所述第一特征融合网络、所述多个第二特征融合网络以及所述第三特征融合网络中均包括多个注意力网络；
    所述通过所述多个特征融合网络对所述全局特征图中的原始图像特征和扩张划痕特征进行融合，得到局部修复特征图，包括：
    对所述全局特征图进行下采样处理，得到第一特征图；
    通过所述第一特征融合网络对所述第一特征图中的扩张划痕特征和原始图像特征进行融合，以得到第二特征图；
    对所述第二特征图进行下采样处理，得到第三特征图；
    通过所述多个第二特征融合网络对所述第三特征图中的扩张划痕特征和原始图像特征进行多次融合，以得到第四特征图；
    对所述第四特征图进行上采样处理，得到第五特征图；
    将所述第一特征图和所述第五特征图进行合并，并输入所述第三特征融合网络，以得到第六特征图；
    对所述第六特征图进行上采样处理，得到所述局部修复特征图。
  8. 根据权利要求7所述的方法，其特征在于，所述注意力网络包括第一图像增强网络以及第二图像增强网络；所述方法还包括：
    将待处理图像输入所述注意力网络；
    所述待处理图像输入所述第一图像增强网络得到第一增强图像；
    所述待处理图像经过所述第二图像增强网络得到第二增强图像；
    将所述第一增强图像与待处理图像进行特征融合获得第一中间图像；
    将所述第一中间图像与所述第二增强图像进行特征融合获得特征增强图像；
    所述第一图像增强网络包括多个卷积层、多个深度可分离卷积层以及池化层；
    所述第二图像增强网络包括一个卷积层以及一个深度可分离卷积层；
    所述第一增强图像网络以及所述第二图像增强网络用于扩大待处理图像的感受野，其中，所述第一图像增强网络的感受野扩大能力大于所述第二图像增强网络。
  9. 根据权利要求1所述的方法，其特征在于，所述图像修复网络的训练方法，包括：
    获取训练数据集，所述训练数据集包括原始待修复图像和训练样本，所述原始待修复图像包括划痕，所述训练样本为图像内容完整的图像；
    将所述原始划痕特征图中的原始划痕处理为扩张划痕；
    将所述扩张划痕覆盖在所述训练样本中的部分区域处，得到样本划痕增强特征图；
    将所述样本划痕增强特征图输入待训练的图像修复网络，以对样本划痕增强特征图中的扩张划痕进行修复，得到所述训练样本对应的修复图像；
    将所述训练样本对应的修复图像与所述训练样本进行对比，得到对比差值；
    响应于所述对比差值大于预设结果，基于所述对比差值对所述待训练的图 像修复网络进行调整，并执行所述将所述原始划痕特征图中的原始划痕处理为扩张划痕的步骤；
    响应于所述对比差值小于或等于预设结果，将所述待训练的图像修复网络确定为所述图像修复网络。
  10. 根据权利要求9所述的方法，其特征在于，所述扩张划痕样本中包括多个所述扩张划痕，所述多个扩张划痕中任意两个扩张划痕的尺寸不同。
  11. 根据权利要求9所述的方法，其特征在于，所述图像修复网络的损失函数包括：
    Figure PCTCN2022123334-appb-100001
    其中，Loss为所述对比差值，
    Figure PCTCN2022123334-appb-100002
    为所述训练样本对应的修复图像，I gt为训练样本。
  12. 根据权利要求2所述的方法，其特征在于，所述预设值为0；
    所述将所述扩张划痕特征图中的扩张划痕覆盖在所述原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图，包括：
    将所述扩张划痕特征图和所述原始待修复图像相乘，以得到所述划痕增强特征图，所述得到划痕增强特征图中扩张划痕的像素点的像素值为0。
  13. 一种图像修复装置，其特征在于，所述装置包括：
    第一获取模块，用于获取原始待修复图像；
    检测模块，用于对所述原始待修复图像进行划痕检测；
    第二获取模块，用于响应于检测到所述原始待修复图像中的原始划痕，得到所述原始待修复图像的原始划痕特征图；
    扩张模块，用于将所述原始划痕特征图中的原始划痕处理为扩张划痕，以得到扩张划痕特征图，所述扩张划痕包括至少一个划痕单元，所述原始划痕位于所述至少一个划痕单元所在的区域中；
    增强模块，用于将所述扩张划痕特征图中的扩张划痕覆盖在所述原始待修复图像中的原始划痕所在位置处，得到划痕增强特征图；
    修复模块，用于将所述划痕增强特征图输入图像修复网络，以对所述划痕 增强特征图中的扩张划痕进行修复，得到所述原始待修复图像对应的修复图像。
  14. 根据权利要求13所述的装置，其特征在于，所述扩张模块，包括：
    第三获取模块，用于获取目标区域，所述目标区域包括原始划痕特征图中原始划痕的像素点；
    遍历模块，用于通过检测框遍历所述目标区域；
    像素处理模块，用于响应于检测到所述检测框中存在所述原始划痕的像素点，将所述检测框中的多个像素点处理为所述划痕单元，所述划痕单元中像素点的像素值为预设值，以得到所述扩张划痕特征图。
  15. 根据权利要求13所述的装置，其特征在于，所述修复模块用于：
    通过所述图像修复网络将所述划痕增强特征图中的原始图像特征与扩张划痕特征进行融合，以对所述划痕增强特征图中的扩张划痕进行修复，得到修复特征图。
  16. 一种图像修复设备，其特征在于，所述图像修复设备包括处理器和存储器，所述存储器中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或指令集由所述处理器加载并执行以实现如权利要求1至12任一所述的图像修复方法。
  17. 一种非瞬态计算机存储介质，其特征在于，所述非瞬态计算机存储介质中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或指令集由处理器加载并执行以实现如权利要求1至12任一所述的图像修复方法。
PCT/CN2022/123334 2022-09-30 2022-09-30 图像修复方法、装置、设备和非瞬态计算机存储介质 WO2024065701A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202280003405.3A CN118140247A (zh) 2022-09-30 2022-09-30 图像修复方法、装置、设备和非瞬态计算机存储介质
PCT/CN2022/123334 WO2024065701A1 (zh) 2022-09-30 2022-09-30 图像修复方法、装置、设备和非瞬态计算机存储介质

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2022/123334 WO2024065701A1 (zh) 2022-09-30 2022-09-30 图像修复方法、装置、设备和非瞬态计算机存储介质

Publications (1)

Publication Number Publication Date
WO2024065701A1 true WO2024065701A1 (zh) 2024-04-04

Family

ID=90475656

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/123334 WO2024065701A1 (zh) 2022-09-30 2022-09-30 图像修复方法、装置、设备和非瞬态计算机存储介质

Country Status (2)

Country Link
CN (1) CN118140247A (zh)
WO (1) WO2024065701A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5974194A (en) * 1995-10-30 1999-10-26 Sony Corporation Projection based method for scratch and wire removal from digital images
CN114187200A (zh) * 2021-12-08 2022-03-15 Oppo广东移动通信有限公司 图像处理方法、装置、电子设备和计算机可读存储介质
CN114331912A (zh) * 2022-01-06 2022-04-12 北京字跳网络技术有限公司 一种图像修复方法及装置
CN114417993A (zh) * 2022-01-18 2022-04-29 北京航空航天大学 一种基于深度卷积神经网络和图像分割的划痕检测方法
CN114998337A (zh) * 2022-08-03 2022-09-02 联宝(合肥)电子科技有限公司 一种划痕检测方法、装置、设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5974194A (en) * 1995-10-30 1999-10-26 Sony Corporation Projection based method for scratch and wire removal from digital images
CN114187200A (zh) * 2021-12-08 2022-03-15 Oppo广东移动通信有限公司 图像处理方法、装置、电子设备和计算机可读存储介质
CN114331912A (zh) * 2022-01-06 2022-04-12 北京字跳网络技术有限公司 一种图像修复方法及装置
CN114417993A (zh) * 2022-01-18 2022-04-29 北京航空航天大学 一种基于深度卷积神经网络和图像分割的划痕检测方法
CN114998337A (zh) * 2022-08-03 2022-09-02 联宝(合肥)电子科技有限公司 一种划痕检测方法、装置、设备及存储介质

Also Published As

Publication number Publication date
CN118140247A (zh) 2024-06-04

Similar Documents

Publication Publication Date Title
CN113822314B (zh) 图像数据处理方法、装置、设备以及介质
CN111696110B (zh) 场景分割方法及系统
CN110569844B (zh) 基于深度学习的船舶识别方法及系统
CN111681273A (zh) 图像分割方法、装置、电子设备及可读存储介质
CN111179196B (zh) 一种基于分而治之的多分辨率深度网络图像去高光方法
CN112906794A (zh) 一种目标检测方法、装置、存储介质及终端
CN114581456B (zh) 一种多图像分割模型的构建方法、图像检测方法及装置
US20230252605A1 (en) Method and system for a high-frequency attention network for efficient single image super-resolution
CN113516126A (zh) 一种基于注意力特征融合的自适应阈值场景文本检测方法
Yang et al. SRDN: A unified super-resolution and motion deblurring network for space image restoration
CN114758145A (zh) 一种图像脱敏方法、装置、电子设备及存储介质
CN114298971A (zh) 一种冠状动脉分割方法、系统、终端以及存储介质
CN113888425A (zh) 基于多帧融合的工业质检图像文字匹配方法及系统
CN110008949A (zh) 一种图像目标检测方法、系统、装置和存储介质
CN117132503A (zh) 一种图像局部高亮区域修复方法、系统、设备及存储介质
WO2024065701A1 (zh) 图像修复方法、装置、设备和非瞬态计算机存储介质
CN116704206A (zh) 图像处理方法、装置、计算机设备和存储介质
Zheng et al. Joint residual pyramid for joint image super-resolution
CN112487943B (zh) 关键帧去重的方法、装置和电子设备
CN115937121A (zh) 基于多维度特征融合的无参考图像质量评价方法及系统
CN114022458A (zh) 骨架检测方法、装置、电子设备和计算机可读存储介质
CN114943655A (zh) 基于循环深度卷积生成对抗网络结构的图像修复系统
CN114723973A (zh) 大尺度变化鲁棒的图像特征匹配方法及装置
CN111861897A (zh) 一种图像处理方法及装置
CN111079624A (zh) 一种样本信息采集的方法、装置、电子设备以及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22960320

Country of ref document: EP

Kind code of ref document: A1