CN117745593B - Diffusion model-based old photo scratch repairing method and system - Google Patents
Diffusion model-based old photo scratch repairing method and system Download PDFInfo
- Publication number
- CN117745593B CN117745593B CN202311852854.8A CN202311852854A CN117745593B CN 117745593 B CN117745593 B CN 117745593B CN 202311852854 A CN202311852854 A CN 202311852854A CN 117745593 B CN117745593 B CN 117745593B
- Authority
- CN
- China
- Prior art keywords
- scratch
- denoising
- noise
- old photo
- diffusion model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000009792 diffusion process Methods 0.000 title claims abstract description 62
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000010586 diagram Methods 0.000 claims abstract description 43
- 238000001514 detection method Methods 0.000 claims description 44
- 238000009826 distribution Methods 0.000 claims description 29
- 230000008439 repair process Effects 0.000 claims description 23
- 238000012545 processing Methods 0.000 claims description 17
- 238000012549 training Methods 0.000 claims description 17
- 238000004364 calculation method Methods 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 14
- 238000010276 construction Methods 0.000 claims description 13
- 238000005520 cutting process Methods 0.000 claims description 5
- 239000006185 dispersion Substances 0.000 claims description 3
- 230000006870 function Effects 0.000 description 9
- 238000011084 recovery Methods 0.000 description 4
- 230000005653 Brownian motion process Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000000605 extraction Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 238000005537 brownian motion Methods 0.000 description 2
- 238000010924 continuous production Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000007476 Maximum Likelihood Methods 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 239000000428 dust Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Abstract
The invention discloses a diffusion model-based old photo scratch repairing method and a diffusion model-based old photo scratch repairing system, wherein the method comprises the following steps: s1: establishing a diffusion model, and carrying out noise adding on the old photo of the scratch to be repaired based on a noise adding mode of the diffusion model to obtain a complete Gaussian noise diagram; s2: establishing a scratch denoising space based on the old photo of the scratch to be repaired; s3: establishing a noise prediction network, performing noise prediction on the full Gaussian noise map and the scratch denoising space to respectively obtain corresponding pure noise maps, and calculating the score of each pure noise map; s4: weighting and calculating the score of each pure noise graph to obtain a state graph; and carrying out reverse iteration denoising treatment on the state diagram based on the denoising mode of the diffusion model to obtain a clean old photo without scratches. The method is more perfect, and the overall look and feel of the obtained clean photo is better.
Description
Technical Field
The invention belongs to the technical field of digital image processing, and particularly relates to a diffusion model-based old photo scratch repairing method and system.
Background
Thanks to the popularization of terminal photographing technologies such as mobile phones, the development of mobile phone scanning technologies, traditional photo electronization becomes simpler and more accessible. Therefore, the method is more convenient and easy to convert the paper old photo of the individual into the digital archive, and the old photo can be permanently stored after being digitized. However, since old photographs are mostly printed on photographic paper, the conditions for preservation are severe, and if the photographs are not preserved properly, irreversible damage such as scratches and creases will occur on the surfaces of the photographs, which will be hereinafter referred to as scratches. Therefore, after the old photo is converted into the electronic photo, the electronic photo is repaired, so that the electronic photo is a popular work in recent years.
At the same time, the repair work of old photos is also a popular technical attack field, and more technologies attempt to recover old photos with dust sealed for a long time. In the repair work of old photos, the repair of structural damage, namely scratches, creases and the like is one of the most complex and time-consuming steps, and two modes of manual repair and related algorithm automatic repair mainly exist at present. The manual repair mainly adopts picture repair software to mark scratch areas and then performs a content filling mode, and the mode has strong subjectivity and depends on personal technical ability of image technicians. The algorithm repair mainly comprises a traditional scratch repair mode and a deep learning-based scratch repair mode, wherein the traditional repair mode mainly depends on manually marking specific positions of scratches, and then filling repair is carried out by adopting a surrounding pixel diffusion mode or a similar block texture mode. With the continuous improvement of the performance and efficiency of various image generation networks and models, the old photo restoration method based on deep learning adopts a target extraction network to extract scratches, and uses a generation model to predict semantic information of the scratches, so that the restored context information is kept consistent. In short, the common feature of these methods is that the specific location of the scratch is obtained first, and then the scratch is filled and repaired according to the location of the scratch. However, this method has two main problems, namely, firstly, the problem of insufficient scratch detection and secondly, the problem of repair quality. For example: the problems of whole tone deviation, scratch residue, detail missing, repair result distortion and the like occur. The most prominent problem is that scratch detection is incomplete, partial scratches still remain in a picture, the existing work is not processed in a targeted way, and meanwhile, the existing work can also have problems in terms of repair quality such as fuzzy semantic inconsistency and the like in scratch area filling. Aiming at the problems, we provide a method and a system for repairing the scratches of the old photo based on a diffusion model.
Disclosure of Invention
The invention aims to solve the defects of the prior art, and provides a diffusion model-based old photo scratch repairing method and a diffusion model-based old photo scratch repairing system, which solve the problem of complex scratch degradation of all old photo subjects. According to the scheme, the scratch characteristics are identified in a targeted manner, high-quality content restoration is performed, no manual intervention is needed, and therefore special treatment such as marking and other treatment on scratches is not needed in actual operation of a user.
In order to achieve the above object, the present invention provides the following solutions:
a scratch repairing method based on a diffusion model old photo comprises the following steps:
S1: establishing a diffusion model, and carrying out noise adding on the old photo of the scratch to be repaired based on a noise adding mode of the diffusion model to obtain a complete Gaussian noise diagram;
S2: establishing a scratch denoising space based on the old photo of the scratch to be repaired;
s3: establishing a noise prediction network, performing noise prediction on the full Gaussian noise map and the scratch denoising space to respectively obtain corresponding pure noise maps, and calculating the score of each pure noise map;
S4: weighting and calculating the score of each pure noise graph to obtain a state graph; wherein the score is a probability density logarithmic gradient function; and carrying out reverse iteration denoising treatment on the state diagram based on the denoising mode of the diffusion model to obtain a clean old photo without scratches.
Preferably, in step S1 and step S4, the diffusion model is used for noise adding and noise removing, and the method for establishing the diffusion model is as follows:
and injecting Gaussian noise into the old photo of the scratch to be repaired at each continuous moment by adopting a random differential equation noise adding mode to obtain the complete Gaussian noise image, wherein the formula is as follows:
dx＝θt(η-x)dt+ξtdw，
wherein x is a state diagram at a time t, theta t (eta-x) and zeta t are respectively a drift equation and a dispersion equation, eta is a scratch old photo, dw is Gaussian noise, and theta t and zeta t are positive parameters related to time;
And denoising the complete Gaussian noise map and the scratch denoising space by adopting a reverse iteration denoising mode, wherein the formula is as follows:
Wherein, Is a probability density logarithmic gradient function;
Based on the random differential equation denoising mode and the reverse iteration denoising mode, converting the discrete denoising and denoising process into a continuous denoising and denoising process, and completing the establishment of the diffusion model.
Preferably, in step S2, the method for establishing the scratch denoising space includes:
based on an odd Gaussian convolution kernel, calculating a center point weighted average value of the old photo of the scratch to be repaired, and completing Gaussian blur processing to obtain a brightness information feature map;
establishing and training a scratch detection network, and performing scratch detection on the old photo of the scratch to be repaired to obtain normal distribution of the scratch;
And establishing the scratch denoising space based on the brightness information feature map and the scratch normal distribution.
Preferably, the method for establishing and training the scratch detection network comprises the following steps:
simulating the old photo based on the real scratch old photo and the synthetic scratch, and establishing a scratch old photo data set;
Marking the scratch area of each old photo in the scratch old photo dataset, enabling the scratch old photo to correspond to the photo to be segmented after marking one by one, and training an image processing network Unet to obtain a scratch detection result;
Based on the scratch detection result, minimizing the difference between the detected scratch mask and the actual scratch mask y by adopting cross entropy loss;
and obtaining an objective function of the scratch detection network based on the difference value, and completing establishment and training of the scratch detection network.
Preferably, the method for obtaining the normal distribution of scratches is as follows:
performing scratch detection on the old photo to be repaired based on the scratch detection network to obtain a binary image mask with scratch position information;
Cutting the binary image mask and the old photo of the scratch to be repaired to obtain a feature image with only scratch information;
and adopting an encoder to encode the characteristic diagram with the scratch information only to obtain the normal distribution of scratches of the old photo to be repaired.
Preferably, in step S4, the method for obtaining the clean old photo without scratches includes:
weighting calculation is carried out on the scores of the pure noise graphs, and a final score calculation result is obtained;
based on the final score calculation result, a next-stage state diagram is obtained;
And performing reverse iteration denoising processing on the state diagram based on the denoising mode of the diffusion model, removing scratch information, generating semantic information which accords with the global state, and obtaining the clean old photo without scratches.
The invention also provides a diffusion model-based old photo scratch repair system for realizing the repair method, which comprises the following steps: the system comprises a noise adding module, a denoising space constructing module, a noise predicting module and a denoising module;
the noise adding module is used for establishing a diffusion model, and adding noise to the old photo of the scratch to be repaired based on the noise adding mode of the diffusion model to obtain a complete Gaussian noise diagram;
The denoising space construction module is used for establishing a scratch denoising space based on the old photo of the scratch to be repaired;
The noise prediction module is used for establishing a noise prediction network, performing noise prediction on the complete Gaussian noise map and the scratch denoising space, respectively obtaining corresponding pure noise maps, and calculating the score of each pure noise map;
the denoising module is used for weighting and calculating the score of each pure noise graph to obtain a state graph; wherein the score is a probability density logarithmic gradient function; and carrying out reverse iteration denoising treatment on the state diagram based on the denoising mode of the diffusion model to obtain a clean old photo without scratches.
Preferably, the denoising space construction module comprises a convolution unit, a detection unit and a denoising space construction unit;
The convolution unit is used for calculating a center point weighted average value of the old photo of the scratch to be repaired based on an odd Gaussian convolution kernel to finish Gaussian blur processing and obtain a brightness information feature map;
The detection unit is used for establishing and training a scratch detection network, and performing scratch detection on the old photo of the scratch to be repaired to obtain normal distribution of the scratch;
The denoising space construction unit is used for establishing the scratch denoising space based on the brightness information feature map and the scratch normal distribution.
Compared with the prior art, the invention has the beneficial effects that:
By training the deep learning model network, the noise prediction of each step in the direction generation of the diffusion model is guided by the scratch distribution feature extraction module, in the process, the scratch distribution can be extracted to guide us to recover scratches from a global angle without depending on the specific positions of the scratches, meanwhile, the generated details are more perfect, and the overall impression is better.
Drawings
In order to more clearly illustrate the technical solutions of the present invention, the drawings that are needed in the embodiments are briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a method for repairing scratches on old photo based on a diffusion model according to an embodiment of the present invention;
FIG. 2 is a diagram of a diffusion model structure in accordance with an embodiment of the present invention;
FIG. 3 is a flow chart of Gaussian blur according to an embodiment of the invention;
FIG. 4 is a block diagram of a scratch distribution extraction flow module according to an embodiment of the present invention;
Fig. 5 is a network diagram of noise prediction according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
Example 1
As shown in FIG. 1, the method for repairing the old photo scratch based on the diffusion model comprises the following steps:
s1: establishing a diffusion model, and carrying out noise adding on the old photo of the scratch to be repaired based on a noise adding mode of the diffusion model to obtain a complete Gaussian noise diagram;
A further embodiment is that, as shown in fig. 2, the diffusion model includes a noise adding portion and a noise removing portion, and in step S1, using the noise adding portion of the diffusion model, a specific method for creating the diffusion model is as follows:
The addition of signals, i.e., brownian motion, by means of the addition of random differential equations is a continuous gaussian process that injects stable, random noise into the system at each successive instant, changing the discrete addition and removal of noise into a continuous process. Where w represents Brownian motion (or Wiener process), θ t (η -x) controls the "flow direction" of the entire SDE and ζ t controls the extent of diffusion or noise addition rate of the SDE. The gaussian noise is randomly sampled at each step and the variance becomes progressively larger, eventually bringing it to a full gaussian noise figure. The method comprises the steps of injecting Gaussian noise into an old photo of a scratch to be repaired at each continuous moment to obtain a complete Gaussian noise diagram, wherein the formula is as follows:
dx＝θt(η-x)dt+ξtdw，
Wherein x is a state diagram at a time t, theta t (eta-x) and zeta t are respectively a drift equation and a dispersion equation, eta is a scratch old photo, dw is Gaussian noise, and theta t and zeta t are positive parameters related to time; in this process, the whole time T is set to be 100, and the noise adding is changed from a discrete process to a continuous process by the reference of the SDE random differential equation diffusion mode.
In step S4, a reverse iterative denoising method is adopted to denoise the complete gaussian noise diagram and the scratch denoising space, and the formula is as follows:
Wherein, Is a probability density logarithmic gradient function; x is the state diagram at time t (namely, the state diagram at time t of the complete Gaussian noise diagram or scratch denoising space)
Based on a random differential equation denoising mode and a reverse iteration denoising mode, the discrete denoising and denoising process is converted into a continuous denoising and denoising process, and the establishment of the diffusion model is completed.
S2: establishing a scratch denoising space based on the old photo of the scratch to be repaired; the spatial construction is to predict noise in a gaussian-blurred picture with the distribution of scratches, and input t and gaussian-blurred pictures and x (t) into a noise prediction network of S5 to perform noise prediction, because it is possible to preserve luminance information of an original picture while eliminating scratch information.
In a further embodiment, in step S2, the method for creating the scratch denoising space includes:
as shown in fig. 3, based on an odd gaussian convolution kernel, calculating a weighted average value of pixel value center points of the old photo of the scratch to be repaired, completing gaussian blur processing, obtaining a brightness information feature map of reserved picture brightness information through the process, and eliminating high-frequency information such as scratches;
establishing and training a scratch detection network, and performing scratch detection on the old photo of the scratch to be repaired to obtain normal distribution of the scratch;
And establishing the scratch denoising space based on the brightness information feature map and the scratch normal distribution.
A further embodiment is that the method for obtaining the luminance information feature map includes:
And calculating a center point weighted average value of the old photo of the scratch to be repaired based on the odd Gaussian convolution kernel, and completing Gaussian blur processing to obtain a brightness information feature map.
The scratch detection segmentation network uses an image processing network Unet with wide application scenes, and the network is subjected to fine tuning and retraining to detect the scratch in the segmentation scenes. Unet undergo multiple downsampling and upsampling operations to extract the scratch features of the picture.
A further embodiment is that the method for establishing and training the scratch detection network comprises the following steps:
simulating the old photo based on the real scratch old photo and the synthetic scratch, and establishing a scratch old photo data set;
Marking the scratch area of each old photo in the scratch old photo data set, enabling the scratch old photo to correspond to the photo to be segmented after marking one by one, and training the image processing network Unet to obtain a scratch detection result;
based on scratch detection results, minimizing the difference between the detected scratch mask and the actual scratch mask y by adopting cross entropy loss;
And obtaining an objective function of the scratch detection network based on the difference value, and completing establishment and training of the scratch detection network.
A further embodiment is that the method for obtaining a normal distribution of scratches comprises:
performing scratch detection on the old photo of the scratch to be repaired based on a scratch detection network to obtain a binary image mask with scratch position information;
Cutting the binary image mask and the old photo of the scratch to be repaired to obtain a feature image with only scratch information;
And (3) encoding the characteristic diagram with the scratch information by adopting an encoder to obtain the normal distribution of scratches of the old photo to be repaired. Specifically, by a simple cutting operation mode, a picture with only scratch content can be obtained, and by encoding the picture, a group of mean values and a group of variances are obtained, wherein the mean values and the variances respectively correspond to a gaussian distribution, as shown in fig. 4.
S3: establishing a noise prediction network, carrying out noise prediction on the complete Gaussian noise image and the scratch denoising space to respectively obtain corresponding pure noise images, calculating the specific score of each pure noise image, and carrying out reverse iterative denoising processing on the output in S1, wherein unknown is onlyThe probability density logarithmic gradient function is score, a noise prediction network based on U-net is trained, and the network is trained in a maximum likelihood estimation mode, so that the noise in the current time state is predicted more accurately. In test generation, the noise in the current state is predicted through the network to obtain a pure noise diagram, and finally the score is calculated, wherein the network diagram is shown in fig. 5.
S4: weighting and calculating the score of each pure noise graph to obtain a state graph; wherein the score is a probability density logarithmic gradient function; and carrying out reverse iteration denoising treatment on the state diagram based on the denoising mode of the diffusion model to obtain a clean old photo without scratches.
In a further embodiment, in step S4, the method for obtaining a clean old photo without scratches includes:
weighting calculation is carried out on the scores of all the pure noise graphs, and a final score calculation result is obtained;
based on the final score calculation result, a next-stage state diagram is obtained;
And performing reverse iterative denoising processing on the state diagram based on the denoising mode of the diffusion model, removing scratch information, generating semantic information which accords with the global state, and obtaining the clean old photo without scratches.
In this embodiment, the number of normal distributions in S3 is set to 3, and the distributions generate three score in the gaussian state space, and the score predicted by the scratch noise map is added to the normal distributions, and the four score are combined to obtain a final score, and the calculation mode of reverse generation in S4 is adopted to obtain the next state diagram according to the final score.
According to the calculated score, x (t) can obtain a state diagram x (t-1) of the next stage, through the repeated iteration, the process can be regarded as reverse recovery of S1, but in the reverse recovery process, the scratch information is eliminated and the semantic information conforming to the whole world is generated, the predicted noise and the scratch are regarded as abnormal information, the noise is taken away, and the removed noise and x (t) are filled according to the inverse formula, so that the process can be simply understood as that in a vector space, the denoising process advances towards the target image which we want to reach. From T to X, the scratches are gradually removed and new contents are generated, and finally a clear picture without scratches, namely a repaired picture, is obtained.
In general, the invention enables a diffusion model to have certain recovery and generation capacity of old photo scratches in a targeted manner by training and fine-tuning the diffusion model, and simultaneously enables a denoising path of the diffusion model to trend towards distribution without scratches by adopting a reverse recovery process of guiding and intervening the diffusion model in order to ensure that more scratches are removed, and finally the model has more pertinence for removing the scratches. Compared with other models, the method has the advantages that the method is independent of the specific positions of scratches, the global repair of the scratches can be realized on the scratch characteristic information, the method has higher scratch removal rate, and the generated effect is finer.
Example two
The invention also provides a diffusion model-based old photo scratch repair system, which is used for repairing a method and comprises the following steps: the system comprises a noise adding module, a denoising space constructing module, a noise predicting module and a denoising module;
the noise adding module is used for establishing a diffusion model, adding noise to the old photo of the scratch to be repaired based on a noise adding mode of the diffusion model, and obtaining a complete Gaussian noise diagram;
The denoising space construction module is used for establishing a scratch denoising space based on the old photo of the scratch to be repaired;
The noise prediction module is used for establishing a noise prediction network, carrying out noise prediction on the complete Gaussian noise map and the scratch denoising space, respectively obtaining corresponding pure noise maps, and calculating the score of each pure noise map;
The denoising module is used for weighting and calculating the score of each pure noise graph to obtain a state graph; wherein the score is a probability density logarithmic gradient function; and carrying out reverse iteration denoising treatment on the state diagram based on the denoising mode of the diffusion model to obtain a clean old photo without scratches.
The further implementation mode is that the denoising space construction module comprises a convolution unit, a detection unit and a denoising space construction unit;
The convolution unit is used for calculating a center point weighted average value of the old photo of the scratch to be repaired based on an odd Gaussian convolution kernel to finish Gaussian blur processing and obtain a brightness information feature map;
The detection unit is used for establishing and training a scratch detection network, and performing scratch detection on the old photo of the scratch to be repaired to obtain normal distribution of the scratch;
and the denoising space construction unit is used for establishing the scratch denoising space based on the brightness information feature map and the scratch normal distribution.
The denoising module comprises a final score calculating unit, a state diagram acquiring unit and a denoising unit;
The final score calculation unit is used for carrying out weighted calculation on the score of each pure noise graph to obtain a final score calculation result;
a state diagram obtaining unit, configured to obtain a next-stage state diagram based on the final score calculation result;
And the denoising unit is used for performing reverse iteration denoising processing on the state diagram based on the denoising mode of the diffusion model, eliminating scratch information and generating semantic information which accords with the global situation, and obtaining the clean old photo without scratches.
The above embodiments are merely illustrative of the preferred embodiments of the present invention, and the scope of the present invention is not limited thereto, but various modifications and improvements made by those skilled in the art to which the present invention pertains are made without departing from the spirit of the present invention, and all modifications and improvements fall within the scope of the present invention as defined in the appended claims.
Claims (5)
1. The old photo scratch repairing method based on the diffusion model is characterized by comprising the following steps of:
S1: establishing a diffusion model, and carrying out noise adding on the old photo of the scratch to be repaired based on a noise adding mode of the diffusion model to obtain a complete Gaussian noise diagram;
s2: establishing a scratch denoising space based on the old photo of the scratch to be repaired; in step S2, the method for establishing the scratch denoising space includes:
based on an odd Gaussian convolution kernel, calculating a center point weighted average value of the old photo of the scratch to be repaired, and completing Gaussian blur processing to obtain a brightness information feature map;
establishing and training a scratch detection network, and performing scratch detection on the old photo of the scratch to be repaired to obtain normal distribution of the scratch;
establishing the scratch denoising space based on the brightness information feature map and the scratch normal distribution;
the method for obtaining the normal distribution of scratches comprises the following steps:
performing scratch detection on the old photo to be repaired based on the scratch detection network to obtain a binary image mask with scratch position information;
Cutting the binary image mask and the old photo of the scratch to be repaired to obtain a feature image with only scratch information;
An encoder is adopted to encode the characteristic diagram with the scratch information only, so that the normal distribution of scratches of the old photo to be repaired is obtained;
s3: establishing a noise prediction network, performing noise prediction on the full Gaussian noise map and the scratch denoising space to respectively obtain corresponding pure noise maps, and calculating the score of each pure noise map;
S4: weighting and calculating the score of each pure noise graph to obtain a state graph; wherein the score is a probability density logarithmic gradient function; and carrying out reverse iteration denoising treatment on the state diagram based on the denoising mode of the diffusion model to obtain a clean old photo without scratches.
2. The diffusion model-based old photo scratch repair method according to claim 1, wherein the diffusion model is used for denoising and denoising in the step S1 and the step S4, respectively, and the method for establishing the diffusion model is as follows:
and injecting Gaussian noise into the old photo of the scratch to be repaired at each continuous moment by adopting a random differential equation noise adding mode to obtain the complete Gaussian noise image, wherein the formula is as follows:
dx＝θt(η-x)dt+ξtdw，
wherein x is a state diagram at a time t, theta t (eta-x) and zeta t are respectively a drift equation and a dispersion equation, eta is a scratch old photo, dw is Gaussian noise, and theta t and zeta t are positive parameters related to time;
And denoising the complete Gaussian noise map and the scratch denoising space by adopting a reverse iteration denoising mode, wherein the formula is as follows:
Wherein, Is a probability density logarithmic gradient function;
Based on the random differential equation denoising mode and the reverse iteration denoising mode, converting the discrete denoising and denoising process into a continuous denoising and denoising process, and completing the establishment of the diffusion model.
3. The diffusion model-based old photo scratch repair method according to claim 1, wherein the method for building and training a scratch detection network is as follows:
simulating the old photo based on the real scratch old photo and the synthetic scratch, and establishing a scratch old photo data set;
Marking the scratch area of each old photo in the scratch old photo dataset, enabling the scratch old photo to correspond to the photo to be segmented after marking one by one, and training an image processing network Unet to obtain a scratch detection result;
Based on the scratch detection result, minimizing the difference between the detected scratch mask and the actual scratch mask y by adopting cross entropy loss;
and obtaining an objective function of the scratch detection network based on the difference value, and completing establishment and training of the scratch detection network.
4. The method for repairing scratches on old photo based on diffusion model according to claim 1, wherein in step S4, the method for obtaining clean old photo without scratches is as follows:
weighting calculation is carried out on the scores of the pure noise graphs, and a final score calculation result is obtained;
based on the final score calculation result, a next-stage state diagram is obtained;
And based on the denoising mode of the diffusion model, performing reverse iteration denoising processing on the next-stage state diagram, eliminating scratch information, generating semantic information which accords with the global state, and obtaining the clean old photo without scratches.
5. A diffusion model-based old photo scratch repair system for implementing the repair method of any one of claims 1-4, comprising: the system comprises a noise adding module, a denoising space constructing module, a noise predicting module and a denoising module;
the noise adding module is used for establishing a diffusion model, and adding noise to the old photo of the scratch to be repaired based on the noise adding mode of the diffusion model to obtain a complete Gaussian noise diagram;
the denoising space construction module is used for establishing a scratch denoising space based on the old photo of the scratch to be repaired; the denoising space construction module comprises a convolution unit, a detection unit and a denoising space construction unit;
The convolution unit is used for calculating a center point weighted average value of the old photo of the scratch to be repaired based on an odd Gaussian convolution kernel to finish Gaussian blur processing and obtain a brightness information feature map;
The detection unit is used for establishing and training a scratch detection network, and performing scratch detection on the old photo of the scratch to be repaired to obtain normal distribution of the scratch;
The denoising space construction unit is used for establishing the scratch denoising space based on the brightness information feature map and the normal scratch distribution;
the method for obtaining the normal distribution of scratches comprises the following steps:
performing scratch detection on the old photo to be repaired based on the scratch detection network to obtain a binary image mask with scratch position information;
Cutting the binary image mask and the old photo of the scratch to be repaired to obtain a feature image with only scratch information;
An encoder is adopted to encode the characteristic diagram with the scratch information only, so that the normal distribution of scratches of the old photo to be repaired is obtained;
The noise prediction module is used for establishing a noise prediction network, performing noise prediction on the complete Gaussian noise map and the scratch denoising space, respectively obtaining corresponding pure noise maps, and calculating the score of each pure noise map;
the denoising module is used for weighting and calculating the score of each pure noise graph to obtain a state graph; wherein the score is a probability density logarithmic gradient function; and carrying out reverse iteration denoising treatment on the state diagram based on the denoising mode of the diffusion model to obtain a clean old photo without scratches.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311852854.8A CN117745593B (en) | 2023-12-29 | Diffusion model-based old photo scratch repairing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311852854.8A CN117745593B (en) | 2023-12-29 | Diffusion model-based old photo scratch repairing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117745593A CN117745593A (en) | 2024-03-22 |
| CN117745593B true CN117745593B (en) | 2024-07-05 |
Family
ID=
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116664450A (en) * | 2023-07-26 | 2023-08-29 | 国网浙江省电力有限公司信息通信分公司 | Diffusion model-based image enhancement method, device, equipment and storage medium |
| CN117252782A (en) * | 2023-11-01 | 2023-12-19 | 河北工业大学 | Image restoration method based on conditional denoising diffusion and mask optimization |
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116664450A (en) * | 2023-07-26 | 2023-08-29 | 国网浙江省电力有限公司信息通信分公司 | Diffusion model-based image enhancement method, device, equipment and storage medium |
| CN117252782A (en) * | 2023-11-01 | 2023-12-19 | 河北工业大学 | Image restoration method based on conditional denoising diffusion and mask optimization |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108986050B (en) | Image and video enhancement method based on multi-branch convolutional neural network | |
| CN111915530A (en) | End-to-end-based haze concentration self-adaptive neural network image defogging method | |
| CN112164011A (en) | Motion image deblurring method based on self-adaptive residual error and recursive cross attention | |
| CN112288632A (en) | Single image super-resolution method and system based on simplified ESRGAN | |
| CN116645260B (en) | Digital watermark attack method based on conditional diffusion model | |
| Guo et al. | Joint raindrop and haze removal from a single image | |
| Zheng et al. | T-net: Deep stacked scale-iteration network for image dehazing | |
| CN117333398A (en) | Multi-scale image denoising method and device based on self-supervision | |
| CN111861935B (en) | Rain removing method based on image restoration technology | |
| CN111524060B (en) | System, method, storage medium and device for blurring portrait background in real time | |
| CN117132503A (en) | Method, system, equipment and storage medium for repairing local highlight region of image | |
| CN117745593B (en) | Diffusion model-based old photo scratch repairing method and system | |
| CN114913095B (en) | Depth deblurring method based on domain adaptation | |
| CN116563133A (en) | Low-illumination color image enhancement method based on simulated exposure and multi-scale fusion | |
| CN117745593A (en) | Diffusion model-based old photo scratch repairing method and system | |
| CN113269167B (en) | Face counterfeiting detection method based on image blocking and disordering | |
| CN113160056A (en) | Deep learning-based noisy image super-resolution reconstruction method | |
| CN113902647A (en) | Image deblurring method based on double closed-loop network | |
| CN113935910A (en) | Image fuzzy length measuring method based on deep learning | |
| CN113177893A (en) | Image fuzziness processing method | |
| CN114596219B (en) | Image motion blur removing method based on condition generation countermeasure network | |
| CN110717873A (en) | Traffic sign deblurring detection recognition algorithm based on multi-scale residual error | |
| Wu et al. | RDS-denoiser: a detail-preserving convolutional neural network for image denoising | |
| CN112200751B (en) | Image enhancement method | |
| CN116977220B (en) | Blind image motion blur removal algorithm based on image quality heuristic |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |