CN114913468A - Object repairing method, repair evaluating method, electronic device, and storage medium - Google Patents
Object repairing method, repair evaluating method, electronic device, and storage medium Download PDFInfo
- Publication number
- CN114913468A CN114913468A CN202210681242.6A CN202210681242A CN114913468A CN 114913468 A CN114913468 A CN 114913468A CN 202210681242 A CN202210681242 A CN 202210681242A CN 114913468 A CN114913468 A CN 114913468A
- Authority
- CN
- China
- Prior art keywords
- repaired
- video frame
- similarity
- weight
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000008439 repair process Effects 0.000 title claims abstract description 67
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000009499 grossing Methods 0.000 claims description 70
- 230000004927 fusion Effects 0.000 claims description 26
- 238000004891 communication Methods 0.000 claims description 15
- 230000000875 corresponding effect Effects 0.000 claims description 12
- 230000002596 correlated effect Effects 0.000 claims description 7
- 239000006185 dispersion Substances 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 2
- 238000011156 evaluation Methods 0.000 abstract description 13
- 230000000694 effects Effects 0.000 abstract description 10
- 238000010586 diagram Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 6
- 238000000605 extraction Methods 0.000 description 4
- 239000000203 mixture Substances 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000003786 synthesis reaction Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/761—Proximity, similarity or dissimilarity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/48—Matching video sequences
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Life Sciences & Earth Sciences (AREA)
- Data Mining & Analysis (AREA)
- Biophysics (AREA)
- Application Of Or Painting With Fluid Materials (AREA)
- Working Measures On Existing Buildindgs (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
The embodiment of the invention provides an object repairing method, a repairing evaluation method, electronic equipment and a storage medium. An object repair method, comprising: pre-repairing an alternative object to be repaired in a video frame to obtain a pre-repaired object of the alternative object to be repaired; selecting an object to be repaired in the alternative objects to be repaired; and fusing based on the object to be repaired and the object to be repaired to obtain a repaired object. In the scheme of the embodiment of the invention, the restoration object is obtained by fusing the object to be restored and the pre-restoration object thereof, so that video frame distortion caused by partial pre-restoration objects is avoided, and the video frame restoration effect is improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to an object repairing method, a repairing evaluation method, electronic equipment and a storage medium.
Background
With the continuous development of cloud computing technology towards the native direction of the cloud, cloud service providers provide basic services (IaaS) such as computing, storage, network, security and the like, and also provide platform services (PaaS) such as databases, big data processing, video cloud and the like, and video repair is one of video processing functions provided by the video cloud. For movie and television works, high-quality video data can be obtained through video restoration, and the huge cost of copying the movie and television works is saved while the watching experience of a user is guaranteed.
In the video repair process, the determination of an object to be repaired in a video frame is an important link, and in a traditional repair mode, manual repair by a professional is mainly used, so that the repair efficiency is low, and the cost is high.
With the development of artificial intelligence technology represented by deep learning, the automatic repair mode based on the artificial intelligence technology realizes lower repair cost and higher processing efficiency, but the video frame repair effect is still not ideal.
Disclosure of Invention
In view of the above, embodiments of the present invention provide an object repairing method, a repair evaluating method, an electronic device, and a storage medium, so as to at least partially solve the above problems.
According to a first aspect of the embodiments of the present invention, there is provided an object repairing method, including: pre-repairing an alternative object to be repaired in a video frame to obtain a pre-repaired object of the alternative object to be repaired; selecting an object to be repaired in the alternative objects to be repaired; and fusing based on the object to be repaired and the object to be repaired to obtain a repaired object.
In another implementation manner of the present invention, the fusing based on the object to be repaired and the pre-repaired object thereof to obtain the repaired object includes: and fusing the object to be repaired of the video frame and the object to be repaired of the video frame to obtain a repaired object based on the fusion weight, wherein the fusion weight is determined based on the initial fusion weight and the smooth weight.
In another implementation of the invention, the method further comprises: determining the smoothing weight based on at least one of an inter-frame correlation weight between a previous video frame of the video frames and the video frame, and an inter-frame distortion smoothing weight.
In another implementation of the invention, the method further comprises: and determining inter-frame distortion smoothing weight of the video frame based on the similarity between the object to be repaired of the video frame and the object to be repaired of the video frame.
In another implementation manner of the present invention, the determining inter-frame distortion smoothing weights of the video frame based on the similarity between the object to be repaired of the video frame and the object to be repaired thereof comprises: when the similarity between an object to be repaired of the video frame and a pre-repaired object of the video frame is greater than a first similarity, determining interframe distortion smoothing weight of the video frame as a first interframe distortion smoothing weight threshold; when the similarity between an object to be repaired of the video frame and a pre-repaired object of the video frame is between the first similarity and the second similarity, determining inter-frame distortion smoothing weight of the video frame as being positively correlated with the similarity; and when the similarity between the object to be repaired of the video frame and the object to be repaired of the video frame is smaller than the second similarity, determining the interframe distortion smoothing weight of the video frame as a second interframe distortion smoothing weight threshold, wherein the value range of the similarity is between the first interframe distortion smoothing weight threshold and the second interframe distortion smoothing weight threshold.
In another implementation of the invention, the method further comprises: and determining the inter-frame correlation weight based on the weight of the candidate object to be repaired determined as the object to be repaired in each previous video frame of the video frames, wherein each previous video frame is stored in a buffer corresponding to the video frame.
In another implementation of the invention, the method further comprises: and determining each initial fusion weight between the object to be repaired of each previous video frame and the object to be repaired of each previous video frame based on the discrete degree of the inter-frame correlation weight.
According to a second aspect of the embodiments of the present invention, there is provided a repair evaluation method including: pre-repairing an alternative object to be repaired in a video frame to obtain a pre-repaired object of the alternative object to be repaired; determining the similarity between the alternative object to be repaired and the pre-repaired object thereof; and determining the alternative object to be repaired with the similarity meeting a preset similarity threshold as the object to be repaired in the video frame.
In another implementation of the invention, the method further comprises: determining an inter-frame distortion smoothing weight corresponding to the video frame, wherein the larger the inter-frame distortion smoothing weight is, the larger the inter-frame difference between the video frame and an adjacent video frame is; and determining a preset similarity threshold positively correlated with the interframe distortion smoothing weight.
In another implementation of the present invention, the inter-frame distortion smoothing weight represents a degree of pixel channel value dispersion between the video frame and an adjacent video frame. The determining the similarity between the candidate object to be repaired and the object to be repaired before repairing comprises: and determining the discrete degree between the pixel channel value of the candidate object to be repaired and the pixel channel value of the object to be repaired as the similarity.
According to a third aspect of embodiments of the present invention, there is provided an electronic apparatus, including: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus; the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the corresponding operation of the method according to the first aspect or the second aspect.
According to a fourth aspect of embodiments of the present invention, there is provided a computer storage medium having stored thereon a computer program which, when executed by a processor, implements a method according to the first or second aspect.
In the object repairing method of the embodiment of the invention, the repairing object is obtained by fusing the object to be repaired and the pre-repairing object thereof, so that the video frame distortion caused by part of the pre-repairing object is avoided, and the video frame repairing effect is improved.
In the repair evaluation method of the embodiment of the invention, the candidate object to be repaired whose similarity meets the preset similarity threshold is determined as the object to be repaired, so that the distortion degree of the repair result of the object to be repaired is low, in other words, the candidate object to be repaired which does not meet the preset similarity threshold is not repaired, and the video frame repair effect is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following descriptions are only some embodiments described in the embodiments of the present invention, and other drawings can be obtained by those skilled in the art according to these drawings.
Fig. 1 is a schematic block diagram of a video repair flow according to an example.
Fig. 2 is a flowchart of the steps of an object repairing method according to an embodiment of the present invention.
FIG. 3 is a flow diagram of the steps of a repair evaluation method according to one embodiment of the invention.
Fig. 4 is a schematic block diagram of a video repair process in conjunction with the embodiments of fig. 2 and 3.
Fig. 5 is a schematic structural diagram of an electronic device according to another embodiment of the invention.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be described in detail below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments of the present invention shall fall within the scope of the protection of the embodiments of the present invention.
The following further describes specific implementation of the embodiments of the present invention with reference to the drawings.
Fig. 1 is a schematic block diagram of a video repair flow according to an example. In this example, the video repair process is sequentially processed based on the video frame extraction module 110, the video frame repair module 120, and the video composition module 130. Specifically, the video data may be first input into the video frame extraction module 110, and the respective video frame data may be extracted based on the time sequence. Then, each video frame is input into the video frame restoration module 120 to perform video frame restoration to obtain a restored video frame, and then the restored video frame is input into the video synthesis module 130 to perform video synthesis on each restored video frame based on the inter-frame relationship of each video frame to obtain restored video data.
For example, the video frame repair module 120 may be a video frame repair model based on a Convolutional Neural Network (CNN), and the video frame repair model is obtained through supervised learning or unsupervised learning, which is beneficial to improving the video frame repair efficiency.
Fig. 2 is a flowchart illustrating steps of a video repair method according to an embodiment of the present invention. The solution of the present embodiment may be applied to any suitable electronic device with data processing capability, including but not limited to: servers, mobile terminals (e.g., cell phones, PADs, etc.), PCs, etc., for example, the electronic device may be configured as the video frame repair module 120 of fig. 1 for performing object repair and performing video repair based on the object repair.
The object repairing method of the embodiment includes:
s210: and pre-repairing the alternative object to be repaired in the video frame to obtain a pre-repaired object of the alternative object to be repaired.
For example, the candidate object to be repaired of the video frame may be an object in the video frame, such as a person, an object, a scene, a background, and the like. Specifically, in video repair of a movie work, the candidate object to be repaired includes at least a facial feature of a person. For another example, the object to be repaired may be an edge pixel of the object in the video frame, i.e., a pixel at a boundary between the object pixel and the background pixel.
For example, the video frame may include at least one candidate object to be repaired, or may not include the candidate object to be repaired. In addition, the pre-repair may be performed using a pre-trained video frame repair model. The video frame restoration model can be trained based on the video frame samples and the labels, and the learning of the video frame restoration model to the context information is facilitated.
S220: and selecting the object to be repaired in the alternative objects to be repaired.
For example, at least part of the candidate objects to be repaired are objects to be repaired, for example, the candidate objects to be repaired include objects to be repaired and other objects. The object to be repaired may be selected from the candidate objects to be repaired based on a preset similarity threshold. In one example, the similarity between the object to be repaired and the object to be repaired is greater than a preset similarity threshold, and the similarity between the other objects and the objects to be repaired is less than the preset similarity threshold.
S230: and fusing based on the object to be repaired and the pre-repaired object thereof to obtain the repaired object.
For example, the object to be repaired and its pre-repair object may be fused based on the fusion weight. In the fusion process, the weight of the object to be repaired is inversely related to the similarity (i.e., the similarity between the object to be repaired and the object to be repaired), and the weight of the object to be repaired is positively related to the similarity (i.e., the similarity between the object to be repaired and the object to be repaired).
In the object repairing method of the embodiment of the invention, the repairing object is obtained by fusing the object to be repaired and the pre-repairing object thereof, so that video frame distortion caused by part of the pre-repairing object is avoided, and the video frame repairing effect is improved.
In other examples, fusing the object to be repaired and the object to be repaired to obtain the repaired object includes: and fusing the object to be repaired of the video frame and the object to be repaired of the video frame to obtain a repaired object based on the fusion weight, wherein the fusion weight is determined based on the initial fusion weight and the smooth weight. The fusion weight takes the initial fusion weight and the smoothing weight into consideration, so that the smoothing weight can be separated out to be used for executing smoothing between frames, and the initial fusion weight is separated out to be used for ensuring the repairing effect of the object to be repaired in the video frames, thereby improving the video repairing efficiency.
In other examples, the video repair method further comprises: the smoothing weight is determined based on at least one of an inter-frame correlation weight between a previous video frame of the video frame and the video frame, and an inter-frame distortion smoothing weight.
For example, the smoothing weight is determined based on inter-frame correlation weights between a previous video frame of the video frame and the video frame. More specifically, W his ＝W prior ⅹW scale 。W his Representing fusion weights, W, taking into account inter-frame dependent weights prior Represents the initial fusion weight, W scale Representing the smoothing weight.
For another example, the smoothing weight is determined based on both the interframe distortion smoothing weights. More specifically, W sim ＝W prior ⅹW scale 。W sim Representing fusion weights, W, taking into account interframe distortion smoothing weights prior Denotes the initial fusion weight, W scale Representing the smoothing weights.
More generally, the smoothing weight is determined based on both an inter-frame correlation weight between a previous video frame of the video frame and the video frame, and an inter-frame distortion smoothing weight. Specifically, the smoothing weight W ═ W his ⅹW sim 。
The inter-frame correlation weight reflects the pixel position correlation of the object to be repaired among different video frames with time series relation, and can reliably reflect inter-frame smoothing. The distortion degree of the object to be repaired in different video frames is low, the smoothness degree is high, and the interframe distortion smoothing weight indirectly reflects the distortion degree and the smoothness degree of the repaired object through the repairing degree of the object to be repaired.
In other examples, the video repair method further comprises: and determining the interframe distortion smoothing weight of the video frame based on the similarity between the object to be repaired of the video frame and the object to be repaired of the video frame.
In other words, the inter-frame distortion smoothing weight indirectly reflects the degree of distortion and the degree of smoothing of the repaired object by the degree of repair of the object to be repaired. For example, the interframe distortion smoothing weight may be determined by a plurality of similarity intervals (e.g., an interval greater than the first similarity, an interval between the first similarity and the second similarity, and an interval less than the second similarity) between the first interframe distortion smoothing weight threshold and the second interframe distortion smoothing weight threshold, thereby efficiently and reliably obtaining the interframe distortion smoothing weight.
Further, in order to determine an inter-frame distortion smoothing weight of the video frame based on the similarity between the object to be repaired of the video frame and the object to be repaired of the video frame, when the similarity between the object to be repaired of the video frame and the object to be repaired of the video frame is greater than the first similarity, the inter-frame distortion smoothing weight of the video frame is determined as a first inter-frame distortion smoothing weight threshold; when the similarity between an object to be repaired of a video frame and a pre-repaired object of the video frame is between a first similarity and a second similarity, determining inter-frame distortion smoothing weight of the video frame as being positively correlated with the similarity; and when the similarity between the object to be repaired of the video frame and the object to be repaired of the video frame is smaller than the second similarity, determining the interframe distortion smoothing weight of the video frame as a second interframe distortion smoothing weight threshold, wherein the value range of the similarity is between the first interframe distortion smoothing weight threshold and the second interframe distortion smoothing weight threshold.
More specifically, the interframe distortion smoothing weight scale(s) represents a function of the similarity s:
in other examples, the video repair method further comprises: the inter-frame correlation weight is determined based on the discrete degree between the initial fusion weights of the object to be repaired of each previous video frame and the object to be repaired of each previous video frame.
Alternatively, the inter-frame correlation weight is determined based on the weight of the candidate object to be repaired determined as the object to be repaired in each previous video frame of the video frames, and each previous video frame is stored in the corresponding buffer of the video frame. The reference value of the previous video frame distant from the video frame in terms of time is lower than that of the previous video frame close to the video frame, the calculation efficiency of each previous video frame stored in the buffer is improved, and furthermore, the inter-frame correlation weight is determined based on the weight of the candidate object to be repaired determined as the object to be repaired in each previous video frame of the video frame, so that the calculation efficiency is improved, for example, for a previous video frame, if the candidate object to be repaired is determined as the object to be repaired, the flag value of the previous video frame may be 1, if the candidate object to be repaired is not determined as the object to be repaired, the flag value of the previous video frame is 0, and more generally, the flag value in the case where the candidate object to be repaired is determined as the object to be repaired is greater than the flag value in the case where the candidate object to be repaired is not determined as the object to be repaired. Accordingly, the above-mentioned weight may be an average value of the sum of the flag values of the respective previous video frames, for example, if the flag values of the previous 5 video frames are 1, 0, 1 and 1, respectively, then the above-mentioned weight is (1+0+1+ 1)/5 is 0.8, that is, the inter-frame correlation weight of the previous 5 video frames and the video frame (current video frame) is 0.8.
In other examples, the video repair method further comprises: the method comprises the steps of determining each initial fusion weight between an object to be repaired of each previous video frame and a pre-repaired object of the previous video frame based on the discrete degree of the inter-frame correlation weight, wherein the discrete degree of the inter-frame correlation weight is considered, each initial fusion weight is determined, the phenomenon that the discrete degree of each initial fusion weight is too large is avoided, and the reliability of inter-frame smoothing is improved.
In other examples, the discrete degree of each initial fusion weight is smaller than the discrete degree of the inter-frame correlation weight, and the inter-frame correlation weight can better ensure the inter-frame smoothing effect than each initial fusion weight, so that the intra-frame repair effect is avoided and the excessive inter-frame effect is avoided.
FIG. 3 is a flow diagram of the steps of a repair assessment method according to one embodiment of the present invention. The solution of the present embodiment may be applied to any suitable electronic device with data processing capability, including but not limited to: servers, mobile terminals (e.g., cell phones, PADs, etc.), PCs, etc., for example, the electronic device may be configured as the video frame repair module 120 of fig. 1 for performing repair evaluation and performing object repair based on the evaluated object to be repaired. Further, the repair evaluation method of the embodiment of the present embodiment may be taken as an example of selecting an object to be repaired from the candidate objects to be repaired (S220).
The repair evaluation method of the embodiment includes:
s310: and pre-repairing the alternative object to be repaired in the video frame to obtain a pre-repaired object of the alternative object to be repaired.
For example, the candidate object to be repaired of the video frame may be an object in the video frame, such as a person, an object, a scene, a background, and the like. Specifically, in video repair of a movie work, the candidate object to be repaired includes at least a facial feature of a person.
The video frame may include at least one candidate object to be repaired or may not include the candidate object to be repaired. In addition, the pre-repair may be performed using a pre-trained video frame repair model. The video frame restoration model can be trained based on the video frame samples and the labels, and the learning of the video frame restoration model to the context information is facilitated.
S320: and determining the similarity between the candidate object to be repaired and the pre-repaired object thereof.
For example, the similarity here may be a cosine similarity, a pierce coefficient, and the like between the candidate object to be repaired and the pre-repaired object thereof. The similarity can be characterized by adopting the distance of the candidate object to be repaired and the object to be repaired in the object vector space.
As an example, when determining the similarity between the candidate object to be repaired and the pre-repaired object thereof, the similarity prediction may also be performed based on a pre-trained similarity prediction model, and the similarity prediction model may be trained based on a video frame object sample and a label thereof, for example, the video frame object sample may be a partial video frame, in other words, the video frame object sample is located in an area where the object is located in the video frame, so that the similarity prediction model includes less context information than the video frame repair model, which is beneficial to improving the reliability of prediction.
As another example, the video frame repair model and the similarity prediction model may each employ a video frame sample or a video frame object sample, except that the label confidence of the video frame sample or the video frame object sample is greater with the training of the video frame repair model than with the training of the similarity prediction model.
S330: and determining the alternative object to be repaired with the similarity meeting the preset similarity condition as the object to be repaired in the video frame.
For example, the preset similarity condition may be a similarity threshold, and the candidate object to be repaired, which is greater than the similarity threshold, is used as the object to be repaired in the video frame.
In the repair evaluation method of the embodiment of the invention, the candidate object to be repaired, the similarity of which meets the preset similarity threshold, is determined as the object to be repaired, so that the distortion degree of the repair result of the object to be repaired is low, in other words, the candidate object to be repaired, the similarity of which does not meet the preset similarity threshold, is not repaired, thereby improving the evaluation accuracy and further improving the video frame repair effect.
In other examples, the repair assessment method further comprises: determining interframe distortion smoothing weights corresponding to the video frames, wherein the larger the interframe distortion smoothing weights are, the larger the interframe difference between the video frames and the adjacent video frames is; and determining a preset similarity threshold value positively correlated with the interframe distortion smoothing weight.
The larger the interframe distortion smoothing weight is, the larger the interframe difference between the video frame and the adjacent video frame is, the accuracy of the preset similarity threshold value is favorably determined, and the evaluation accuracy of the object to be repaired is further improved. In addition, the preset similarity threshold value is not required to be determined based on the prior data between the candidate object to be repaired and the object to be repaired.
Further, the air conditioner is provided with a fan,
in other examples, the inter-frame distortion smoothing weight represents a degree of pixel channel value dispersion between a video frame and an adjacent video frame. Determining the similarity between the candidate object to be repaired and the pre-repaired object thereof, including: and determining the discrete degree between the pixel channel value of the candidate object to be repaired and the pixel channel value of the object to be repaired as the similarity.
The preset similarity threshold value is positively correlated with the interframe distortion smoothing weight, and the discrete degree between the pixel channel value of the candidate object to be repaired and the pixel channel value of the pre-repaired object is used as the similarity, so that the evaluation accuracy of the object to be repaired is further improved.
Further, the air conditioner is characterized in that,
fig. 4 is a schematic block diagram of a video repair process in conjunction with the embodiments of fig. 2 and 3. In the video repair process shown in fig. 4, the processing is performed based on the video frame extraction module 410, the pre-repair module 421, the determination module 422, the repair module 423, and the video composition module 430.
First, video data is input to the video frame extraction module 410, and video frame data is extracted based on a time series.
Then, each video frame data is input to the pre-repair module 421 to pre-repair the candidate object to be repaired in each video frame, so as to obtain a pre-repaired object.
Then, each candidate object to be repaired and the object to be repaired are input into the determination module 422 for similarity calculation, and the object to be repaired is determined.
Then, the video frame is input to the repair module 423 to repair the object to be repaired, so as to obtain a repaired video frame.
Then, the repaired video frames are input into the video composition module 430, and video composition is performed on each repaired video frame based on the inter-frame relationship of each video frame, so as to obtain repaired video data.
The electronic device corresponding to fig. 2 and 3 will be described below with reference to fig. 5. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.
As shown in fig. 5, the electronic device may include: a processor (processor)502, a communication Interface 504, a memory 506 storing a program 510, and a communication bus 508.
The processor, the communication interface, and the memory communicate with each other via a communication bus.
And the communication interface is used for communicating with other electronic equipment or servers.
And the processor is used for executing the program, and particularly can execute the relevant steps in the method embodiment.
In particular, the program may include program code comprising computer operating instructions.
The processor may be a processor CPU, or an application Specific Integrated circuit (asic), or one or more Integrated circuits configured to implement embodiments of the present invention. The intelligent device comprises one or more processors which can be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
And the memory is used for storing programs. The memory may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The program may specifically be adapted to cause a processor to perform the following operations: pre-repairing an alternative object to be repaired in a video frame to obtain a pre-repaired object of the alternative object to be repaired; selecting an object to be repaired in the alternative objects to be repaired; and fusing based on the object to be repaired and the object to be repaired to obtain a repaired object.
Alternatively, the program may be specifically adapted to cause a processor to perform the following operations: pre-repairing an alternative object to be repaired in a video frame to obtain a pre-repaired object of the alternative object to be repaired; determining the similarity between the alternative object to be repaired and the pre-repaired object thereof; and determining the alternative object to be repaired with the similarity meeting a preset similarity threshold as the object to be repaired in the video frame.
In addition, for specific implementation of each step in the program, reference may be made to corresponding steps and corresponding descriptions in units in the foregoing method embodiments, which are not described herein again. It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described devices and modules may refer to the corresponding process descriptions in the foregoing method embodiments, and are not described herein again.
It should be noted that, according to the implementation requirement, each component/step described in the embodiment of the present invention may be divided into more components/steps, and two or more components/steps or partial operations of the components/steps may also be combined into a new component/step to achieve the purpose of the embodiment of the present invention.
The above-described method according to an embodiment of the present invention may be implemented in hardware, firmware, or as software or computer code storable in a recording medium such as a CD ROM, a RAM, a floppy disk, a hard disk, or a magneto-optical disk, or as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein may be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware such as an ASIC or FPGA. It will be appreciated that a computer, processor, microprocessor controller, or programmable hardware includes memory components (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code that, when accessed and executed by a computer, processor, or hardware, implements the methods described herein. Further, when a general-purpose computer accesses code for implementing the methods illustrated herein, execution of the code transforms the general-purpose computer into a special-purpose computer for performing the methods illustrated herein.
Those of ordinary skill in the art will appreciate that the various illustrative elements and method steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present embodiments.
The above embodiments are only for illustrating the embodiments of the present invention and not for limiting the embodiments of the present invention, and those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention, so that all equivalent technical solutions also belong to the scope of the embodiments of the present invention, and the scope of patent protection of the embodiments of the present invention should be defined by the claims.
Claims (12)
1. An object repair method, comprising:
pre-repairing an alternative object to be repaired in a video frame to obtain a pre-repaired object of the alternative object to be repaired;
selecting an object to be repaired in the alternative objects to be repaired;
and fusing based on the object to be repaired and the object to be repaired to obtain the repaired object.
2. The method according to claim 1, wherein the obtaining of the repair object based on the fusion of the object to be repaired and the object to be repaired, comprises:
and fusing the object to be repaired of the video frame and the object to be repaired of the video frame to obtain a repaired object based on the fusion weight, wherein the fusion weight is determined based on the initial fusion weight and the smooth weight.
3. The method of claim 2, wherein the method further comprises:
determining the smoothing weight based on at least one of an inter-frame correlation weight between a previous video frame of the video frames and the video frame, and an inter-frame distortion smoothing weight.
4. The method of claim 3, wherein the method further comprises:
and determining inter-frame distortion smoothing weight of the video frame based on the similarity between the object to be repaired of the video frame and the object to be repaired of the video frame.
5. The method of claim 4, wherein the determining inter-frame distortion smoothing weights for the video frame based on similarities between objects to be repaired of the video frame and pre-repaired objects thereof comprises:
when the similarity between an object to be repaired of the video frame and a pre-repaired object of the video frame is greater than a first similarity, determining interframe distortion smoothing weight of the video frame as a first interframe distortion smoothing weight threshold;
when the similarity between an object to be repaired of the video frame and a pre-repaired object of the video frame is between the first similarity and the second similarity, determining inter-frame distortion smoothing weight of the video frame as being positively correlated with the similarity;
and when the similarity between the object to be repaired of the video frame and the object to be repaired of the video frame is smaller than the second similarity, determining the interframe distortion smoothing weight of the video frame as a second interframe distortion smoothing weight threshold, wherein the value range of the similarity is between the first interframe distortion smoothing weight threshold and the second interframe distortion smoothing weight threshold.
6. The method of claim 3, wherein the method further comprises:
and determining the inter-frame related weight based on the weight of the candidate object to be repaired determined as the object to be repaired in each previous video frame of the video frames, wherein each previous video frame is stored in a buffer corresponding to the video frame.
7. The method of claim 3, wherein the method further comprises:
and determining each initial fusion weight between the object to be repaired of each previous video frame and the object to be repaired of each previous video frame based on the discrete degree of the inter-frame correlation weight.
8. A repair assessment method, comprising:
pre-repairing an alternative object to be repaired in a video frame to obtain a pre-repaired object of the alternative object to be repaired;
determining the similarity between the alternative object to be repaired and the pre-repaired object thereof;
and determining the alternative object to be repaired with the similarity meeting a preset similarity threshold as the object to be repaired in the video frame.
9. The method of claim 8, wherein the method further comprises:
determining an inter-frame distortion smoothing weight corresponding to the video frame, wherein the larger the inter-frame distortion smoothing weight is, the larger the inter-frame difference between the video frame and an adjacent video frame is;
and determining a preset similarity threshold positively correlated with the interframe distortion smoothing weight.
10. The method of claim 9, wherein the inter-frame distortion smoothing weight represents a degree of pixel channel value dispersion between the video frame and an adjacent video frame;
the determining the similarity between the candidate object to be repaired and the object to be repaired before repairing comprises:
and determining the discrete degree between the pixel channel value of the candidate object to be repaired and the pixel channel value of the object to be repaired as the similarity.
11. An electronic device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus; the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the corresponding operation of the method according to any one of claims 1-10.
12. A computer storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210681242.6A CN114913468A (en) | 2022-06-16 | 2022-06-16 | Object repairing method, repair evaluating method, electronic device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210681242.6A CN114913468A (en) | 2022-06-16 | 2022-06-16 | Object repairing method, repair evaluating method, electronic device, and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114913468A true CN114913468A (en) | 2022-08-16 |
Family
ID=82770953
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210681242.6A Pending CN114913468A (en) | 2022-06-16 | 2022-06-16 | Object repairing method, repair evaluating method, electronic device, and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114913468A (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110782445A (en) * | 2019-10-25 | 2020-02-11 | 北京华捷艾米科技有限公司 | No-reference image quality evaluation method and system |
| CN110910331A (en) * | 2019-11-29 | 2020-03-24 | Oppo广东移动通信有限公司 | Image processing method, image processing apparatus, electronic device, and computer-readable storage medium |
| CN111553865A (en) * | 2020-04-30 | 2020-08-18 | 深圳市商汤科技有限公司 | Image restoration method and device, electronic equipment and storage medium |
| CN111553864A (en) * | 2020-04-30 | 2020-08-18 | 深圳市商汤科技有限公司 | Image restoration method and device, electronic equipment and storage medium |
| CN111696051A (en) * | 2020-05-14 | 2020-09-22 | 维沃移动通信有限公司 | Portrait restoration method and electronic equipment |
| CN112634170A (en) * | 2020-12-30 | 2021-04-09 | 平安科技（深圳）有限公司 | Blurred image correction method and device, computer equipment and storage medium |
| CN113066034A (en) * | 2021-04-21 | 2021-07-02 | 腾讯科技（深圳）有限公司 | Face image restoration method and device, restoration model, medium and equipment |
| WO2021169740A1 (en) * | 2020-02-28 | 2021-09-02 | Oppo广东移动通信有限公司 | Image restoration method and apparatus, computer device, and storage medium |
| WO2021233032A1 (en) * | 2020-05-19 | 2021-11-25 | Oppo广东移动通信有限公司 | Video processing method, video processing apparatus, and electronic device |
| KR20220046808A (en) * | 2020-10-08 | 2022-04-15 | 에스케이텔레콤 주식회사 | Method and Apparatus for Video Inpainting |
-
2022
- 2022-06-16 CN CN202210681242.6A patent/CN114913468A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110782445A (en) * | 2019-10-25 | 2020-02-11 | 北京华捷艾米科技有限公司 | No-reference image quality evaluation method and system |
| CN110910331A (en) * | 2019-11-29 | 2020-03-24 | Oppo广东移动通信有限公司 | Image processing method, image processing apparatus, electronic device, and computer-readable storage medium |
| WO2021169740A1 (en) * | 2020-02-28 | 2021-09-02 | Oppo广东移动通信有限公司 | Image restoration method and apparatus, computer device, and storage medium |
| CN111553865A (en) * | 2020-04-30 | 2020-08-18 | 深圳市商汤科技有限公司 | Image restoration method and device, electronic equipment and storage medium |
| CN111553864A (en) * | 2020-04-30 | 2020-08-18 | 深圳市商汤科技有限公司 | Image restoration method and device, electronic equipment and storage medium |
| CN111696051A (en) * | 2020-05-14 | 2020-09-22 | 维沃移动通信有限公司 | Portrait restoration method and electronic equipment |
| WO2021233032A1 (en) * | 2020-05-19 | 2021-11-25 | Oppo广东移动通信有限公司 | Video processing method, video processing apparatus, and electronic device |
| KR20220046808A (en) * | 2020-10-08 | 2022-04-15 | 에스케이텔레콤 주식회사 | Method and Apparatus for Video Inpainting |
| CN112634170A (en) * | 2020-12-30 | 2021-04-09 | 平安科技（深圳）有限公司 | Blurred image correction method and device, computer equipment and storage medium |
| CN113066034A (en) * | 2021-04-21 | 2021-07-02 | 腾讯科技（深圳）有限公司 | Face image restoration method and device, restoration model, medium and equipment |
Non-Patent Citations (1)
| Title |
|---|
| 曹燕;金炜;符冉迪;: "基于加权优先级和分类匹配的图像修复方法", 电信科学, no. 04, 20 April 2017 (2017-04-20) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110533097B (en) | Image definition recognition method and device, electronic equipment and storage medium | |
| CN110211119B (en) | Image quality evaluation method and device, electronic equipment and readable storage medium | |
| WO2020052270A1 (en) | Video review method and apparatus, and device | |
| CN113903036B (en) | Text recognition method and device, electronic equipment, medium and product | |
| CN112950626A (en) | Definition determination method, medium, device and computing equipment | |
| CN109035257A (en) | portrait dividing method, device and equipment | |
| CN111814749A (en) | Human body feature point screening method and device, electronic equipment and storage medium | |
| CN113780578B (en) | Model training method, device, electronic equipment and readable storage medium | |
| CN112966754B (en) | Sample screening method, sample screening device and terminal equipment | |
| CN114170558A (en) | Method, system, device, medium and article for video processing | |
| CN108509876B (en) | Object detection method, device, apparatus, storage medium, and program for video | |
| CN113963149A (en) | Medical bill picture fuzzy judgment method, system, equipment and medium | |
| CN117253110A (en) | Diffusion model-based target detection model generalization capability improving method | |
| CN112949649A (en) | Text image identification method and device and computing equipment | |
| CN117095019A (en) | Image segmentation method and related device | |
| CN114913468A (en) | Object repairing method, repair evaluating method, electronic device, and storage medium | |
| CN111435445A (en) | Training method and device of character recognition model and character recognition method and device | |
| CN114549904B (en) | Visual processing and model training method, device, storage medium and program product | |
| CN113628192B (en) | Image blur detection method, apparatus, device, storage medium, and program product | |
| CN115439734A (en) | Quality evaluation model training method and device, electronic equipment and storage medium | |
| CN115080770A (en) | Multimedia data processing method and device, electronic equipment and readable storage medium | |
| CN114222181A (en) | Image processing method, device, equipment and medium | |
| CN113642443A (en) | Model testing method and device, electronic equipment and storage medium | |
| CN108875630B (en) | Moving target detection method based on video in rainy environment | |
| CN113011409A (en) | Image identification method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |