CN115396743A - Video watermark removing method, device, equipment and storage medium - Google Patents
Video watermark removing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN115396743A CN115396743A CN202211032315.5A CN202211032315A CN115396743A CN 115396743 A CN115396743 A CN 115396743A CN 202211032315 A CN202211032315 A CN 202211032315A CN 115396743 A CN115396743 A CN 115396743A
- Authority
- CN
- China
- Prior art keywords
- image frame
- image
- repaired
- frame
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/835—Generation of protective data, e.g. certificates
- H04N21/8358—Generation of protective data, e.g. certificates involving watermark
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
Abstract
The method comprises the steps of performing framing processing on a video to be processed to obtain an image frame set, performing primary repairing on the image frame set to obtain a primary image set, calculating a score difference value of each frame and a first frame in the primary image set, using a frame with the score difference value exceeding a first preset threshold value as an image frame to be repaired to obtain the image frame set to be repaired, using any image frame to be repaired in the image frame set to be repaired as a second reference image frame, performing deep repairing on the second reference image frame to obtain a deep repair image frame, calculating the similarity between each frame and the second reference image frame in the image frame set to be repaired, filling repairing contents of the deep repair image frame into the corresponding image frame if the similarity exceeds the second preset threshold value, and performing deep repairing on the corresponding image frame if the similarity is lower than the second preset threshold value. The invention reduces the time for removing the watermark from the video and lightens the video flicker condition.
Description
Technical Field
The present application relates to the field of video processing, and in particular, to a method, an apparatus, a device, and a storage medium for removing a watermark from a video.
Background
With the wide popularization of shooting devices (mobile phones, digital cameras and the like), the great reduction of shooting cost and the rise of short video content sharing platforms, videos play an increasingly important spreading form in the life of users. The method has the advantages that various watermarks are superposed on the video in the transmission process, so that the watching experience of a user is greatly reduced, how to effectively remove the watermark Logo in the video and improve the watching experience of the user become a problem to be solved urgently in the field of video processing.
The current video watermark removing technology can be divided into two categories, namely a traditional algorithm and a deep learning algorithm, and is particularly roughly divided into the following categories: (1) And (4) carrying out interpolation filling on the watermark area by using pixels around the watermark mask. For example, a delay function in ffmpeg, a fuzzy filter is added to the position of the video watermark mask, and logo is removed through simple interpolation of surrounding pixels; (2) Directly carrying out fuzzy processing on the watermark area to ensure that the watermark cannot be seen clearly; (3) The watermark region is repaired and filled by a repairing algorithm based on deep learning, so that the integral texture and structure consistency or the semantic and visual credibility is achieved, and the watermark of the video is removed from the video frame by applying the image repairing algorithm. Although the scheme of filling by an interpolation technology based on surrounding pixels can remove watermarks, the generated interpolation effect is too obvious, color change has great influence, while the scheme of blurring the watermark region can remove watermarks to a certain extent and keep smooth transition of surrounding pictures, but as only blurring is performed, when the blurring degree is small, the watermark pattern is still visible, and the problem of incomplete watermark removal exists, and if the blurring degree is too large, the smooth transition cannot be ensured, although the repairing effect of a single frame of a frame repairing algorithm for deep learning is obviously better than that of a traditional algorithm, the condition of flicker between frames occurs in video application, and the video de-watermarking effect is poor. There is a need for a method for de-watermarking video with good effect, fast speed and no flicker in the repair area.
Disclosure of Invention
The embodiment of the application aims to provide a video watermarking removing method, a video watermarking removing device, video watermarking removing equipment and a storage medium, so that video watermarking removing is achieved, the time of video watermarking removing is shortened, and flicker of a repair area is relieved.
In order to solve the above technical problem, an embodiment of the present application provides a method for removing a watermark from a video, including:
the method comprises the steps of obtaining a video to be processed, and performing framing processing on the video to be processed to obtain an image frame set, wherein the obtained video to be processed comprises a smearing area, and the position of the smearing area in each image frame of the image frame set is fixed;
preliminarily repairing the smearing region of each image frame in the image frame set through a preset algorithm to obtain a preliminary image set;
taking a first frame image frame in the preliminary image set as a first reference image frame, and calculating a score difference value of each image frame in the preliminary image set and the first reference image frame to obtain a difference value score set;
comparing each difference score in the difference score set with a first preset threshold, and taking the image frame corresponding to the difference score exceeding the first preset threshold as an image frame to be repaired to obtain an image frame set to be repaired;
taking any image frame in the image frame set to be repaired as a second reference image frame, and performing deep repair on the smearing area of the second reference image frame to obtain a deep repair image frame, wherein the deep repair image frame comprises repair content of the smearing area;
calculating the similarity between each image frame to be repaired in the image frame set to be repaired and the second reference image frame;
judging whether the similarity exceeds a second preset threshold, if so, taking the image frame to be repaired, which corresponds to the similarity exceeding the second preset threshold, as a first repaired image, and filling the repaired content of the second reference image frame into the smearing area of the first repaired image;
and if the similarity is lower than the second preset threshold, taking the image frame to be repaired, corresponding to the similarity lower than the second preset threshold, as a second repaired image, and performing the depth repair on the smearing area of the second repaired image.
In order to solve the foregoing technical problem, an embodiment of the present application provides an apparatus for removing a watermark from a video, including:
the device comprises a mask generation module, a frame segmentation module and a frame segmentation module, wherein the mask generation module is used for acquiring a video to be processed and performing frame segmentation on the video to be processed to obtain an image frame set, the acquired video to be processed comprises an smearing region, and the position of the smearing region in each image frame of the image frame set is fixed;
the preliminary repairing module is used for preliminarily repairing the smearing area of each image frame in the image frame set through a preset algorithm to obtain a preliminary image set;
a difference value calculating module, configured to use a first frame image frame in the preliminary image set as a first reference image frame, and calculate a score difference value between each image frame in the preliminary image set and the first reference image frame to obtain a difference value score set;
the difference value comparison module is used for comparing each difference value score in the difference value score set with a first preset threshold value, and taking the image frame corresponding to the difference value score exceeding the first preset threshold value as an image frame to be repaired to obtain an image frame set to be repaired;
the sample repairing module is used for taking any image frame in the image frame set to be repaired as a second reference image frame and performing deep repairing on the smearing area of the second reference image frame to obtain a deep repairing image frame, wherein the deep repairing image frame comprises repairing contents of the smearing area;
the similarity calculation module is used for calculating the similarity between each image frame to be repaired in the image frame set to be repaired and the second reference image frame;
the filling module is used for judging whether the similarity exceeds a second preset threshold value, if so, taking the image frame to be repaired, which corresponds to the similarity exceeding the second preset threshold value, as a first repaired image, and filling the repaired content of the second reference image frame into the smearing area of the first repaired image;
and the depth repairing module is used for taking the image frame to be repaired, corresponding to the similarity lower than the second preset threshold value, as a second repaired image and performing the depth repairing on the smearing area of the second repaired image if the similarity is lower than the second preset threshold value.
In order to solve the technical problems, the invention adopts a technical scheme that: a computer device comprising a memory having a computer program stored therein and a processor that when executed implements a video de-watermarking method as defined in any of the above.
In order to solve the technical problems, the invention adopts a technical scheme that: a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a video watermarking method as recited in any of the above.
The embodiment of the invention provides a video watermark removing method, device, equipment and storage medium. According to the embodiment of the invention, the frame division processing is carried out on the video to be processed to obtain an image frame set, the image frame set is subjected to primary restoration to obtain a primary image set, the score difference value of each frame and the first frame in the primary image set is calculated, the frame corresponding to the score difference value exceeding a first preset threshold value is taken as the image frame to be restored to obtain the image frame set to be restored, and the processing time for removing the watermark from the video is reduced; taking any image frame to be repaired in the image frame set to be repaired as a second reference image frame, performing depth repair on the second reference image frame to obtain a depth repair image frame, calculating the similarity between each frame in the image frame set to be repaired and the second reference image frame, filling the repair content of the depth repair image frame into the corresponding image frame if the similarity exceeds a second preset threshold, and performing depth repair on the corresponding image frame if the similarity is lower than the second preset threshold. According to the embodiment of the invention, the image to be repaired is preliminarily repaired, then the image to be repaired is evaluated and scored, the image frame corresponding to the score difference value exceeding the first preset threshold value is screened out, the similarity between the image frame corresponding to the score difference value exceeding the first preset threshold value and the second repaired image is calculated, the image frame to be repaired corresponding to the similarity exceeding the second preset threshold value is taken as the first repaired image, the repaired content of the second reference image frame is filled in the smearing area of the first repaired image, the image frame to be repaired corresponding to the similarity lower than the second preset threshold value is taken as the second repaired image, the smearing area of the second repaired image is deeply repaired, each frame of a video is prevented from being repaired, the time for removing the watermark of the video is shortened, the flicker condition of the video repairing area is reduced, and the watching experience of a user is improved.
Drawings
In order to more clearly illustrate the solution of the present application, the drawings used in the description of the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the description below are some embodiments of the present application, and that other drawings may be obtained by those skilled in the art without inventive effort.
Fig. 1 is a flowchart of an implementation of a video watermarking removing method provided in an embodiment of the present application;
FIG. 2 is a flowchart illustrating another implementation of a sub-process in a video watermarking removing method according to an embodiment of the present disclosure;
FIG. 3 is a flowchart of another implementation of a sub-process in a video watermarking removing method according to an embodiment of the present application;
FIG. 4 is a flowchart of another implementation of a sub-process in a video watermarking removing method according to an embodiment of the present application;
FIG. 5 is a flowchart illustrating another implementation of a sub-process in a video watermarking removing method according to an embodiment of the present disclosure;
FIG. 6 is a flowchart of another implementation of a sub-process in a video watermarking removing method according to an embodiment of the present application;
FIG. 7 is a flowchart of another implementation of a sub-process in a video watermarking removing method according to an embodiment of the present application;
FIG. 8 is a schematic diagram of a video watermarking apparatus according to an embodiment of the present application;
fig. 9 is a schematic diagram of a computer device provided in an embodiment of the present application.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "including" and "having," and any variations thereof, in the description and claims of this application and the description of the above figures are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or in the foregoing drawings are used for distinguishing between different objects and not for describing a particular sequential order.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein may be combined with other embodiments.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
The present invention will be described in detail below with reference to the accompanying drawings and embodiments.
It should be noted that the video watermarking removing method provided by the embodiment of the present application is generally executed by a server, and accordingly, the video watermarking removing apparatus is generally configured in the server.
Referring to fig. 1, fig. 1 shows an embodiment of a video watermarking method.
It should be noted that, if the result is substantially the same, the method of the present invention is not limited to the flow sequence shown in fig. 1, and the method includes the following steps:
s1: the method comprises the steps of obtaining a video to be processed, and performing framing processing on the video to be processed to obtain an image frame set, wherein the video to be processed comprises a smearing area, and the position of the smearing area in each image frame of the image frame set is fixed.
Specifically, a user side manually paints a to-be-repaired area on a to-be-processed video to generate a painting area including the to-be-repaired area, obtains the to-be-processed video obtained after manual painting, and frames the to-be-processed video. The video to be processed is obtained by the server from the local database, and it can be understood that the video to be processed may also be obtained in other manners, for example, the user side sends the video to the server, where no limitation is imposed on the data source of the video to be processed.
Referring to fig. 2, fig. 2 shows an embodiment of step S1, which is described in detail as follows:
s11: and acquiring an initial video, and returning the video to be processed to the user side so that the user side paints the watermark region of the video to be processed to obtain the video to be processed.
S12: and performing frame division processing on the video to be processed to obtain an image frame set.
Specifically, the server acquires a video to be processed from the database and sends the video to the client, the client manually paints the video to be processed after receiving the video to be processed, then sends the painted video to be processed to the server, performs framing processing on the video to be processed, and sequentially saves each frame of image. The user side coats the watermark area of the video to be processed, so that the watermark area is blurred, the visibility of the watermark is reduced, and the area of the video to be repaired is separated from other areas, so that the identification and the repair are facilitated.
S2: and carrying out preliminary repair on the smearing area of each image frame in the image frame set through a preset algorithm to obtain a preliminary image set.
Specifically, the embodiment performs the initial repair by using FFmpeg software, which provides a delay filter to remove the watermark by simply interpolating pixels around the watermark region. It should be noted that, the conventional algorithm FFmpeg delay is better in repairing effect on a part of frames, the processing speed of the FFmpeg delay is greatly better than that of other algorithms, and the FFmpeg delay algorithm is introduced, so that certain processing time is reduced.
Referring to fig. 3, fig. 3 shows an embodiment of step S2, which is described in detail as follows:
s21: for any image frame in the image frame set, a smearing region in each image frame is identified, and a rectangular region is constructed based on the smearing region, wherein the rectangular region comprises the smearing region.
Specifically, a rectangular frame including the smearing area is set according to the vertex coordinates of the upper left corner and the width and height of the rectangle. It should be noted that when the repair is completed, the rectangular frame is automatically deleted by default, but the rectangular frame may be retained by changing the parameter setting.
S22: and acquiring pixels except the smearing area in the rectangular frame as pixels to be filled.
And S23, filling the pixel to be filled into the smearing area to obtain a primary repairing image.
And S24, obtaining a preliminary image set when all the image frames in the image frame set are repaired.
Specifically, pixels in the rectangular frame except the smearing area are obtained and serve as pixels to be filled, the pixels to be filled are filled into the smearing area, and when each frame in the image frame set is repaired, a preliminary image set is obtained.
S3: and taking the first frame image frame in the preliminary image set as a first reference image frame, and calculating the score difference value of each image frame in the preliminary image set and the first reference image frame to obtain a difference value score set.
Specifically, a threshold reference is determined according to the image quality of the first frame, then a preset evaluation scoring algorithm is adopted to evaluate and score each image frame, each image frame obtains a corresponding score value, the difference between the score value corresponding to each image frame and the score value corresponding to the first image frame is calculated respectively, and a difference value score set is obtained.
Referring to fig. 4, fig. 4 shows an embodiment of step S3, which is described in detail as follows:
s31: and taking the first frame image frame in the preliminary image set as a first reference image frame.
And S32, adopting a preset evaluation scoring algorithm to evaluate and score the preliminary image set to obtain the score of each image frame.
S33: and calculating the difference between the score of each image frame and the score of the first reference image frame to obtain a score difference value.
Specifically, in the embodiment, a MetaIQA algorithm is used for evaluating and scoring the image frames, and the MetaIQA algorithm learns the shared quality prior model through a series of NR-IQA tasks of known distortion types by using a meta-learning technique, and then fine-tunes the quality prior model by using the NR-IQA task of unknown distortion. The generalization was studied by the double-layer gradient descent method. In the first step, firstly, the training data of NR-IQA is divided into a support set and a query set; secondly, calculating the gradient of the model parameters by using a support set, and trying to update by using SGD; finally, the query set is used to verify whether the updated model is executing efficiently. The quality prior model training is carried out on a series of digital image quality evaluation tasks with specific distortion, then the model is used as prior knowledge to carry out fine adjustment on the digital image quality evaluation tasks with unknown distortion, finally, a quality evaluation model is obtained, the quality evaluation model is used for evaluating and scoring a preliminary image set to obtain the score of each image frame, the difference value of each image frame score and the first image frame score is calculated, and a difference value score set is obtained.
It should be noted that the embodiment adopts the MetaIQA algorithm to evaluate and score the image frames, or can be replaced by other conventional indexes or algorithms, for example, the Brisque algorithm can also be adopted, the Brisque algorithm principle is to extract an average subtraction contrast normalization coefficient MSCN from the images, then fit the average subtraction contrast normalization coefficient MSCN into an asymmetric generalized gaussian distribution AGGD, extract the features of the fitted high-speed distribution, input the features into the support vector machine SVM to perform regression, thereby obtaining the evaluation result of the image quality, and the specific method for evaluating and scoring the image frames is not limited herein.
S4: and comparing each difference score in the difference score set with a first preset threshold, and taking the image frame corresponding to the difference score exceeding the first preset threshold as the image frame to be repaired to obtain the image frame set to be repaired.
Specifically, if the difference between a certain image frame score and the first image frame score exceeds a first preset threshold, it indicates that the delaogo repair effect is poor and further repair is needed, and if the difference between a certain image frame score and the first image frame score does not exceed the threshold, it indicates that the delaogo repair effect is good and further repair is not needed. It should be noted that the first preset threshold is set according to actual conditions, and the value of the first preset threshold is not limited herein. In a specific embodiment, the first predetermined threshold is 0.3.
Furthermore, the image is restored by the delaogo at a high speed, and image frames with good effect after being restored by the delaogo are not further restored by adopting a mode of combining MetaIQA and the delaogo, so that certain processing time is reduced.
S5: any image frame in the image frame set to be repaired is used as a second reference image frame, and the smearing area of the second reference image frame is subjected to deep repair to obtain a deep repair image frame, wherein the deep repair image frame comprises the repair content of the smearing area.
Specifically, the method adopted for deep repair in this embodiment is large mask repair LaMa, the method has a high receptive field and a large training mask, and is used for sensing loss and releasing the potential of the front component, and a repair network architecture of fast fourier convolution FFC is used, the architecture has a wide image receptive field, and can be popularized to a high-resolution image after training on low-resolution data, and can capture and generate a complex periodic structure, and has robustness to the large mask.
Referring to fig. 5, fig. 5 shows an embodiment of step S5, which is described in detail as follows:
s51: taking any image frame in the image frame set to be repaired as a second reference image frame, and performing negation processing on the smearing area of the second reference image frame to obtain a negated smearing image frame, wherein the negated smearing image frame comprises the negated smearing area.
S52: and multiplying the reversed smearing image frame and the second reference image frame to obtain a first target image, wherein the first target image comprises the reversed smearing area.
S53: and overlapping the first target image and the smearing area to obtain a second target image.
Specifically, a second reference image frame and a smear region image of the second reference image frame are obtained, the smear region image of the second reference image frame is subjected to negation processing, that is, the smear region image is subjected to gray processing to obtain a gray image, an original black pixel in the gray image is replaced by a white pixel, the original white pixel is replaced by a black pixel, the original white pixel is multiplied by the second reference image frame image to obtain a second reference image frame image with a smear region, and then the second reference image frame image with the smear region and the smear region image of the second reference image frame are subjected to superposition processing based on a channel to obtain a second target image.
S54: and repairing the second target image to obtain a depth repair image frame.
Referring to fig. 6, fig. 6 shows an embodiment of step S54, which is described in detail as follows:
s541: and performing down-sampling operation on the second target image to obtain a down-sampled image.
S542: and performing fast Fourier She Juanji FFC processing on the down-sampled image to obtain a convolution image.
S543: and performing up-sampling operation on the convolution image to obtain a depth restoration image frame.
Specifically, in the processing of the fast fourier She Juanji FFC, the second target image is divided into 2 parts based on the channel, and 2 different branches are respectively taken. One branch is responsible for extracting local information, called a local branch. The other branch is responsible for extracting global information, called global branch. In the global branch, the FFC is used to extract global features. And finally, performing cross fusion on the local information and the global information, and splicing based on channels to obtain a final output result depth repair image frame.
S6: and calculating the similarity between each image frame to be repaired in the image frame set to be repaired and the second reference image frame.
Specifically, in this embodiment, the image similarity is evaluated by using a structural similarity metric SSIM, which measures the image similarity from three aspects of brightness, contrast, and structure. It should be noted that the local SSIM index is better than global, because the statistical features of the image are usually distributed unevenly in the space, the distortion of the image is also variable in the space, and in the normal viewing distance, people can focus the sight on only one region of the image, so the local processing is more consistent with the characteristics of the human visual system.
Referring to fig. 7, fig. 7 shows an embodiment of step S6, which is described in detail as follows:
s61: and carrying out blocking processing on any image frame to be repaired and the second image frame to obtain a first block and a second block.
S62, respectively calculating the first block and the second block by adopting a Gaussian weighting algorithm to obtain a first calculation result and a second calculation result, wherein the first calculation result comprises the mean value, the variance and the covariance of the first block, and the second calculation result comprises the mean value, the variance and the covariance of the second block.
S63: calculating the structural similarity of the first block according to the first calculation result to obtain the similarity of the first block; and calculating the structural similarity of the second block according to the second calculation result to obtain the similarity of the second block.
S64: and calculating the average value of the structural similarity values of the first block and the second block to obtain the similarity between any image frame to be repaired and the second image frame.
Specifically, the images in the image frame set to be restored are blocked by using a Gaussian sliding window, then the mean value, the variance and the covariance of each window are calculated by adopting a Gaussian weighting algorithm, then the structural similarity SSIM of the corresponding block is calculated, and finally the mean value is used as the structural similarity measurement of the two images. The structural similarity SSIM is calculated according to the formula:
wherein X, Y are images and X i ,y j MN is the number of local windows for the location of the local SSIM index in the map.
It should be noted that in this embodiment, structural similarity SSIM is used to evaluate the image similarity, and other image similarity evaluation indexes may also be used to measure inter-frame similarity, for example, mean square error MSE is used to evaluate the similarity of images, and a specific method for evaluating the image similarity is not limited herein.
S7: and judging whether the similarity exceeds a second preset threshold, if so, taking the image frame to be repaired corresponding to the similarity exceeding the second preset threshold as a first repaired image, and filling the repaired content of the second reference image frame into the smearing area of the first repaired image.
Specifically, if the similarity exceeds a second preset threshold, it is determined that the similarity of the two frame repair regions is high, and frame skipping processing can be performed, that is, the frame to be repaired is directly filled with the repair content of the current frame, and image repair is not required to be performed by adopting a deep repair Lama algorithm. It should be noted that the second preset threshold is set according to actual conditions, and the value of the second preset threshold is not limited herein. In one embodiment, the second predetermined threshold is 0.05.
Furthermore, the frame skipping processing avoids deep repair of each frame, so that the processing time of removing the watermark of the video is greatly reduced, and the flicker condition of a video repair area is reduced to a certain extent, thereby greatly improving the watching experience of a user.
S8: and if the similarity is lower than a second preset threshold, taking the image frame to be repaired, corresponding to the similarity lower than the second preset threshold, as a second repair image, and performing deep repair on the smearing area of the second repair image.
Specifically, if the similarity is lower than the second preset threshold, it is determined that the similarity of the two repair regions is low, and it is necessary to perform depth repair on the current frame, where the depth repair is performed by using a large mask repair LaMa, which has been explained in the above steps and is not described herein again.
In the embodiment, a video to be processed is obtained, and the video to be processed is subjected to framing processing to obtain an image frame set, wherein the video to be processed comprises a smearing region, and the position of the smearing region in each image frame of the image frame set is fixed; preliminarily repairing the smearing area of each image frame in the image frame set through a preset algorithm to obtain a preliminary image set; taking the first frame image frame in the preliminary image set as a first reference image frame, and calculating the score difference value of each image frame in the preliminary image set and the first reference image frame to obtain a difference value score set; comparing each difference score in the difference score set with a first preset threshold, and taking an image frame corresponding to the difference score exceeding the first preset threshold as an image frame to be repaired to obtain an image frame set to be repaired; any image frame in the image frame set to be repaired is used as a second reference image frame, and the smearing area of the second reference image frame is subjected to deep repair to obtain a deep repair image frame, wherein the deep repair image frame comprises the repair content of the smearing area; calculating the similarity between each image frame to be repaired in the image frame set to be repaired and a second reference image frame; judging whether the similarity exceeds a second preset threshold, if so, taking the image frame to be repaired, which corresponds to the similarity exceeding the second preset threshold, as a first repaired image, and filling the repaired content of a second reference image frame into the smearing area of the first repaired image; and if the similarity is lower than a second preset threshold, taking the image frame to be repaired, corresponding to the similarity lower than the second preset threshold, as a second repair image, and performing deep repair on the smearing area of the second repair image. In the embodiment of the invention, the frames with better effect after some initial repairs are not subjected to depth repairs, and in addition, on the basis of depth repairs, a frame skipping strategy is adopted for some frames with higher similarity to avoid processing each frame, so that the processing time for removing the watermark from the video can be greatly reduced; meanwhile, the frame skipping strategy avoids repairing each frame, and the flicker condition of a video repairing area is reduced to a certain extent, so that the watching experience of a user is greatly improved.
Referring to fig. 8, as an implementation of the method shown in fig. 1, the present application provides an embodiment of a video watermarking removing apparatus, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 1, and the apparatus can be applied to various electronic devices.
As shown in fig. 8, the video watermarking apparatus of the present embodiment includes: a mask generation module 81, a preliminary restoration module 82, a difference calculation module 83, a difference comparison module 84, a sample restoration module 85, a similarity calculation module 86, a filling module 87, and a depth restoration module 88, wherein:
the mask generation module is used for acquiring a video to be processed and performing framing processing on the video to be processed to obtain an image frame set, wherein the acquired video to be processed comprises a smearing area, and the position of the smearing area in each image frame of the image frame set is fixed;
the preliminary repairing module is used for carrying out preliminary repairing on the smearing area of each image frame in the image frame set through a preset algorithm to obtain a preliminary image set;
the difference value calculating module is used for taking the first frame image frame in the preliminary image set as a first reference image frame, calculating the score difference value of each image frame in the preliminary image set and the first reference image frame to obtain a difference value score set;
the difference value comparison module is used for comparing each difference value score in the difference value score set with a first preset threshold value, and taking the image frame corresponding to the difference value score exceeding the first preset threshold value as an image frame to be repaired to obtain an image frame set to be repaired;
the sample restoration module is used for taking any image frame in the image frame set to be restored as a second reference image frame and carrying out deep restoration on the smearing area of the second reference image frame to obtain a deep restoration image frame, wherein the deep restoration image frame comprises restoration content of the smearing area;
the similarity calculation module is used for calculating the similarity between each image frame to be repaired in the image frame set to be repaired and the second reference image frame;
the filling module is used for judging whether the similarity exceeds a second preset threshold, if so, taking the image frame to be repaired, corresponding to the similarity exceeding the second preset threshold, as a first repaired image, and filling the repaired content of the second reference image frame into a smearing area of the first repaired image;
and the depth repairing module is used for taking the image frame to be repaired, which has the similarity lower than the second preset threshold value, as a second repaired image and performing depth repairing on the smearing area of the second repaired image if the similarity is lower than the second preset threshold value.
Further, the mask generating module 81 includes:
the video acquisition unit is used for acquiring an initial video and returning the video to be processed to the user side so that the user side can smear the watermark area of the video to be processed to obtain the video to be processed;
and the framing unit is used for performing framing processing on the video to be processed to obtain an image frame set.
Further, the preliminary repair module 82 includes:
the image processing device comprises a rectangular construction unit, a processing unit and a processing unit, wherein the rectangular construction unit is used for identifying a smearing region in each image frame aiming at any image frame in an image frame set and constructing a rectangular region based on the smearing region, and the rectangular region comprises the smearing region;
the pixel obtaining unit is used for obtaining pixels except for the smearing area in the rectangular frame and taking the pixels as pixels to be filled;
the pixel filling unit is used for filling pixels to be filled into the smearing area to obtain a primary repairing image;
and the preliminary image set unit is used for obtaining a preliminary image set when all the image frames in the image frame set are repaired.
Further, the difference calculation module 83 includes:
the first frame reference unit is used for taking a first frame image frame in the preliminary image set as a first reference image frame;
the evaluation scoring unit is used for evaluating and scoring the preliminary image set by adopting a preset evaluation scoring algorithm to obtain the score of each image frame;
and the score difference unit is used for calculating the difference between the score of each image frame and the score of the first reference image frame to obtain the score difference.
Further, the difference comparison module 84 includes:
the comparison unit is used for comparing each difference score in the difference score set with a first preset threshold value;
and the to-be-repaired unit is used for taking the image frame corresponding to the difference value exceeding the first preset threshold value as the image frame to be repaired to obtain the image frame set to be repaired.
Further, the sample repairing module 85 includes:
the negation unit is used for taking any image frame in the image frame set to be repaired as a second reference image frame and negating the smearing area of the second reference image frame to obtain a negated smearing image frame, wherein the negated smearing image frame comprises the smeared area after negation;
the multiplying unit is used for multiplying the reversed smearing image frame and the second reference image frame to obtain a first target image, wherein the first target image comprises a reversed smearing area;
the overlapping unit is used for overlapping the first target image and the smearing area to obtain a second target image;
and the restoration unit is used for restoring the second target image to obtain a depth restoration image frame.
Further, the similarity calculation module 86 includes:
the image restoration device comprises a blocking unit, a restoration unit and a restoration unit, wherein the blocking unit is used for carrying out blocking processing on any image frame to be restored and a second image frame to obtain a first block and a second block;
the calculating unit is used for respectively calculating the first block and the second block by adopting a Gaussian weighting algorithm to obtain a first calculation result and a second calculation result, wherein the first calculation result comprises the mean value, the variance and the covariance of the first block, and the second calculation result comprises the mean value, the variance and the covariance of the second block;
the block similarity unit is used for calculating the structural similarity of the first block according to the first calculation result to obtain the first block similarity; calculating the structural similarity of the second block according to the second calculation result to obtain the similarity of the second block;
and the average similarity unit is used for calculating the average value of the structural similarity values of the first block and the second block to obtain the similarity between any image frame to be repaired and the second image frame.
Further, the padding module 87 includes:
the judging unit is used for judging whether the similarity exceeds a second preset threshold value or not, and if the similarity exceeds the second preset threshold value, the image frame to be repaired corresponding to the similarity exceeding the second preset threshold value is used as a first repair image;
and the filling and repairing unit is used for filling the repairing content of the second reference image frame into the smearing area of the first repairing image.
Further, the depth repair module includes:
the similarity judging unit is used for taking the image frame to be repaired, which corresponds to the similarity lower than the second preset threshold value, as a second repair image if the similarity is lower than the second preset threshold value;
and the second repairing unit is used for performing the depth repairing on the smearing area of the second repairing image.
In order to solve the technical problem, an embodiment of the present application further provides a computer device. Referring to fig. 9, fig. 9 is a block diagram of a basic structure of a computer device according to the present embodiment.
The computer device 9 includes a memory 91, a processor 92, and a network interface 93 communicatively connected to each other via a system bus. It is noted that only the computer device 9 having three components memory 91, processor 92, network interface 93 is shown, but it is understood that not all of the shown components are required to be implemented, and more or fewer components may be implemented instead. As will be understood by those skilled in the art, the computer device is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The computer device may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The computer equipment can carry out man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch panel or voice control equipment and the like.
The memory 91 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 91 may be an internal storage unit of the computer device 9, such as a hard disk or a memory of the computer device 9. In other embodiments, the memory 91 may also be an external storage device of the computer device 9, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the computer device 9. Of course, the memory 91 may also comprise both an internal storage unit of the computer device 9 and an external storage device thereof. In this embodiment, the memory 91 is generally used for storing an operating system installed in the computer device 9 and various application software, such as program codes of a video watermarking method. Further, the memory 91 can also be used to temporarily store various types of data that have been output or are to be output.
The network interface 93 may include a wireless network interface or a wired network interface, and the network interface 93 is generally used to establish a communication connection between the computer device 9 and other electronic devices.
The present application further provides another embodiment, which is to provide a computer-readable storage medium storing a computer program, which is executable by at least one processor to cause the at least one processor to perform the steps of a video de-watermarking method as described above.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method of the embodiments of the present application.
It is to be understood that the above-described embodiments are merely illustrative of some, but not restrictive, of the broad invention, and that the appended drawings illustrate preferred embodiments of the invention and do not limit the scope of the invention. This application is capable of embodiments in many different forms and is provided for the purpose of enabling a thorough understanding of the disclosure of the application. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to one skilled in the art that the present application may be practiced without modification or with equivalents of some of the features described in the foregoing embodiments. All equivalent structures made by using the contents of the specification and the drawings of the present application are directly or indirectly applied to other related technical fields and are within the protection scope of the present application.
Claims (10)
1. A method for video watermarking, comprising:
acquiring a video to be processed, and performing frame division processing on the video to be processed to obtain an image frame set, wherein the acquired video to be processed comprises a smearing area, and the position of the smearing area in each image frame of the image frame set is fixed;
preliminarily repairing the smearing region of each image frame in the image frame set through a preset algorithm to obtain a preliminary image set;
taking the first frame image frame in the preliminary image set as a first reference image frame, and calculating the score difference value of each image frame in the preliminary image set and the first reference image frame to obtain a difference value score set;
comparing each difference score in the difference score set with a first preset threshold, and taking the image frame corresponding to the difference score exceeding the first preset threshold as an image frame to be repaired to obtain an image frame set to be repaired;
taking any image frame in the image frame set to be repaired as a second reference image frame, and performing deep repair on the smearing area of the second reference image frame to obtain a deep repair image frame, wherein the deep repair image frame comprises repair content of the smearing area;
calculating the similarity between each image frame to be repaired in the image frame set to be repaired and the second reference image frame;
judging whether the similarity exceeds a second preset threshold, if so, taking the image frame to be repaired, which corresponds to the similarity exceeding the second preset threshold, as a first repaired image, and filling the repaired content of the second reference image frame into the smearing area of the first repaired image;
and if the similarity is lower than the second preset threshold, taking the image frame to be repaired, corresponding to the similarity lower than the second preset threshold, as a second repaired image, and performing the deep repair on the smearing area of the second repaired image.
2. The method according to claim 1, wherein the acquiring the video to be processed and performing frame division processing on the video to be processed to obtain an image frame set comprises:
acquiring an initial video, and returning the video to be processed to a user side so that the user side can smear the watermark area of the video to be processed to obtain the video to be processed;
and performing frame division processing on the video to be processed to obtain the image frame set.
3. The method as claimed in claim 1, wherein said preliminarily repairing the smear region of each image frame in the image frame set by a predetermined algorithm to obtain a preliminary image set comprises:
identifying the smearing region in each image frame aiming at any image frame in the image frame set, and constructing a rectangular region based on the smearing region, wherein the rectangular region comprises the smearing region;
acquiring pixels in the rectangular frame except the smearing area as pixels to be filled;
filling the pixels to be filled into the smearing area to obtain a primary repairing image;
and when all the image frames in the image frame set are repaired, obtaining the preliminary image set.
4. The method according to claim 1, wherein the step of calculating a score difference value between each image frame in the preliminary image set and the first reference image frame by using a first image frame in the preliminary image set as a first reference image frame to obtain a difference score set comprises:
taking a first frame image frame in the preliminary image set as a first reference image frame;
evaluating and scoring the preliminary image set by adopting a preset evaluation scoring algorithm to obtain the score of each image frame;
and calculating the difference between the score of each image frame and the score of the first reference image frame to obtain the score difference value.
5. The method according to claim 1, wherein the step of using any image frame in the image frame set to be repaired as a second reference image frame and performing depth repair on the smear region of the second reference image frame to obtain a depth-repaired image frame comprises:
taking any image frame in the image frame set to be repaired as a second reference image frame, and performing negation processing on the smearing area of the second reference image frame to obtain a negated smearing image frame, wherein the negated smearing image frame comprises the negated smearing area;
multiplying the reversed smearing image frame and the second reference image frame to obtain a first target image, wherein the first target image comprises the reversed smearing area;
overlapping the first target image and the smearing area to obtain a second target image;
and repairing the second target image to obtain the depth repair image frame.
6. The method for video de-watermarking according to claim 5, wherein the repairing the second target image to obtain the depth-repaired image frame comprises:
performing down-sampling operation on the second target image to obtain a down-sampled image;
performing fast Fourier convolution processing on the down-sampling image to obtain a convolution image;
and performing up-sampling operation on the convolution image to obtain the depth restoration image frame.
7. The method according to claim 1, wherein said calculating the similarity between any image frame to be repaired in the image frame set to be repaired and a second image frame comprises:
blocking any image frame to be repaired and the second image frame to obtain a first block and a second block;
respectively calculating the first block and the second block by adopting a Gaussian weighting algorithm to obtain a first calculation result and a second calculation result, wherein the first calculation result comprises the mean value, the variance and the covariance of the first block, and the second calculation result comprises the mean value, the variance and the covariance of the second block;
calculating the structural similarity of the first block according to the first calculation result to obtain the similarity of the first block; calculating the structural similarity of the second block according to the second calculation result to obtain the similarity of the second block;
and calculating the average value of the structural similarity values of the first block and the second block to obtain the similarity between any image frame to be repaired and the second image frame.
8. An apparatus for video watermarking, comprising:
the device comprises a mask generation module, a frame segmentation module and a frame segmentation module, wherein the mask generation module is used for acquiring a video to be processed and performing frame segmentation on the video to be processed to obtain an image frame set, the acquired video to be processed comprises an smearing region, and the position of the smearing region in each image frame of the image frame set is fixed;
the preliminary repairing module is used for preliminarily repairing the smearing area of each image frame in the image frame set through a preset algorithm to obtain a preliminary image set;
the difference value calculation module is used for taking a first frame image frame in the preliminary image set as a first reference image frame, calculating a score difference value of each image frame in the preliminary image set and the first reference image frame to obtain a difference value score set;
the difference value comparison module is used for comparing each difference value score in the difference value score set with a first preset threshold value, and taking the image frame corresponding to the difference value score exceeding the first preset threshold value as an image frame to be repaired to obtain an image frame set to be repaired;
the sample repairing module is used for taking any image frame in the image frame set to be repaired as a second reference image frame and performing deep repairing on the smearing area of the second reference image frame to obtain a deep repairing image frame, wherein the deep repairing image frame comprises repairing contents of the smearing area;
the similarity calculation module is used for calculating the similarity between each image frame to be repaired in the image frame set to be repaired and the second reference image frame;
the filling module is used for judging whether the similarity exceeds a second preset threshold value, if so, taking the image frame to be repaired, which corresponds to the similarity exceeding the second preset threshold value, as a first repaired image, and filling the repaired content of the second reference image frame into the smearing area of the first repaired image;
and the depth repairing module is used for taking the image frame to be repaired, corresponding to the similarity lower than the second preset threshold value, as a second repaired image and performing the depth repairing on the smearing area of the second repaired image if the similarity is lower than the second preset threshold value.
9. A computer device comprising a memory having a computer program stored therein and a processor that, when executed, implements the video de-watermarking method of any of claims 1 to 7.
10. A computer-readable storage medium, having stored thereon a computer program which, when executed by a processor, implements a video de-watermarking method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211032315.5A CN115396743B (en) | 2022-08-26 | 2022-08-26 | Video watermark removing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211032315.5A CN115396743B (en) | 2022-08-26 | 2022-08-26 | Video watermark removing method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115396743A true CN115396743A (en) | 2022-11-25 |
| CN115396743B CN115396743B (en) | 2023-08-11 |
Family
ID=84123424
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211032315.5A Active CN115396743B (en) | 2022-08-26 | 2022-08-26 | Video watermark removing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115396743B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101472161A (en) * | 2007-12-27 | 2009-07-01 | 华为技术有限公司 | Method, device and system for embedding and removing watermark |
| EP2437498A1 (en) * | 2010-09-30 | 2012-04-04 | British Telecommunications Public Limited Company | Digital video fingerprinting |
| CN109714599A (en) * | 2017-10-26 | 2019-05-03 | 北京京东尚科信息技术有限公司 | The method and device of video watermark processing based on HEVC |
| WO2021259005A1 (en) * | 2020-06-23 | 2021-12-30 | 平安科技（深圳）有限公司 | Video-based micro-expression recognition method and apparatus, computer device, and storage medium |
| CN114302252A (en) * | 2021-12-31 | 2022-04-08 | 深圳万兴软件有限公司 | Method and device for removing watermark from video, computer equipment and storage medium |
-
2022
- 2022-08-26 CN CN202211032315.5A patent/CN115396743B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101472161A (en) * | 2007-12-27 | 2009-07-01 | 华为技术有限公司 | Method, device and system for embedding and removing watermark |
| EP2437498A1 (en) * | 2010-09-30 | 2012-04-04 | British Telecommunications Public Limited Company | Digital video fingerprinting |
| CN109714599A (en) * | 2017-10-26 | 2019-05-03 | 北京京东尚科信息技术有限公司 | The method and device of video watermark processing based on HEVC |
| WO2021259005A1 (en) * | 2020-06-23 | 2021-12-30 | 平安科技（深圳）有限公司 | Video-based micro-expression recognition method and apparatus, computer device, and storage medium |
| CN114302252A (en) * | 2021-12-31 | 2022-04-08 | 深圳万兴软件有限公司 | Method and device for removing watermark from video, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 张洪彬;伏长虹;陈锐霖;萧允治;苏卫民: "3D-HEVC深度图像快速帧内编码方法", 中国图象图形学报, vol. 21, no. 7 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115396743B (en) | 2023-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11170210B2 (en) | Gesture identification, control, and neural network training methods and apparatuses, and electronic devices | |
| Shi et al. | Normalised gamma transformation‐based contrast‐limited adaptive histogram equalisation with colour correction for sand–dust image enhancement | |
| CN112308095A (en) | Picture preprocessing and model training method and device, server and storage medium | |
| US20170109873A1 (en) | Image enhancement using self-examples and external examples | |
| Xiao et al. | Scene‐aware image dehazing based on sky‐segmented dark channel prior | |
| CN106650615A (en) | Image processing method and terminal | |
| CN104166968A (en) | Image dehazing method and device and mobile terminal | |
| CN112991197B (en) | Low-illumination video enhancement method and device based on detail preservation of dark channel | |
| CN113657528B (en) | Image feature point extraction method and device, computer terminal and storage medium | |
| JP2021531571A (en) | Certificate image extraction method and terminal equipment | |
| CN113592776A (en) | Image processing method and device, electronic device and storage medium | |
| Agrawal et al. | Dense haze removal by nonlinear transformation | |
| Ju et al. | IDBP: Image dehazing using blended priors including non-local, local, and global priors | |
| CN115660945A (en) | Coordinate conversion method and device, electronic equipment and storage medium | |
| Chen et al. | Improve transmission by designing filters for image dehazing | |
| CN113507570B (en) | Exposure compensation method and device and electronic equipment | |
| Fuh et al. | Mcpa: A fast single image haze removal method based on the minimum channel and patchless approach | |
| CN113259605A (en) | Video matting method, system and storage medium based on prediction foreground mask prediction | |
| US20120170861A1 (en) | Image processing apparatus, image processing method and image processing program | |
| Bartani et al. | An adaptive optic-physic based dust removal method using optimized air-light and transfer function | |
| CN108961293B (en) | Background subtraction method, device, equipment and storage medium | |
| CN110633705A (en) | Low-illumination imaging license plate recognition method and device | |
| CN115396743B (en) | Video watermark removing method, device, equipment and storage medium | |
| CN115410133A (en) | Video dense prediction method and device | |
| CN110766153A (en) | Neural network model training method and device and terminal equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |