WO2020156009A1 - 视频修复方法及装置、电子设备和存储介质 - Google Patents

视频修复方法及装置、电子设备和存储介质 Download PDF

Info

Publication number
WO2020156009A1
WO2020156009A1 PCT/CN2019/130428 CN2019130428W WO2020156009A1 WO 2020156009 A1 WO2020156009 A1 WO 2020156009A1 CN 2019130428 W CN2019130428 W CN 2019130428W WO 2020156009 A1 WO2020156009 A1 WO 2020156009A1
Authority
WO
WIPO (PCT)
Prior art keywords
optical flow
image
reverse
optimized
optimization
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/130428
Other languages
English (en)
French (fr)
Inventor
徐瑞
李晓潇
周博磊
吕健勤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Sensetime Technology Development Co Ltd
Original Assignee
Beijing Sensetime Technology Development Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Sensetime Technology Development Co Ltd filed Critical Beijing Sensetime Technology Development Co Ltd
Priority to KR1020207036725A priority Critical patent/KR102406354B1/ko
Priority to SG11202010145WA priority patent/SG11202010145WA/en
Priority to JP2021506038A priority patent/JP7125541B2/ja
Publication of WO2020156009A1 publication Critical patent/WO2020156009A1/zh
Priority to US17/072,527 priority patent/US11544820B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
    • H04N7/0135—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes
    • H04N7/0137—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes dependent on presence/absence of motion, e.g. of motion zones

Definitions

  • the present disclosure relates to the field of video repair, in particular to a video repair method and device, electronic equipment and storage medium.
  • Image restoration has always been an important issue in computer vision, and it has important applications in many fields.
  • image restoration can be used to achieve image watermarking and image restoration.
  • Video repair is an extension of the image repair problem on video. Compared with the simple image repair problem, it is much more difficult to implement.
  • the application of video restoration algorithms has broader prospects, such as the need to remove video watermarks and remove certain parts of the video.
  • the embodiments of the present disclosure provide a video repair solution.
  • a video repair method which includes: acquiring a first forward optical flow and a first reverse optical flow between adjacent images in a continuous multi-frame image; A forward optical flow and a first reverse optical flow perform optical flow optimization processing to obtain a second forward optical flow corresponding to the first forward optical flow, and a second forward optical flow corresponding to the first reverse optical flow. 2.
  • Reverse optical flow use the second forward optical flow to perform forward conduction optimization on the continuous multi-frame image, and use the second reverse optical flow to perform reverse on the continuous multi-frame image Conduction optimization, until the images in the optimized continuous multi-frame images meet the repair requirements; according to the optimized images obtained by the forward conduction optimization and the reverse conduction optimization, the repaired image of the continuous multi-frame images is obtained.
  • the obtaining the first forward optical flow and the first reverse optical flow between adjacent images in the continuous multi-frame images includes: obtaining the information of each image in the multi-frame image
  • the mask map represents the area to be repaired of the corresponding image;
  • the forward optical flow between adjacent images is determined according to the forward sequence of the time frames of the multi-frame image, and according to the multi-frame image The reverse order of the time frame to determine the reverse optical flow between adjacent images;
  • the forward optical flow between the i-th image and the i+1-th image and the mask of the i+1-th image Multiply the images to obtain the first forward optical flow between the i-th image and the i+1-th image, and the reverse optical flow between the i+1-th image and the i-th image and the i-th image Multiply the mask image to obtain the first reverse optical flow between the i+1th image and the ith image, where i is a positive integer greater than or equal to 1 and less than G, and G is the image in the multi-frame image Quantity.
  • the performing optical flow optimization processing on the first forward optical flow and the first reverse optical flow respectively includes: connecting the first forward optical flow to obtain a first connected optical flow , And connect the first reverse optical flow to obtain a second connected optical flow; respectively perform N optical flow optimization processing on the first connected optical flow and the second connected optical flow to obtain the corresponding first connected optical flow N is a positive integer greater than or equal to 1; according to the first optimized optical flow, the corresponding first forward optical flow is obtained according to the first optimized optical flow of the second connected optical flow and the second optimized optical flow corresponding to the second connected optical flow. A second forward optical flow, and a second reverse optical flow corresponding to each first reverse optical flow is obtained according to the second optimized optical flow.
  • the performing N times of optical flow optimization processing on the first connected optical flow and the second connected optical flow respectively includes: performing the first connected optical flow and the second connected optical flow
  • the first optical flow optimization processing obtains the first optimized sub-optical flow corresponding to the first connected optical flow and the first optimized sub-optical flow corresponding to the second connected optical flow;
  • the Nth optimized sub-optical flow of the first connected optical flow is obtained Determine as the first optimized optical flow, and determine the obtained Nth optimized sub-optical flow of the second connected optical flow as the second optimized optical flow, wherein each optical flow optimization process includes residual processing And upsampling processing.
  • the use of the second forward optical flow to perform forward conduction optimization on the continuous multi-frame image includes: obtaining a mask image of each image in the multi-frame image
  • the mask map represents the area to be repaired of the corresponding image; according to the forward sequence of the time frames of the multi-frame image, the first repair map of the image other than the first image is obtained according to the second forward optical flow ; Adjust the mask image of the corresponding image based on the obtained first restoration image, and return to perform the previous step until the adjusted mask image indicates that there is no longer an area to be restored, then determine the image in the multi-frame image Meet the repair requirements.
  • the obtaining a first restoration image of each image according to the second forward optical flow according to the forward sequence of the time frames of the multi-frame image includes: The forward sequence of the time frame of the image, the first image is offset according to the first second forward optical flow to obtain the first restoration image of the second image; the kth image and the kth The repaired image of each image is added and processed to obtain the first added image of the k-th image, and the k-th first added image is offset according to the k-th second forward optical flow to obtain The first restoration image of the k+1th image, where k is a positive integer greater than 1 and less than G, and G represents the number of images in multiple frames.
  • the using the second reverse optical flow to perform reverse conduction optimization on the continuous multi-frame image includes: obtaining a mask image of each image in the multi-frame image
  • the mask map represents the area to be repaired of the corresponding image; according to the reverse order of the time frames of the multi-frame image, a second repaired image of the image other than the first image is obtained according to the second reverse optical flow ; Adjust the mask map of the corresponding image based on the obtained second restoration map, and return to perform the previous step until the adjusted mask map indicates that there is no longer an area to be repaired, then determine the image in the multi-frame image Meet the repair requirements.
  • the obtaining a second restoration image of an image other than the first image according to the second reverse optical flow according to the reverse order of the time frames of the multi-frame image includes: In the reverse order of the time frames of the multi-frame image, the first image is offset according to the first and second reverse optical flow to obtain the second restoration image of the second image; the hth image and The repaired image of the hth image performs summation processing to obtain the second summation image of the hth image, and performs the biasing on the hth second summation image according to the hth second reverse optical flow Shift processing to obtain the second restoration image of the h+1th image, where h is a positive integer greater than 1 and less than or equal to G, and G represents the number of images in multiple frames.
  • the obtaining the repaired image of the continuous multi-frame image according to the optimized image obtained according to the forward conduction optimization and the reverse conduction optimization includes: following the forward sequence of the time frames of the multi-frame image , Obtain the first optimized image obtained by the forward conduction optimization; obtain the second optimized image obtained by the reverse conduction optimization according to the forward sequence of the time frames of the multi-frame image; according to the first optimized image of each image And the mean value of the second optimized image to obtain a repaired image of the image to obtain a repaired image of each image in the multiple frames of images.
  • the method includes: performing the optical flow optimization processing on the first forward optical flow and the first reverse optical flow respectively by an optical flow optimization module, to obtain the same The second forward optical flow corresponding to the forward optical flow, and the second reverse optical flow corresponding to the first reverse optical flow; the method further includes the step of training the optical flow optimization module, The method includes: obtaining training samples, the training samples including multiple sets of optical flow samples, and mask samples and real optical flow corresponding to each optical flow sample; performing optical flow on the optical flow samples by the optical flow optimization module Optimize processing to obtain the corresponding optimized optical flow; based on the comparison result of the optimized optical flow and the real optical flow, select the first pixel that meets the first requirement from the optimized optical flow, and adjust the first pixel point
  • the mask value of a pixel in the mask image is used to obtain a new mask sample; according to the preset loss function, the loss value is obtained based on the optimized optical flow, the original mask sample and the new mask sample; In the case that the loss value meets the training requirements, the trained optical flow optimization module
  • the expression of the preset loss function is:
  • L represents the loss function of the optical flow optimization processing
  • M is the original mask sample
  • M h represents the new mask sample
  • f represents the optimized optical flow obtained
  • Re represents the real optical flow
  • ⁇ is the weight.
  • a video repair device which includes:
  • An acquisition module which is used to acquire the first forward optical flow and the first reverse optical flow between adjacent images in continuous multi-frame images; an optical flow optimization module, which is used to separately compare the first forward optical flow The optical flow and the first reverse optical flow are optimized to obtain a second forward optical flow corresponding to the first forward optical flow, and a second reverse optical flow corresponding to the first reverse optical flow.
  • conduction optimization module which is used to use the second forward optical flow to perform forward conduction optimization on the continuous multi-frame image, and use the second reverse optical flow to perform the forward conduction optimization on the continuous multi-frame image Perform reverse conduction optimization until the images in the optimized continuous multi-frame images meet the repair requirements; the repair module is used to obtain the continuous multiple images based on the optimized images obtained by forward conduction optimization and reverse conduction optimization. Repaired image of frame image.
  • the acquisition module is further configured to acquire a mask image of each image in the multi-frame image, and determine the one between adjacent images according to the forward sequence of the time frame of the multi-frame image. And determine the reverse optical flow between adjacent images according to the reverse order of the time frames of the multi-frame images; and divide the i-th image and the i+1-th image
  • the forward optical flow of is multiplied by the mask image of the i+1th image to obtain the first forward optical flow between the ith image and the i+1th image, and the i+1th image and the
  • the reverse optical flow between the i images is multiplied by the mask image of the i image to obtain the first reverse optical flow between the i+1 image and the i image, wherein the mask image Indicates the area to be repaired of the corresponding image, i is a positive integer greater than or equal to 1 and less than G, and G is the number of images in a multi-frame image.
  • the optical flow optimization module is further configured to connect the first forward optical flow to obtain a first connected optical flow, and connect the first reverse optical flow to obtain a second connected optical flow, And respectively perform N optical flow optimization processing on the first connected optical flow and the second connected optical flow to obtain the first optimized optical flow corresponding to the first connected optical flow, and the second optical flow corresponding to the second connected optical flow.
  • N is a positive integer greater than or equal to 1
  • obtain the second forward optical flow corresponding to each first forward optical flow according to the first optimized optical flow, and obtain the second forward optical flow corresponding to each first forward optical flow according to the second optimized optical flow Obtain the second reverse optical flow corresponding to each first reverse optical flow.
  • the optical flow optimization module is further configured to perform a first optical flow optimization process on the first connected optical flow and the second connected optical flow to obtain the corresponding optical flow of the first connected optical flow.
  • the first optimized sub-optical flow, and the first optimized sub-optical flow corresponding to the second connected optical flow; and the i-th optimized sub-optical flow of the first connected optical flow and the second connected optical flow are executed respectively +1 optical flow optimization processing to obtain the i+1th optimized sub-optical flow corresponding to the first connected optical flow and the i+1th optimized sub-optical flow corresponding to the second connected optical flow, where i is greater than 1 and A positive integer less than N; wherein, through the Nth optimization process, the obtained Nth optimized sub-optical flow of the first connected optical flow is determined as the first optimized optical flow, and the obtained second The Nth optimized sub-optical flow connected to the optical flow is determined to be the second optimized optical flow, wherein each optical flow optimization processing includes residual processing and up-sampling processing.
  • the conduction optimization module includes a first conduction optimization unit, which is used to obtain a mask map of each image in the multi-frame images, and the mask map represents a corresponding image to be repaired Area; according to the forward sequence of the time frames of the multi-frame image, according to the second forward optical flow to obtain a first restoration image of the image other than the first image; adjust the corresponding based on the obtained first restoration image The mask map of the image, and return to the previous step until the adjusted mask map indicates that there is no longer a region to be repaired, then it is determined that the images in the multi-frame image meet the repair requirements.
  • a first conduction optimization unit which is used to obtain a mask map of each image in the multi-frame images, and the mask map represents a corresponding image to be repaired Area
  • the first conduction optimization unit is further configured to perform offset on the first image according to the first and second forward optical flow according to the forward sequence of the time frames of the multi-frame image. Processing to obtain the first restored image of the second image; and perform addition processing on the kth image and the restored image of the kth image to obtain the first summed image of the kth image, and according to the kth image
  • the second forward optical flow performs offset processing on the k-th first addition image to obtain the first restoration image of the k+1-th image, where k is a positive integer greater than 1 and less than G, and G represents The number of images in a multi-frame image.
  • the conduction optimization module further includes a second conduction optimization unit, which is used to obtain a mask map of each image in the multi-frame images, and the mask map represents the pending image of the corresponding image.
  • Repair area according to the reverse order of the time frame of the multi-frame image, obtain a second repair image of images other than the first image according to the second reverse optical flow; adjust based on the obtained second repair image Corresponding to the mask map of the image, and return to perform the previous step, until the adjusted mask map indicates that there is no longer an area to be repaired, then it is determined that the images in the multi-frame image meet the repair requirements.
  • the second conduction optimization unit is further configured to perform offset on the first image according to the first second reverse optical flow according to the reverse order of the time frames of the multi-frame image Processing to obtain the second restored image of the second image; and perform sum processing on the hth image and the restored image of the hth image to obtain the second summed image of the hth image, and according to the hth image
  • the second reverse optical flow performs offset processing on the h-th second summation image to obtain a second restoration image of the h+1-th image, where h is a positive integer greater than 1 and less than or equal to G, G represents the number of images in a multi-frame image.
  • the repair module is further configured to obtain the first optimized image obtained by the forward conduction optimization according to the forward sequence of the time frames of the multi-frame image, and according to the time frame of the multi-frame image In the forward sequence, the second optimized image obtained by the reverse conduction optimization is obtained; and according to the average value of the first optimized image and the second optimized image of each image, a repaired image of the image is obtained to obtain the multi-frame image The repaired image of each image in.
  • it further includes a training module for training the optical flow optimization module, and for inputting training samples to the optical flow optimization module, the training samples including multiple sets of optical flow samples, and The mask sample and the real optical flow corresponding to each optical flow sample; use the optical flow optimization module to perform optical flow optimization processing on the optical flow sample to obtain the corresponding optimized optical flow; based on the optimized optical flow and the The comparison result of the real optical flow, the first pixel that meets the first requirement is selected from the optimized optical flow, and the mask value of the first pixel in the mask image is adjusted to obtain a new mask sample According to the preset loss function, the loss value is obtained based on the optimized optical flow, the original mask sample and the new mask sample; in the case that the loss value meets the training requirements, the trained optical flow optimization module is obtained, Otherwise, adjust the network parameters of the optical flow optimization module and re-execute the optical flow optimization processing of the optical flow samples until the obtained loss value meets the training requirements.
  • a training module for training the optical flow optimization module, and for inputting training samples to the optical flow optimization
  • the expression of the preset loss function is
  • L represents the loss function of the optical flow optimization processing
  • M is the original mask sample
  • M h represents the new mask sample
  • f represents the optimized optical flow obtained
  • Re represents the real optical flow
  • ⁇ is the weight.
  • an electronic device including:
  • a memory for storing processor executable instructions
  • the processor is configured to execute the method described in any one of the first aspect.
  • a computer-readable storage medium on which computer program instructions are stored.
  • the computer program instructions are characterized in that, when the computer program instructions are executed by a processor, the Methods.
  • a computer program includes computer-readable code, and when the computer-readable code is executed in an electronic device, a processor in the electronic device executes the first The method of any one of the aspects.
  • the embodiments of the present disclosure can perform repair processing on video images of multiple consecutive time frames at the same time.
  • the forward optical flow and the reverse optical flow between adjacent frame images can be optimized first to obtain the repaired light with higher accuracy. Then through the optimized forward optical flow and reverse optical flow to guide the video repair to complete the modification of the video image, this process can improve the accuracy of the video repair, while improving the continuity and consistency between frames .
  • the embodiments of the present disclosure have a faster repair speed, which satisfies the real-time nature of repair.
  • Fig. 1 shows a flowchart of a video repair method according to an embodiment of the present disclosure
  • Fig. 2 shows a flowchart of step S100 in a video repair method according to an embodiment of the present disclosure
  • Fig. 3 shows a flowchart of step S200 in a video repair method according to an embodiment of the present disclosure
  • Figure 4 shows a schematic structural diagram of an optical flow optimization network according to an embodiment of the present disclosure
  • Fig. 5 shows a flowchart of step S300 of a video repair method according to an embodiment of the present disclosure
  • Fig. 6 shows a flowchart of step S302 of a video repair method according to an embodiment of the present disclosure
  • FIG. 7 shows a flowchart of step S300 in a video repair method according to an embodiment of the present disclosure
  • FIG. 8 shows a flowchart of step S3002 of a video repair method according to an embodiment of the present disclosure
  • Fig. 9 shows a flowchart of step S400 in a video repair method according to an embodiment of the present disclosure
  • Fig. 10 shows a flowchart of a training method of an optical flow optimization module according to an embodiment of the present disclosure
  • Fig. 11 shows a block diagram of a video repairing device according to an embodiment of the present disclosure
  • FIG. 12 shows a block diagram of an electronic device 800 according to an embodiment of the present disclosure
  • FIG. 13 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure.
  • FIG. 1 shows a flowchart of a video repair method according to an embodiment of the present disclosure.
  • the video repair method of the embodiment of the present disclosure can be used to repair video images, and the embodiment of the present disclosure can ensure the consistency of continuous video frame repair, and The repair accuracy is high and the efficiency is fast.
  • the embodiments of the present disclosure can be applied to image processing equipment or video capture equipment, where the image processing equipment can be any terminal equipment or server equipment, such as mobile phones, cameras, laptops, PADs, cloud servers, etc., as long as they have image processing functions or
  • the ability to collect video information means that the video repair method of the embodiment of the present disclosure can be applied.
  • the embodiments of the present disclosure are described in detail below.
  • the video repair method of the embodiment of the present disclosure may include:
  • S100 Acquire a first forward optical flow and a first reverse optical flow between adjacent images in consecutive multiple frames of images
  • S200 Perform optical flow optimization processing on the first forward optical flow and the first reverse optical flow respectively to obtain a second forward optical flow corresponding to the first forward optical flow, and The second reverse optical flow corresponding to the reverse optical flow;
  • S300 Use the second forward optical flow to perform forward conduction optimization on the continuous multi-frame image, and use the second reverse optical flow to perform reverse conduction optimization on the continuous multi-frame image, until The images in the optimized continuous multi-frame images meet the restoration requirements;
  • S400 Obtain a repaired image of the continuous multi-frame image according to the optimized image obtained by the forward conduction optimization and the reverse conduction optimization.
  • the video repair process can be performed through the constructed neural network.
  • the video image can be repaired through the deep learning neural network, but the present disclosure does not specifically limit this.
  • the embodiments of the present disclosure can also be combined with the present disclosure.
  • the algorithm that matches the function defined in the disclosed embodiment performs the video repair process.
  • the video image to be repaired can be acquired first, and continuous video frame images (ie, the above-mentioned continuous multi-frame images) can be acquired.
  • continuous video frame images ie, the above-mentioned continuous multi-frame images
  • the embodiment of the present disclosure can acquire a preset number of images of continuous time frames, such as 5 frames.
  • the preset number can also be other values, which can be set by those skilled in the art according to different needs and situations.
  • the multi-frame image After acquiring the continuous multi-frame image, the multi-frame image can be repaired.
  • the optical flow between adjacent images can be obtained, where the adjacent images refer to two adjacent frames of images. Since the time frame is time-sequential, the embodiments of the present disclosure can start from the forward and reverse of time. The image is processed in two directions to achieve video restoration.
  • optical flow can be used to represent changes between moving images, and refers to the speed of mode movement in time-varying images.
  • the target object is moving
  • the brightness mode of its corresponding point on the image is also moving, so the optical flow can be used to represent the changes between images. Since it contains the information of the target movement, it can be used by the observer to determine The movement of the target.
  • the embodiment of the present disclosure may obtain the optical flow between every two adjacent images in step S100, for example, obtain the first forward optical flow between the two adjacent images according to the forward sequence of the time frame, And according to the reverse order of the time frame, the first forward optical flow between two adjacent images is obtained.
  • the multi-frame images in the embodiment of the present disclosure are ⁇ P 1 , P 2 ,...P G ⁇ , which may include G frame images, for example, G may be 5 or other values.
  • the first forward optical flow ⁇ f 1 ⁇ 2 ,f 2 ⁇ 3 ,...,f G-1 ⁇ G ⁇ can be obtained, where f i ⁇ i+1 represents the image P i to The first forward optical flow of the image Pi +1 , i is any integer greater than or equal to 1 and less than G.
  • the first reverse optical flow ⁇ f G ⁇ G-1 ,f G-1 ⁇ G-2 ,...,f 2 ⁇ 1 ⁇ can be obtained according to the reverse order of the time frame, where f j+1 ⁇ j represents the first reverse optical flow from image P j+1 to image P j , and j is an integer greater than or equal to 1 and less than G.
  • the embodiment of the present disclosure may use the optical flow estimation model to obtain the first forward optical flow and the first reverse optical flow between the images.
  • the optical flow estimation model may be FlowNet 2.0.
  • the optical flow between the images can also be obtained in other ways, which is not specifically limited in the present disclosure.
  • the embodiments of the present disclosure may also process the optical flow between images in combination with the mask map to obtain the first forward optical flow and the first reverse optical flow.
  • Fig. 2 shows a flowchart of step S100 in a video repair method according to an embodiment of the present disclosure.
  • the acquiring the first forward optical flow and the first reverse optical flow between adjacent images in the continuous multiple frames of images may include:
  • S101 Obtain a mask map of each image in the multi-frame images, where the mask map represents a region to be repaired of the corresponding image;
  • each of the obtained multi-frame images may have a corresponding mask image, and the mask image may be determined artificially.
  • a corresponding mask map can be established according to the area in each image that needs to be repaired, where the scale of the mask map is the same as the scale of the corresponding image, and the pixels in the mask map are the same as the pixels in the corresponding image.
  • the pixel points in the area to be repaired can be defined as 1 in the mask image, and the pixels in the remaining areas that do not need to be repaired are defined as 0 in the mask image, or also The pixel points in the area to be repaired can be defined as 0 in the mask image, and the pixels in the remaining areas that do not need to be repaired are defined as 1 in the mask image.
  • Those skilled in the art can use different The mask needs to be set, and this disclosure does not specifically limit this.
  • S102 Determine the forward optical flow between adjacent images according to the forward sequence of the time frames of the multi-frame images, and determine the forward optical flow between adjacent images according to the reverse sequence of the time frames of the multi-frame images The reverse optical flow;
  • optical flow estimation model can be used to obtain the forward optical flow and the reverse optical flow between adjacent images in multiple frames. Go into details.
  • step S101 and step S102 is not specifically limited in the embodiment of the present disclosure.
  • Step S101 can be performed before step S102, can also be performed after step S102, or step S101 and step S102 can be performed simultaneously .
  • the mask map of each image and the forward optical flow and reverse optical flow between any two adjacent images are determined in step S103, it can be used as an embodiment of the present disclosure.
  • S103 Multiply the forward optical flow between the i-th image and the i+1-th image by the mask image of the i+1-th image to obtain the first image between the i-th image and the i+1-th image.
  • a forward optical flow, and the reverse optical flow between the i+1 image and the i image is multiplied by the mask image of the i image to obtain the i+1 image and the i image between the first reverse optical flow, where i is a positive integer greater than or equal to 1 and less than G, and G is the number of images in multiple frames.
  • the embodiment of the present disclosure can multiply the forward optical flow by the corresponding mask map, that is, determine the optical flow area in the forward optical flow that needs to be optimized, and obtain the corresponding first forward optical flow.
  • the reverse optical flow can also be multiplied by the corresponding mask map to determine the optical flow area that needs to be optimized in the reverse optical flow, and the corresponding first reverse optical flow can be obtained.
  • the multi-frame images can be ⁇ P 1 ,P 2 ,...P G ⁇
  • the obtained forward optical flow between adjacent images can be ⁇ f 1 ⁇ 2 ,f 2 ⁇ 3 ,...,f G-1 ⁇ G ⁇
  • the reverse optical flow between images can be ⁇ f G ⁇ G-1 ,f G-1 ⁇ G-2 ,...,f 2 ⁇ 1 ⁇ .
  • the mask image of each image can be expressed as ⁇ M 1 , M 2 ,..., M G ⁇ , where M k represents the mask image of the k-th image P k , and k is greater than or equal to 1 and An integer less than or equal to G.
  • the forward optical flow can be compared with the mask map M of the second image P 2 2 Multiply, such as multiplying the corresponding elements, to obtain the first forward optical flow between the first image P 1 and the second image P 2 .
  • the forward optical flow between the second image P 2 and the third image P 3 can be obtained, and the forward optical flow can be multiplied by the mask image M 3 of the third image P 3 , as The corresponding elements are multiplied to obtain the first forward optical flow between the second image P 2 and the third image P 3 .
  • the forward optical flow between the i-th image and the i+1-th image can be multiplied by the mask image of the i+1-th image, such as multiplying the corresponding elements to obtain the i-th image and the i-th image.
  • the first positive optical flow between i+1 images Through the above method, the first forward optical flow ⁇ f 1 ⁇ 2 , f 2 ⁇ 3 ,..., f G-1 ⁇ G ⁇ between adjacent images can be obtained.
  • the embodiment of the present disclosure may be combined with the penultimate image.
  • the mask map M G-1 of P G-1 is multiplied, such as corresponding elements are multiplied, to obtain the first reverse optical flow between the last image P G and the penultimate image P G-1 .
  • the reverse optical flow between the penultimate image P G-1 and the penultimate image P G-2 can be obtained, and the reverse optical flow can be compared with that of the penultimate image P G-2 .
  • Multiplying the mask image MG-2 obtains the first reverse optical flow between the penultimate image P G-1 and the penultimate image P G-2 .
  • the reverse optical flow between the i+1th image and the i-th image can be multiplied by the mask image of the i-th image, such as multiplying the corresponding elements to obtain the i+1th image and the i-th image.
  • the first reverse optical flow ⁇ f G ⁇ G-1 , f G-1 ⁇ G-2 ,..., f 2 ⁇ 1 ⁇ between adjacent images can be obtained through the above method.
  • the embodiment of the present disclosure obtains the corresponding first forward optical flow and first reverse optical flow by multiplying the obtained forward optical flow and reverse optical flow with the mask map of the corresponding image.
  • the forward optical flow and the first reverse optical flow may include areas that need to be optimized for optical flow, which can improve the accuracy of optical flow optimization.
  • the optical flow may be optimized, that is, step S200 may be executed.
  • the first forward optical flow and the first reverse optical flow may be optimized separately to obtain the corresponding second forward optical flow and the second reverse optical flow.
  • the optical flow optimization process of the embodiments of the present disclosure may be implemented by an optical flow optimization network, which may include at least one optical flow optimization network module, each optical flow optimization network module is the same, and the number of optical flow optimization network modules can be The number of optical flow optimization is determined.
  • the embodiment of the present disclosure does not specifically limit the number of optical flow optimization processing. Those skilled in the art can select and set according to different needs. The optimization process of the first forward optical flow and the first reverse optical flow will be described in detail below.
  • FIG. 3 shows a flowchart of step S200 in a video repair method according to an embodiment of the present disclosure, wherein the performing optical flow optimization processing on the first forward optical flow and the first reverse optical flow respectively includes :
  • the first forward optical flows between adjacent images are connected separately, such as cascaded separately.
  • the first forward optical flow is connected in the depth direction to form a first connected optical flow composed of multiple layers of first forward optical flow.
  • the first reverse optical flow between adjacent images can also be connected separately, for example, the first reverse optical flow is connected in the depth direction to form a second connected light composed of multiple layers of first direction optical flow. flow.
  • S202 Perform N optical flow optimization processing on the first connected optical flow and the second connected optical flow respectively, to obtain the first optimized optical flow corresponding to the first connected optical flow, and the second connected optical flow corresponding to the second connected optical flow.
  • N is a positive integer greater than or equal to 1;
  • the optical flow optimization processing can be performed on the first connection optical flow and the second connection optical flow respectively.
  • the embodiment of the present disclosure can be performed at least once Optical flow optimization process.
  • each optical flow optimization process in the embodiments of the present disclosure may be performed by an optical flow optimization module, which may be composed of a neural network, or a corresponding algorithm may also be used to perform optimization operations.
  • N optical flow optimization network modules connected in sequence can be included, the input of the latter optical flow optimization network module is the output of the previous optical flow optimization network module, and the last optical flow optimization network module The output of the optimized network module is the optimized result of the first connection optical flow and the second connection optical flow.
  • the optical flow optimization network module can be used to perform optimization processing on the first connected optical flow to obtain the first optimized sub-optical flow corresponding to the first connected optical flow, and the optical flow optimization network module pair
  • the second connected optical flow performs optimization processing to obtain the second optimized sub-optical flow corresponding to the second connected optical flow.
  • the optical flow optimization processing can include residual processing and up-sampling processing. That is, the optical flow optimization network module may further include a residual unit and an up-sampling unit.
  • the scale of the first connected optical flow after residual processing by the residual unit becomes smaller, such as the input connected optical flow.
  • a quarter of the scale this disclosure does not specifically limit this, and can be set according to requirements.
  • the up-sampling process can be performed on the first connected optical flow or the second connected optical flow after the residual processing, and the scale of the output first optimized sub-optical flow can be adjusted to the first through the up-sampling process.
  • the scale of the connected optical flow, and the scale of the output second optimized sub-optical flow is adjusted to the scale of the second connected optical flow. And through the optical flow optimization process, the characteristics of multiple optical flows can be combined, and the optical flow accuracy can be improved at the same time.
  • the optical flow optimization module may also include multiple optical flow optimization network modules, such as N optical flow optimization network modules.
  • the first optical flow optimization network module can receive the first connection optical flow and the second connection optical flow, and perform the first optical flow optimization processing on the first connection optical flow and the second connection optical flow respectively.
  • the optical flow optimization processing includes residual processing and up-sampling processing, in which the specific process is the same as the above-mentioned embodiment, and will not be repeated here. Through the first optical flow optimization processing, the first optimized sub-optical flow of the first connected optical flow and the first optimized sub-optical flow of the second connected optical flow can be obtained.
  • the second optical flow optimization network module receives the second optical flow optimization processing from the second optical flow optimization network module, and performs the second optical flow optimization processing, wherein the second optical flow optimization network module can also be used It includes a residual unit that performs residual processing and an up-sampling unit that performs up-sampling processing.
  • the optical flow optimization processing of the first connected sub-optical flow and the second connected sub-optical flow can be performed through the second optical flow optimization network module.
  • the specific process is the same as the above embodiment, and will not be repeated here.
  • the second optimized sub-optical flow of the first connected optical flow and the second optimized sub-optical flow of the second connected optical flow can be obtained.
  • each optical flow optimization network module can be used to perform an optical flow optimization process, that is, the i+1th optical flow optimization network module can be used to analyze the first connection optical flow and the second connection optical flow.
  • the i-optimized sub-optical flow performs the i+1th optical flow optimization process to obtain the i+1-th optimized sub-optical flow corresponding to the first connected optical flow and the i+1-th optimized sub-optical flow corresponding to the second connected optical flow Stream, where i is a positive integer greater than 1 and less than N.
  • the Nth optimized sub-optical flow of the first connected optical flow and the Nth optimized sub-optical flow of the second connected optical flow can be obtained, and the obtained The Nth optimized sub-optical flow of the first connected optical flow is determined as the first optimized optical flow, and the obtained Nth optimized sub-optical flow of the second connected optical flow is determined as the second optimized optical flow light flow.
  • the optical flow optimization processing process performed by each optical flow optimization network module may be residual processing and up-sampling processing. That is, each optical flow optimization network module can be the same optical flow optimization module.
  • FIG. 4 shows a schematic structural diagram of an optical flow optimization network according to an embodiment of the present disclosure, which may include three optical flow optimization network modules A, B, and C.
  • the three optical flow optimization network modules can be composed of a residual unit and an up-sampling unit respectively.
  • the first optical flow optimization network module A can perform the first optical flow optimization processing of the first connected optical flow f 0 and the second connected optical flow f 1 to obtain the first connector of the first connected optical flow optical flow f 1, and a first linker connected to a second optical flow optical flow f 1 '.
  • the first connected optical flow f 1 and f 1' are respectively input to the second optical flow optimization network module B, and the second optical flow optimization processing is performed to obtain the second connected optical flow f of the first connected optical flow.
  • the second sub-connector 2 and the second optical connector light stream flow f 2 ' are respectively input to the third optical flow optimization network module C, and the third optical flow optimization processing is performed respectively to obtain the third optimization corresponding to the first connected optical flow.
  • the third optimized sub-optical flow of the first connected optical flow obtained in the last optical flow optimization processing can be determined as the first optimized optical flow
  • the third optimized optical flow of the second connected optical flow obtained in the last optical flow optimization processing can be determined.
  • the optimized sub-optical flow is determined as the second optimized optical flow.
  • S203 Obtain a second forward optical flow corresponding to each first forward optical flow according to the first optimized optical flow, and obtain a second forward optical flow corresponding to each first reverse optical flow according to the second optimized optical flow Reverse optical flow.
  • the scale of the first optimized optical flow is the same as the scale of the first connected optical flow
  • the first optimized optical flow can be split into G second positive optical flows according to the depth direction
  • the G second forward optical flows respectively correspond to the optimization results of the first forward optical flow.
  • the scale of the second optimized optical flow is the same as the scale of the second connected optical flow.
  • the second optimized optical flow can be split into G second reflections according to the depth direction.
  • the G second reverse optical flows respectively correspond to the optimization results of the first reverse optical flow.
  • the second forward optical flow optimized by the first forward optical flow between images and the second reverse optical flow optimized by the first reverse optical flow between images can be obtained.
  • the accuracy of video repair can be achieved by optimizing the optical flow.
  • the optimized second forward optical flow and the second reverse optical flow can be used to perform forward guidance and reverse guidance of multiple frames of images, respectively, to obtain the final repaired image.
  • the following describes the process of forward guidance and reverse guidance respectively.
  • FIG. 5 shows a flowchart of step S300 of a video repair method according to an embodiment of the present disclosure, wherein the use of the second forward optical flow to perform forward conduction optimization on the continuous multi-frame image includes :
  • S301 Obtain a mask map of each image in the multi-frame images, where the mask map represents a region to be repaired of the corresponding image;
  • each image in the embodiments of the present disclosure may correspond to a mask map, which is used to indicate the area to be repaired in the corresponding image.
  • the mask corresponding to the pixel in the area may be 1, and the rest The mask of the area may be 0, or the mask of the pixels in the area to be repaired may be 0, and the mask of the pixels in the remaining areas may be 1. This disclosure does not specifically limit this.
  • S302 Obtain a first restoration image of an image other than the first image according to the second forward optical flow according to the forward sequence of the time frames of the multi-frame image;
  • the obtained second forward optical flow represents the optical flow between two adjacent images in the forward sequence of the time frame, that is, the optical flow from the previous image to the next image. Therefore, the image can be guided according to the obtained second forward optical flow, that is, it is determined that each image is a new image obtained under the condition that the optical flow of the corresponding second forward optical flow changes, and the new image is The first restoration of the next image.
  • Fig. 6 shows a flowchart of step S302 of a video repair method according to an embodiment of the present disclosure.
  • the forward sequence of the time frames of the multi-frame image according to the second positive
  • S3021 Perform offset processing on the first image according to the first second forward optical flow according to the forward sequence of the time frames of the multi-frame image to obtain a first restoration image of the second image;
  • S3022 Perform sum processing on the kth image and the restored image of the kth image to obtain the first summed image of the kth image, and perform the summation processing on the kth image according to the kth second forward optical flow. Offset processing is performed on the first summed images to obtain the first restored image of the k+1th image, where k is a positive integer greater than 1 and less than G, and G represents the number of images in the multi-frame image.
  • the second forward optical flow can be expressed as ⁇ f′ 1 ⁇ 2 ,f′ 2 ⁇ 3 ,...,f′ G-1 ⁇ G ⁇ , where f′ i ⁇ i-1 denotes a second forward optical flow between the first image i-1 P i-1 and i-th image P i, i is an integer greater than 1 and less than or equal to G.
  • the second reverse optical flow can be expressed as ⁇ f′ G ⁇ G-1 ,f′ G-2 ⁇ G-3 ,...,f 2 ⁇ 1 ⁇ , where f′ j+1 ⁇ j means In the second reverse optical flow between the j+1th image P j+1 and the jth image P j , j is an integer greater than or equal to 1 and less than G.
  • the first image P 1 can be offset to obtain a new image P′ 2
  • the P′ 2 can be This is the first restoration image of the second image P 2
  • the second image P 2 and the first restored image P′ 2 can be added together, such as the corresponding addition of elements, to obtain the first added image of the second image P 2 .
  • the optimized second forward optical flow can be used to realize the restoration of the area to be repaired of the second image, for example, to supplement the corresponding pixel value, or adjust the pixel value.
  • the optical flow change of the second second forward optical flow f′ 2 ⁇ 3 can be used to offset the first summation image of the second image P 1 to obtain a new image P′ 3 .
  • P′ 3 may be the first restoration image of the third image P 3 .
  • the third image P 3 and the first repaired image P′ 3 can be added together, such as adding corresponding elements to obtain the first added image of the third image P 3 .
  • the optimized second forward optical flow such as supplementing the corresponding pixel value, or adjusting the pixel value.
  • the optical flow change of the kth second forward optical flow f′ k ⁇ k+1 can be used to offset the first summation image of the kth image P k to obtain a new image P ' K
  • the P'k may be the first restored image of the k-th image P k .
  • the k-th image P k and the first restored image P′ k can be added together, such as the corresponding addition of elements, to obtain the first sum image of the k+1-th image P k+1 .
  • the optimized second forward optical flow such as supplementing the corresponding pixel value, or adjusting the pixel value.
  • a first restoration map of each image can be obtained, and the pixel values of a part of the region to be repaired may be supplemented (that is, restored) in the first restoration map.
  • the first restoration image of the image is obtained, if a certain pixel in the area to be restored in the first restoration image of the k-th image is restored, that is, the pixel value is restored, the The mask value of the corresponding pixel in the mask image of the k images is returned. If the original mask value is 1, it is adjusted to 0, and vice versa.
  • step S302 can be repeated for the obtained first restoration image until there is no more area to be restored in the mask image corresponding to each image, and the image forward guidance process is completed. , You can get the first optimized image of each image.
  • Fig. 7 shows a flowchart of step S300 in a video repair method according to an embodiment of the present disclosure, wherein the use of the second reverse optical flow to perform reverse conduction optimization on the continuous multi-frame image may include :
  • S3001 Obtain a mask map of each image in the multi-frame images, where the mask map represents a region to be repaired of the corresponding image;
  • step S301 This process is the same as step S301, and will not be repeated here.
  • S3002 Obtain a second restored image of an image other than the first image according to the second reverse optical flow according to the reverse order of the time frames of the multi-frame image;
  • the obtained second reverse optical flow represents the optical flow between two adjacent images in the reverse order of the time frame, that is, the optical flow from the next image to the previous image. Therefore, the image can be reverse-directed according to the obtained second reverse optical flow, that is, it is determined that each image is a new image obtained under the condition that the optical flow of the corresponding second reverse optical flow changes. It is the first restoration image of the next image.
  • step S3002 of a video repair method according to an embodiment of the present disclosure.
  • the reverse order of the time frames of the multi-frame image according to the second reverse To obtain the second restoration image of each image by Xiang Guangliu, it can include:
  • S30021 Perform offset processing on the first image according to the first second reverse optical flow according to the reverse order of the time frames of the multi-frame image to obtain a second restored image of the second image;
  • S30022 Perform sum processing on the hth image and the restored image of the hth image to obtain a second summed image of the hth image, and perform the summation processing on the hth image according to the hth second reverse optical flow. Offset processing is performed on the second summed images to obtain the second restored image of the h+1th image, where h is a positive integer greater than 1 and less than or equal to G, and G represents the number of images in the multi-frame image.
  • the image can be expressed as ⁇ P 1 , P 2 ,..., P k ⁇
  • the image according to the reverse sequence of the time frame can be It is expressed as ⁇ P G ,P G-1 ,...,P 1 ⁇ .
  • the first image in the reverse order may be P G
  • the corresponding second image may be P G-1
  • the Gth image is P 1 .
  • the second reverse optical flow can be expressed as ⁇ f′ 1 ⁇ 2 ,f′ 2 ⁇ 3 ,...,f′ G-1 ⁇ G ⁇ according to the reverse order of the time frame, where f′ i ⁇ i-1 denotes a second forward optical flow between the first image i-1 P i-1 and i-th image P i, i is an integer greater than 1 and less than or equal to G.
  • the second reverse optical flow can be expressed as ⁇ f′ G ⁇ G-1 , f′ G-1 ⁇ G-2 ,..., f′ 2 ⁇ 1 ⁇ , where f′ j+1 ⁇ j represents a second counter j + 1 between the image P j + 1 and j-th optical flow image P j, j is an integer greater than or equal to 1 and less than G.
  • step S30021 according to the optical flow change of the first second reverse optical flow f′G ⁇ G-1 , the first image P G in the reverse order may be offset to obtain a new image P′′ G-1 , the P" G-1 may be the second restored image of the second image P G-1 in the reverse order. Then, the second image P G-1 in the reverse order and its second restored image P′′ G-1 can be added together, such as adding corresponding elements to obtain the second addition of the second image P G-1 Sum image.
  • the optimized second reverse optical flow can be used to achieve the repair of the area to be repaired in the penultimate image, such as supplementing the corresponding pixel value or adjusting the pixel value.
  • the optical flow change of the second second reverse optical flow f′ G-1 ⁇ G-2 can be used to offset the second summation image of the second image P G-1 in the reverse order.
  • a new image P" G-2 is obtained , and this P" G-2 may be the second restored image of the third image P G-2 in reverse order.
  • the third image P G-2 in the reverse order and its first restored image P′′ G-2 can be added together, such as the corresponding addition of the elements, to obtain the third image P G- in the reverse order.
  • the second counter may be implemented using the optimized optical flow area to be achieved repair the last third of the image, such as supplemental respective pixel values, or adjustment of the pixel values Wait.
  • the optical flow change of the hth second reverse optical flow f′ h+1 ⁇ h can be used to offset the second summation image of the hth image P G-h+1 in the reverse order.
  • a new image P" Gh is obtained , and the P" Gh can be the second restored image of the h+ 1th image P Gh in the reverse order.
  • the h+ 1th image P Gh in the reverse order and its second restored image P′′ Gh can be added together, such as the corresponding addition of the elements, to obtain the first image P Gh in the reverse order h+1 Sum image.
  • the optimized second reverse optical flow can be used to repair the region to be repaired of the last h+1 image, such as supplementing the corresponding pixel value, or adjusting the pixel value.
  • a second restoration image of each image can be obtained, and the pixel values of a part of the area to be restored may be supplemented (that is, restored) in the second restoration image.
  • S3003 Adjust the mask image of the corresponding image based on the obtained second restoration image, and return to perform the previous step until the adjusted mask image indicates that there is no longer a region to be repaired, then determine that the image in the multi-frame image The image meets the restoration requirements.
  • the second restoration image can be The mask value of the corresponding pixel in the mask image of the k images is returned. If the original mask value is 1, it is adjusted to 0, and vice versa.
  • step S3002 can be repeated for the obtained second restoration map until there is no more area to be repaired in the mask map corresponding to each image, and the reverse guidance process of the image is completed. , That is, the second optimized image of each image can be obtained.
  • step S400 may be executed.
  • Fig. 9 shows a flowchart of step S400 in a video repair method according to an embodiment of the present disclosure.
  • the obtaining the repaired image of the continuous multi-frame image according to the optimized image obtained by the forward conduction optimization and the reverse conduction optimization may include:
  • S402 Obtain a second optimized image obtained by the reverse conduction optimization according to the forward sequence of the time frames of the multi-frame image
  • S403 Obtain a repaired image of the image according to the average value of the first optimized image and the second optimized image of each image to obtain the repaired image of each image in the multiple frames of images.
  • the first repaired image of the image at the end of the forward guidance can be used as the first optimized image
  • the second repaired image of the image at the end of the reverse guidance can be used as the second optimized image
  • the first optimized image and the second optimized image corresponding to the image are obtained, the first optimized image and the second optimized image can be averaged, that is, the average of the pixel values of the same pixel is obtained, and the final repaired image is obtained.
  • the results of the optimized image obtained in the two directions of the forward guidance and the reverse guidance can be combined to obtain a repaired image with higher accuracy.
  • the embodiments of the present disclosure can use the adjustment mask and repeat the process of forward guidance and reverse guidance, which can achieve complete restoration of image pixels.
  • the embodiments of the present disclosure also provide a training process of an optical flow optimization network that performs optical flow optimization processing, wherein, as described in the above embodiments, the optical flow optimization network may include N optical flow optimization network modules.
  • the optical flow optimization network modules in can be the same, and the corresponding training methods can be the same. The following describes the training process of an optical flow optimization network module.
  • Fig. 10 shows a flowchart of a training method of an optical flow optimization module according to an embodiment of the present disclosure.
  • the step of training each optical flow optimization network module in the optical flow optimization module may include:
  • S501 Obtain training samples, where the training samples include multiple sets of optical flow samples, and mask samples and real optical flow corresponding to each set of optical flow samples;
  • the training samples used to train the optical flow optimization network module in the embodiments of the present disclosure may include multiple sets of optical flow samples, where each set of optical flow samples is the sum of the forward optical flow between adjacent images in the corresponding multi-frame image samples. / Or reverse optical flow, in addition, the training sample also includes a mask sample corresponding to each group of optical flow samples, and the mask sample is the mask image corresponding to the above-mentioned image sample. At the same time, the training sample may also include the real optical flow corresponding to the optical flow sample, that is, the real optical flow between the image samples.
  • S502 Use the optical flow optimization network module to perform optical flow optimization processing on the optical flow sample to obtain a corresponding optimized optical flow;
  • the group of optical flow samples is optimized through the initial optical flow network module of the embodiment of the present disclosure.
  • the specific optimization process is the same as the description of step S200 above, and the description will not be repeated here. Through this process, the corresponding optical flow sample can be obtained. Optimize optical flow.
  • S503 Based on the comparison result of the optimized optical flow and the real optical flow, select a first pixel that meets the first requirement from the optimized optical flow, and adjust the corresponding mask of the first pixel. The mask value in the figure to get a new mask sample;
  • the embodiment of the present disclosure can obtain the optical flow difference between the optimized optical flow and the corresponding pixel point in the real optical flow, and select the first pixel point that meets the first requirement. For example, Arrange the optical flow difference of the pixels from large to small, and select a preset ratio (for example, 50%) of pixels with a larger optical flow difference as the first pixel that meets the first requirement. Or in other embodiments, the preset ratio may be another value, which is not specifically limited in the present disclosure.
  • the mask value of the first pixel in the mask image corresponding to the optimized optical flow may be determined as the mask value corresponding to the area to be repaired. That is, in the obtained optimized optical flow, if the optical flow value error of some pixels is large, the mask value of the corresponding pixel in the corresponding mask image can be reset to the mask represented by the area to be repaired Value (such as 1) to obtain a new mask map.
  • the expression of the preset loss function may be as shown in formula (1):
  • L represents the loss function of the optical flow optimization processing
  • M is the original mask sample
  • M h represents the new mask sample
  • f represents the optimized optical flow obtained
  • Re represents the real optical flow
  • ⁇ is the weight.
  • the embodiment of the present disclosure can obtain the loss value of the optimization process according to the loss function.
  • the obtained loss value is less than the loss threshold can it be determined to meet the training requirements, and training can be terminated at this time, otherwise the parameters of the network module are adjusted until the obtained loss value meets the training requirements.
  • the embodiments of the present disclosure can perform repair processing on video images of multiple consecutive time frames at the same time.
  • the forward optical flow and the reverse optical flow between adjacent frame images can be optimized first to obtain a higher accuracy.
  • High optical flow, and then through the optimized forward optical flow and reverse optical flow to guide the video repair which can improve the accuracy of video repair, while improving the continuity and consistency between frames.
  • the embodiments of the present disclosure have a faster repair speed, which satisfies the real-time nature of repair
  • the current traditional method cannot handle the situation where there is complicated movement information in the missing area. Once the missing area has detailed information such as complicated shape, the traditional method cannot achieve video repair well, and the embodiments of the present disclosure can be used in In the absence of complex details, it can also repair video images well.
  • the writing order of the steps does not mean a strict execution order but constitutes any limitation on the implementation process.
  • the specific execution order of the steps should be based on their functions and possible internal logic. determine.
  • the present disclosure also provides video repair devices, electronic equipment, computer-readable storage media, and programs, all of which can be used to implement any of the video repair methods provided in the present disclosure.
  • video repair devices electronic equipment, computer-readable storage media, and programs, all of which can be used to implement any of the video repair methods provided in the present disclosure.
  • FIG. 11 shows a block diagram of a video repair device according to an embodiment of the present disclosure. As shown in FIG. 11, the video repair device includes:
  • the obtaining module 10 is used to obtain the first forward optical flow and the first reverse optical flow between adjacent images in the continuous multi-frame image; the optical flow optimization module 20 is used to separately perform the correction of the first positive optical flow. Perform optical flow optimization processing toward the optical flow and the first reverse optical flow to obtain a second forward optical flow corresponding to the first forward optical flow, and a second reverse optical flow corresponding to the first reverse optical flow.
  • Optical flow a conduction optimization module 30, which is used to use the second forward optical flow to perform forward conduction optimization on the continuous multi-frame image, and use the second reverse optical flow to optimize the continuous Multi-frame images perform reverse conduction optimization until the images in the optimized continuous multi-frame images meet the repair requirements; the repair module 40 is used to obtain the optimized images obtained according to the forward conduction optimization and the reverse conduction optimization. The repaired image of the continuous multi-frame image.
  • the acquisition module is further configured to acquire a mask image of each image in the multi-frame image, and determine the one between adjacent images according to the forward sequence of the time frame of the multi-frame image. And determine the reverse optical flow between adjacent images according to the reverse order of the time frames of the multi-frame images; and divide the i-th image and the i+1-th image
  • the forward optical flow of is multiplied by the mask image of the i+1th image to obtain the first forward optical flow between the ith image and the i+1th image, and the i+1th image and the
  • the reverse optical flow between the i images is multiplied by the mask image of the i image to obtain the first reverse optical flow between the i+1 image and the i image, wherein the mask image Indicates the area to be repaired of the corresponding image, i is a positive integer greater than or equal to 1 and less than G, and G is the number of images in a multi-frame image.
  • the optical flow optimization module is further configured to connect the first forward optical flow to obtain a first connected optical flow, and connect the first reverse optical flow to obtain a second connected optical flow, And respectively perform N optical flow optimization processing on the first connected optical flow and the second connected optical flow to obtain the first optimized optical flow corresponding to the first connected optical flow, and the second optical flow corresponding to the second connected optical flow.
  • N is a positive integer greater than or equal to 1
  • obtain a second forward optical flow corresponding to each first forward optical flow according to the first optimized optical flow, and according to the second optimized optical flow Obtain the second reverse optical flow corresponding to each first reverse optical flow.
  • the optical flow optimization module is further configured to perform a first optical flow optimization process on the first connected optical flow and the second connected optical flow to obtain the corresponding optical flow of the first connected optical flow.
  • the first optimized sub-optical flow, and the first optimized sub-optical flow corresponding to the second connected optical flow; and the i-th optimized sub-optical flow of the first connected optical flow and the second connected optical flow are executed respectively +1 optical flow optimization processing to obtain the i+1th optimized sub-optical flow corresponding to the first connected optical flow and the i+1th optimized sub-optical flow corresponding to the second connected optical flow, where i is greater than 1 and A positive integer less than N; wherein, through the Nth optimization process, the obtained Nth optimized sub-optical flow of the first connected optical flow is determined as the first optimized optical flow, and the obtained second The Nth optimized sub-optical flow connected to the optical flow is determined to be the second optimized optical flow, wherein each optical flow optimization processing includes residual processing and up-sampling processing.
  • the conduction optimization module includes a first conduction optimization unit, which is used to obtain a mask map of each image in the multi-frame images, and the mask map represents a corresponding image to be repaired Area; according to the forward sequence of the time frames of the multi-frame image, according to the second forward optical flow to obtain a first restoration image of the image other than the first image; adjust the corresponding based on the obtained first restoration image The mask map of the image, and return to the previous step until the adjusted mask map indicates that there is no longer a region to be repaired, then it is determined that the images in the multi-frame image meet the repair requirements.
  • a first conduction optimization unit which is used to obtain a mask map of each image in the multi-frame images, and the mask map represents a corresponding image to be repaired Area
  • the first conduction optimization unit is further configured to perform offset on the first image according to the first and second forward optical flow according to the forward sequence of the time frames of the multi-frame image. Processing to obtain the first restored image of the second image; and perform addition processing on the kth image and the restored image of the kth image to obtain the first summed image of the kth image, and according to the kth image
  • the second forward optical flow performs offset processing on the k-th first addition image to obtain the first restoration image of the k+1-th image, where k is a positive integer greater than 1 and less than G, and G represents The number of images in a multi-frame image.
  • the conduction optimization module further includes a second conduction optimization unit, which is used to obtain a mask map of each image in the multi-frame images, and the mask map represents the pending image of the corresponding image.
  • Repair area according to the reverse order of the time frame of the multi-frame image, obtain a second repair image of images other than the first image according to the second reverse optical flow; adjust based on the obtained second repair image Corresponding to the mask map of the image, and return to perform the previous step, until the adjusted mask map indicates that there is no longer an area to be repaired, then it is determined that the images in the multi-frame image meet the repair requirements.
  • the second conduction optimization unit is further configured to perform offset on the first image according to the first second reverse optical flow according to the reverse order of the time frames of the multi-frame image Processing to obtain the second restored image of the second image; and perform sum processing on the hth image and the restored image of the hth image to obtain the second summed image of the hth image, and according to the hth image
  • the second reverse optical flow performs offset processing on the h-th second summation image to obtain a second restoration image of the h+1-th image, where h is a positive integer greater than 1 and less than or equal to G, G represents the number of images in a multi-frame image.
  • the repair module is further configured to obtain the first optimized image obtained by the forward conduction optimization according to the forward sequence of the time frames of the multi-frame image, and according to the time frame of the multi-frame image In the forward sequence, the second optimized image obtained by the reverse conduction optimization is obtained; and according to the average value of the first optimized image and the second optimized image of each image, a repaired image of the image is obtained to obtain the multi-frame image The repaired image of each image in.
  • the device further includes a training module for training the optical flow optimization module, which is used for inputting training samples to the optical flow optimization module, and the training samples include multiple sets of optical flow.
  • Sample, and the mask sample and real optical flow corresponding to each optical flow sample use the optical flow optimization module to perform optical flow optimization processing on the optical flow sample to obtain the corresponding optimized optical flow; based on the optimized optical flow Compared with the real optical flow, the first pixel that meets the first requirement is selected from the optimized optical flow, and the mask value of the first pixel in the mask image is adjusted to obtain a new Mask sample; according to the preset loss function, the loss value is obtained based on the optimized optical flow, the original mask sample and the new mask sample; when the loss value meets the training requirements, the trained optical flow is obtained
  • the optimization module otherwise, adjust the network parameters of the optical flow optimization module and re-execute the optical flow optimization processing of the optical flow samples until the obtained loss value meets the training requirements.
  • the expression of the preset loss function is:
  • L represents the loss function of the optical flow optimization processing
  • M is the original mask sample
  • M h represents the new mask sample
  • f represents the optimized optical flow obtained
  • Re represents the real optical flow
  • ⁇ is the weight.
  • the functions or modules contained in the apparatus provided in the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments.
  • the functions or modules contained in the apparatus provided in the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments.
  • the embodiments of the present disclosure also provide a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the foregoing method when executed by a processor.
  • the computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.
  • An embodiment of the present disclosure also provides an electronic device, including: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured as the above method.
  • the electronic device can be provided as a terminal, server or other form of device.
  • FIG. 12 shows a block diagram of an electronic device 800 according to an embodiment of the present disclosure.
  • the electronic device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and other terminals.
  • the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, and a sensor component 814 , And communication component 816.
  • the processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations.
  • the processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the foregoing method.
  • the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components.
  • the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
  • the memory 804 is configured to store various types of data to support operations in the electronic device 800. Examples of these data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc.
  • the memory 804 can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable and Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic Disk or Optical Disk.
  • SRAM static random access memory
  • EEPROM electrically erasable programmable read-only memory
  • EPROM erasable and Programmable Read Only Memory
  • PROM Programmable Read Only Memory
  • ROM Read Only Memory
  • Magnetic Memory Flash Memory
  • Magnetic Disk Magnetic Disk or Optical Disk.
  • the power supply component 806 provides power for various components of the electronic device 800.
  • the power supply component 806 may include a power management system, one or more power supplies, and other components associated with the generation, management, and distribution of power for the electronic device 800.
  • the multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user.
  • the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user.
  • the touch panel includes one or more touch sensors to sense touch, sliding, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure related to the touch or slide operation.
  • the multimedia component 808 includes a front camera and/or a rear camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.
  • the audio component 810 is configured to output and/or input audio signals.
  • the audio component 810 includes a microphone (MIC).
  • the microphone is configured to receive an external audio signal.
  • the received audio signal may be further stored in the memory 804 or transmitted via the communication component 816.
  • the audio component 810 further includes a speaker for outputting audio signals.
  • the I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module.
  • the peripheral interface module may be a keyboard, a click wheel, a button, and the like. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
  • the sensor component 814 includes one or more sensors for providing the electronic device 800 with various aspects of state evaluation.
  • the sensor component 814 can detect the on/off status of the electronic device 800 and the relative positioning of the components.
  • the component is the display and the keypad of the electronic device 800.
  • the sensor component 814 can also detect the electronic device 800 or the electronic device 800.
  • the position of the component changes, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration/deceleration of the electronic device 800, and the temperature change of the electronic device 800.
  • the sensor component 814 may include a proximity sensor configured to detect the presence of nearby objects when there is no physical contact.
  • the sensor component 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications.
  • the sensor component 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.
  • the communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices.
  • the electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof.
  • the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel.
  • the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication.
  • the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
  • RFID radio frequency identification
  • IrDA infrared data association
  • UWB ultra-wideband
  • Bluetooth Bluetooth
  • the electronic device 800 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field A programmable gate array (FPGA), controller, microcontroller, microprocessor, or other electronic components are implemented to implement the above methods.
  • ASIC application specific integrated circuits
  • DSP digital signal processors
  • DSPD digital signal processing devices
  • PLD programmable logic devices
  • FPGA field A programmable gate array
  • controller microcontroller, microprocessor, or other electronic components are implemented to implement the above methods.
  • a non-volatile computer-readable storage medium such as the memory 804 including computer program instructions, which can be executed by the processor 820 of the electronic device 800 to complete the foregoing method.
  • FIG. 13 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure.
  • the electronic device 1900 may be provided as a server. 13
  • the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as application programs.
  • the application program stored in the memory 1932 may include one or more modules each corresponding to a set of instructions.
  • the processing component 1922 is configured to execute instructions to perform the above-described methods.
  • the electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input output (I/O) interface 1958 .
  • the electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.
  • a non-volatile computer-readable storage medium such as the memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to complete the foregoing method.
  • the present disclosure may be a system, method, and/or computer program product.
  • the computer program product may include a computer-readable storage medium loaded with computer-readable program instructions for enabling a processor to implement various aspects of the present disclosure.
  • the computer-readable storage medium may be a tangible device that can hold and store instructions used by the instruction execution device.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Non-exhaustive list of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) Or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a printer with instructions stored thereon
  • RAM random access memory
  • ROM read only memory
  • EPROM erasable programmable read only memory
  • flash memory flash memory
  • SRAM static random access memory
  • CD-ROM compact disk read-only memory
  • DVD digital versatile disk
  • memory stick floppy disk
  • mechanical encoding device such as a printer with instructions stored thereon
  • the computer-readable storage medium used here is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (for example, light pulses through fiber optic cables), or through wires Transmission of electrical signals.
  • the computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing/processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network.
  • the network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.
  • the network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network, and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing/processing device .
  • the computer program instructions used to perform the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or in one or more programming languages.
  • Source code or object code written in any combination, the programming language includes object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as "C" language or similar programming languages.
  • Computer-readable program instructions can be executed entirely on the user’s computer, partly on the user’s computer, as a stand-alone software package, partly on the user’s computer and partly on a remote computer, or entirely on the remote computer or server carried out.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to access the Internet connection).
  • LAN local area network
  • WAN wide area network
  • an electronic circuit such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be customized by using the status information of the computer-readable program instructions.
  • the computer-readable program instructions are executed to realize various aspects of the present disclosure.
  • These computer-readable program instructions can be provided to the processors of general-purpose computers, special-purpose computers, or other programmable data processing devices, thereby producing a machine that makes these instructions when executed by the processors of the computer or other programmable data processing devices , A device that implements the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams is produced. It is also possible to store these computer-readable program instructions in a computer-readable storage medium. These instructions make computers, programmable data processing apparatuses, and/or other devices work in a specific manner. Thus, the computer-readable medium storing the instructions includes An article of manufacture, which includes instructions for implementing various aspects of the functions/actions specified in one or more blocks in the flowchart and/or block diagram.
  • each block in the flowchart or block diagram may represent a module, program segment, or part of an instruction, and the module, program segment, or part of an instruction contains one or more components for realizing the specified logical function.
  • Executable instructions may also occur in a different order from the order marked in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved.
  • each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart can be implemented by a dedicated hardware-based system that performs the specified functions or actions Or it can be realized by a combination of dedicated hardware and computer instructions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)
  • Television Systems (AREA)
  • Picture Signal Circuits (AREA)

Abstract

一种视频修复方法及装置、电子设备和存储介质，所述方法包括：获取连续的多帧图像中相邻图像之间的第一正向光流和第一反向光流（S100）；分别对所述第一正向光流和第一反向光流执行光流优化处理（S200），得到与所述第一正向光流对应的第二正向光流，以及与所述第一反向光流对应的第二反向光流；利用所述第二正向光流对所述连续的多帧图像执行正向传导优化，以及利用所述第二反向光流对所述连续的多帧图像执行反向传导优化，直至优化后的连续的多帧图像中的图像均满足修复要求（S300）；根据正向传导优化以及反向传导优化获得的优化图像，获得所述连续的多帧图像的修复图像（S400）。

Description

视频修复方法及装置、电子设备和存储介质
本公开要求在2019年1月31日提交中国专利局、申请号为201910099762.4、申请名称为“视频修复方法及装置、电子设备和存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本公开涉及视频修复领域，特别涉及一种视频修复方法及装置、电子设备和存储介质。
背景技术
图像修复一直是计算机视觉中的一个重要问题，在许多领域都有着重要的应用，例如，可以通过图像修复实现图像去水印以及图像修补等目的。视频修复作为图像修复问题在视频上的一种延伸，其与单纯的图像修复问题相比，实现难度大大增加。但是随着互联网的发展，视频修复算法的应用有着更加广泛的前景，例如需要去除视频水印以及去除视频中的某些部分等。
发明内容
本公开实施例提供了一种视频修复方案。
根据本公开的一方面，提供了一种视频修复方法，其包括：获取连续的多帧图像中相邻图像之间的第一正向光流和第一反向光流；分别对所述第一正向光流和第一反向光流执行光流优化处理，得到与所述第一正向光流对应的第二正向光流，以及与所述第一反向光流对应的第二反向光流；利用所述第二正向光流对所述连续的多帧图像执行正向传导优化，以及利用所述第二反向光流对所述连续的多帧图像执行反向传导优化，直至优化后的连续的多帧图像中的图像均满足修复要求；根据正向传导优化以及反向传导优化获得的优化图像，获得所述连续的多帧图像的修复图像。
在一些可能的实施方式中，所述获取连续的多帧图像中相邻图像之间的第一正向光流和第一反向光流，包括：获取所述多帧图像中每个图像的掩码图，所述掩码图表示对应图像的待修复区域；根据所述多帧图像的时间帧的正向顺序，确定相邻图像之间的正向光流，以及根据所述多帧图像的时间帧的反向顺序，确定相邻图像的之间的反向光流；将第i个图像和第i+1个图像之间的正向光流与第i+1个图像的掩码图相乘，得到第i个图像和第i+1个图像之间第一正向光流，以及将第i+1个图像和第i个图像之间的反向光流与第i个图像的掩码图相乘，得到第i+1个图像和第i个图像之间第一反向光流，其中i为大于或者等于1且小于G的正整数，G为多帧图像中图像的数量。
在一些可能的实施方式中，所述分别对所述第一正向光流和第一反向光流执行光流优化处理，包括：连接所述第一正向光流得到第一连接光流，以及连接所述第一反向光流得到第二连接光流；分别对所述第一连接光流和第二连接光流执行N次光流优化处理，得到所述第一连接光流对应的第一优化光流，以及第二连接光流对应的第二优化光流，N为大于或者等于1的正整数；根据所述第一优化光流得到每个第一正向光流对应的第二正向光流，以及根据所述第二优化光流得到每个第一反向光流对应的第二反向光流。
在一些可能的实施方式中，所述分别对所述第一连接光流和第二连接光流执行N次光流优化处理，包括：对所述第一连接光流和第二连接光流执行第一次光流优化处理，得到所述第一连接光流对应的第一优化子光流，以及第二连接光流对应的第一优化子光流；以及分别对所述第一连接光流和所述第二连接光流的第i优化子光流执行第i+1次光流优化处理，得到所述第一连接光流对应的第i+1优化子光流，以及第二连接光流对应的第i+1优化子光流，其中i为大于1且小于N的正整数；其中，通过第N次优化处理，将得到的所述第一连接光流的第N优化子光流确定为所述第一优化光流，以及将得到的所述第二连接光流的第N优化子光流确定为所述第二优化光流，其中，各次光流优化处理包括残差处理和上采样处理。
在一些可能的实施方式中，所述利用所述第二正向光流对所述连续的多帧图像执行正向传导优化，包括：获取所述多帧图像中的每个图像的掩码图，所述掩码图表示对应图像的待修复区域；按照所述多帧图像的时间帧的正向顺序，根据所述第二正向光流得到第一个图像以外的图像的第一修复图；基于得到的所述第一修复图调整对应图像的掩码图，并返回执行前一步骤，直至调整后的掩码图表示不再存在待修复区域，则确定所述多帧图像中的图像满足所述修复要求。
在一些可能的实施方式中，所述按照所述多帧图像的时间帧的正向顺序，根据所述第二正向光流得到每个图像的第一修复图，包括：按照所述多帧图像的时间帧的正向顺序，根据第一个第二正向光流对第一个图像执行偏移处理，得到第二个图像的第一修复图；将第k个图像和所述第k个图像的修复图执行加和处理，得到第k个图像的第一加和图像，以及按照第k个第二正向光流对所述第k个第一加和图像执行偏移处理，得到第k+1个图像的第一修复图，其中k为大于1且小于G的正整数，G表示多帧图像中图像的数量。
在一些可能的实施方式中，所述利用所述第二反向光流对所述连续的多帧图像执行反向传导优化，包括：获取所述多帧图像中的每个图像的掩码图，所述掩码图表示对应图像的待修复区域；按照所述多帧图像的时间帧的反向顺序，根据所述第二反向光流得到第一个图像以外的图像的第二修复图；基于得到的所述第二修复图调整对应图像的掩码图，并返回执行前一步骤，直至调整后的掩码图表示不再存在待修复区域，则确定所述多帧图像中的图像满足所述修复要求。
在一些可能的实施方式中，所述按照所述多帧图像的时间帧的反向顺序，根据所述第二反向光流得到第一个图像以外的图像的第二修复图，包括：按照所述多帧图像的时间帧的反向顺序，根据第一个第二反向光流对第一个图像执行偏移处理，得到第二个图像的第二修复图；将第h个图像和所述第h个图像的修复图执行加和处理，得到第h个图像的第二加和图像，以及按照第h个第二反向光流对所述第h个第二加和图像执行偏移处理，得到第h+1个图像的第二修复图，其中h为大于1且小于或等于G的正整数，G表示多帧图像中图像的数量。
在一些可能的实施方式中，所述根据正向传导优化以及反向传导优化获得的优化图像，获得所述连续的多帧图像的修复图像，包括：按照多帧图像的时间帧的正向顺序，获得所述正向传导优化得到的第一优化图像；按照多帧图像的时间帧的正向顺序，获得所述反向传导优化得到的第二优化图像；根据每个图像的第一优化图像和第二优化图像的均值，得到该图像的修复图像，以得到所述多帧图像中每个图像的修复图像。
在一些可能的实施方式中，所述方法包括：通过光流优化模块执行所述分别对所述第一正向光流和第一反向光流执行光流优化处理，得到与所述第一正向光流对应的第二正向光流，以及与所述第一反向光流对应的第二反向光流；所述方法还包括：对所述光流优化模块进行训练的步骤，其包括：获取训练样本，所述训练样本包括多组光流样本，以及每个光流样本对应的掩码样本和真实光流；利用所述光流优化模块对所述光流样本进行光流优化处理，得到对应的优化光流；基于所述优化光流和所述真实光流的比较结果，从所述优化光流中选择出满足第一要求的第一像素点，并调整所述第一像素点在掩码图中的掩码值以得到新的掩码样本；按照预设损失函数，基于所述优化光流、原始的掩码样本以及新的掩码样本得到损失值；在所述损失值满足训练要求的情况下，得到训练完成的光流优化模块，否则，调整所述光流优化模块的网络参数并重新执行光流样本的光流优化处理，直至得到的损失值满足训练要求。
在一些可能的实施方式中，所述预设损失函数的表达式为：
Figure PCTCN2019130428-appb-000001
其中，L表示光流优化处理的损失函数，M为原始的掩码样本，M h表示新的掩码样本，f表示得到的优化光流，
Figure PCTCN2019130428-appb-000002
表示真实光流，λ为权值。
根据本公开的第二方面，提供了一种视频修复装置，其包括：
获取模块，其用于获取连续的多帧图像中相邻图像之间的第一正向光流和第一反向光流；光流优化模块，其用于分别对所述第一正向光流和第一反向光流执行光流优化处理，得到与所述第一正向光流对应的第二正向光流，以及与所述第一反向光流对应的第二反向光流；传导优化模块，其用于利用所述第二正向光流对所述连续的多帧图像执行正向传导优化，以及利用所述第二反向光流对所述连续 的多帧图像执行反向传导优化，直至优化后的连续的多帧图像中的图像均满足修复要求；修复模块，其用于根据正向传导优化以及反向传导优化获得的优化图像，获得所述连续的多帧图像的修复图像。
在一些可能的实施方式中，所述获取模块还用于获取所述多帧图像中每个图像的掩码图，并根据所述多帧图像的时间帧的正向顺序，确定相邻图像之间的正向光流，以及根据所述多帧图像的时间帧的反向顺序，确定相邻图像的之间的反向光流；以及将第i个图像和第i+1个图像之间的正向光流与第i+1个图像的掩码图相乘，得到第i个图像和第i+1个图像之间第一正向光流，以及将第i+1个图像和第i个图像之间的反向光流与第i个图像的掩码图相乘，得到第i+1个图像和第i个图像之间第一反向光流，其中，所述掩码图表示对应图像的待修复区域，i为大于或者等于1且小于G的正整数，G为多帧图像中图像的数量。
在一些可能的实施方式中，所述光流优化模块还用于连接所述第一正向光流得到第一连接光流，以及连接所述第一反向光流得到第二连接光流，并分别对所述第一连接光流和第二连接光流执行N次光流优化处理，得到所述第一连接光流对应的第一优化光流，以及第二连接光流对应的第二优化光流，N为大于或者等于1的正整数；以及根据所述第一优化光流得到每个第一正向光流对应的第二正向光流，以及根据所述第二优化光流得到每个第一反向光流对应的第二反向光流。
在一些可能的实施方式中，所述光流优化模块还用于对所述第一连接光流和第二连接光流执行第一次光流优化处理，得到所述第一连接光流对应的第一优化子光流，以及第二连接光流对应的第一优化子光流；以及分别对所述第一连接光流和所述第二连接光流的第i优化子光流执行第i+1次光流优化处理，得到所述第一连接光流对应的第i+1优化子光流，以及第二连接光流对应的第i+1优化子光流，其中i为大于1且小于N的正整数；其中，通过第N次优化处理，将得到的所述第一连接光流的第N优化子光流确定为所述第一优化光流，以及将得到的所述第二连接光流的第N优化子光流确定为所述第二优化光流，其中，各次光流优化处理包括残差处理和上采样处理。
在一些可能的实施方式中，所述传导优化模块包括第一传导优化单元，其用于获取所述多帧图像中的每个图像的掩码图，所述掩码图表示对应图像的待修复区域；按照所述多帧图像的时间帧的正向顺序，根据所述第二正向光流得到第一个图像以外的图像的第一修复图；基于得到的所述第一修复图调整对应图像的掩码图，并返回执行前一步骤，直至调整后的掩码图表示不再存在待修复区域，则确定所述多帧图像中的图像满足所述修复要求。
在一些可能的实施方式中，所述第一传导优化单元还用于按照所述多帧图像的时间帧的正向顺序，根据第一个第二正向光流对第一个图像执行偏移处理，得到第二个图像的第一修复图；以及将第k个图像和所述第k个图像的修复图执行加和处理，得到第k个图像的第一加和图像，以及按照第k个第二正向光流对所述第k个第一加和图像执行偏移处理，得到第k+1个图像的第一修复图，其中k为大于1且小于G的正整数，G表示多帧图像中图像的数量。
在一些可能的实施方式中，所述传导优化模块还包括第二传导优化单元，其用于获取所述多帧图像中的每个图像的掩码图，所述掩码图表示对应图像的待修复区域；按照所述多帧图像的时间帧的反向顺序，根据所述第二反向光流得到第一个图像以外的图像的第二修复图；基于得到的所述第二修复图调整对应图像的掩码图，并返回执行前一步骤，直至调整后的掩码图表示不再存在待修复区域，则确定所述多帧图像中的图像满足所述修复要求。
在一些可能的实施方式中，所述第二传导优化单元还用于按照所述多帧图像的时间帧的反向顺序，根据第一个第二反向光流对第一个图像执行偏移处理，得到第二个图像的第二修复图；以及将第h个图像和所述第h个图像的修复图执行加和处理，得到第h个图像的第二加和图像，以及按照第h个第二反向光流对所述第h个第二加和图像执行偏移处理，得到第h+1个图像的第二修复图，其中h为大于1且小于或等于G的正整数，G表示多帧图像中图像的数量。
在一些可能的实施方式中，所述修复模块还用于按照多帧图像的时间帧的正向顺序，获得所述正向传导优化得到的第一优化图像，并按照多帧图像的时间帧的正向顺序，获得所述反向传导优化得到的第二优化图像；以及根据每个图像的第一优化图像和第二优化图像的均值，得到该图像的修复图像，以得到所述多帧图像中每个图像的修复图像。
在一些可能的实施方式中，还包括训练模块，其用于训练所述光流优化模块，其用于向所述光流优化模块输入训练样本，所述训练样本包括多组光流样本，以及每个光流样本对应的掩码样本和真实光流；利用所述光流优化模块对所述光流样本进行光流优化处理，得到对应的优化光流；基于所述优化光流和所述真实光流的比较结果，从所述优化光流中选择出满足第一要求的第一像素点，并调整所述第一像素点在掩码图中的掩码值以得到新的掩码样本；按照预设损失函数，基于所述优化光流、原始的掩码样本以及新的掩码样本得到损失值；在所述损失值满足训练要求的情况下，得到训练完成的光流优化模块，否则，调整所述光流优化模块的网络参数并重新执行光流样本的光流优化处理，直至得到的损失值满足训练要求。
在一些可能的实施方式中，所述预设损失函数的表达式为
Figure PCTCN2019130428-appb-000003
其中，L表示光流优化处理的损失函数，M为原始的掩码样本，M h表示新的掩码样本，f表示得到的优化光流，
Figure PCTCN2019130428-appb-000004
表示真实光流，λ为权值。
根据本公开的第三方面，提供了一种电子设备，其包括：
处理器；
用于存储处理器可执行指令的存储器；
其中，所述处理器被配置为：执行第一方面中任意一项所述的方法。
根据本公开的第四方面，提供了一种计算机可读存储介质，其上存储有计算机程序指令，其特征在于，所述计算机程序指令被处理器执行时实现第一方面中任意一项所述的方法。
根据本公开的第五方面，一种计算机程序，所述计算机程序包括计算机可读代码，当所述计算机可读代码在电子设备中运行时，所述电子设备中的处理器执行用于实现第一方面中任意一项所述的方法。
本公开实施例可以同时多个连续时间帧的视频图像执行修复处理，其中，可以首先对相邻帧图像之间的正向光流和反向光流进行优化，得到精度较高修复后的光流，而后再通过优化后的正向光流以及反向光流对视频修复进行引导，完成视频图像的修改，该过程可以提高视频修复的精确度，同时提高帧之间的连续性和一致性。另外，本公开实施例具有较快的修复速度，满足修复的实时性。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，而非限制本公开。
根据下面参考附图对示例性实施例的详细说明，本公开的其它特征及方面将变得清楚。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，这些附图示出了符合本公开的实施例，并与说明书一起用于说明本公开的技术方案。
图1示出根据本公开实施例的一种视频修复方法的流程图；
图2示出根据本公开实施例的一种视频修复方法中的步骤S100的流程图；
图3示出根据本公开实施例的一种视频修复方法中步骤S200的流程图；
图4示出根据本公开实施例的光流优化网络的结构示意图；
图5示出根据本公开实施例的一种视频修复方法的步骤S300的流程图；
图6示出根据本公开实施例的一种视频修复方法的步骤S302的流程图；
图7示出根据本公开实施例的一种视频修复方法中步骤S300的流程图；
图8示出根据本公开实施例的一种视频修复方法的步骤S3002的流程图；
图9示出根据本公开实施例的一种视频修复方法中步骤S400的流程图；
图10示出根据本公开实施例的光流优化模块的训练方法的流程图；
图11示出根据本公开实施例的一种视频修复装置的框图；
图12示出根据本公开实施例的一种电子设备800的框图；
图13示出根据本公开实施例的一种电子设备1900的框图。
具体实施方式
以下将参考附图详细说明本公开的各种示例性实施例、特征和方面。附图中相同的附图标记表示功能相同或相似的元件。尽管在附图中示出了实施例的各种方面，但是除非特别指出，不必按比例绘制附图。在这里专用的词“示例性”意为“用作例子、实施例或说明性”。这里作为“示例性”所说明的任何实施例不必解释为优于或好于其它实施例。
本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中术语“至少一种”表示多种中的任意一种或多种中的至少两种的任意组合，例如，包括A、B、C中的至少一种，可以表示包括从A、B和C构成的集合中选择的任意一个或多个元素。
另外，为了更好地说明本公开，在下文的具体实施方式中给出了众多的具体细节。本领域技术人员应当理解，没有某些具体细节，本公开同样可以实施。在一些实例中，对于本领域技术人员熟知的方法、手段、元件和电路未作详细描述，以便于凸显本公开的主旨。
图1示出根据本公开实施例的一种视频修复方法的流程图，本公开实施例的视频修复方法可以用于修复视频图像，并且本公开实施例可以保证连续视频帧修复的一致性，且修复精度高，效率快。
本公开实施例可以应用在图像处理设备或者视频采集设备中，其中图像处理设备可以为任意的终端设备或者服务器设备，如手机、摄像机、笔记本电脑、PAD、云服务器等，只要具有图像处理功能或者能够采集视频信息，即可以应用本公开实施例的视频修复方法。下面对本公开实施例详细说明。
如图1所示，本公开实施例的视频修复方法可以包括：
S100：获取连续的多帧图像中相邻图像之间的第一正向光流和第一反向光流；
S200：分别对所述第一正向光流和第一反向光流执行光流优化处理，得到与所述第一正向光流对应的第二正向光流，以及与所述第一反向光流对应的第二反向光流；
S300：利用所述第二正向光流对所述连续的多帧图像执行正向传导优化，以及利用所述第二反向光流对所述连续的多帧图像执行反向传导优化，直至优化后的连续的多帧图像中的图像均满足修复要求；
S400：根据正向传导优化以及反向传导优化获得的优化图像，获得所述连续的多帧图像的修复图像。
本公开实施例中，可以通过构建的神经网络执行视频的修复过程，例如可以通过深度学习神经网络实现视频图像的修复，但本公开对此不进行具体限定，本公开实施例也可以通过与本公开实施例限定的功能相匹配的算法执行视频修复过程。
步骤S100之前，可以首先获取待修复的视频图像，获取连续的视频帧图像(即上述连续的多帧图像)，其中，本公开实施例可以获取预设数量个连续时间帧的图像，例如5帧图像，或者在其他实施例中，预设数量也可以为其他数值，本领域技术人员可以根据不同的需求和情况自行设定。
在获取连续的多帧图像之后，即可以对该多帧图像进行修复处理。首先，在步骤S100中可以获取相邻图像之间的光流，其中相邻图像是指相邻的两帧图像，由于时间帧具有时序性，本公开实施例可以从时间的正向和反向两个方向对图像进行处理，实现视频的修复。
另外，光流(optical flow)可以用于表示运动图像之间的变化，是指时变图像中模式运动速度。当目标物体在运动时，它在图像上对应点的亮度模式也在运动，因此光流可以用于表示图像之间的变化，由于它包含了目标运动的信息，因此可被观察者用来确定目标的运动情况。
对应的，本公开实施例可以在步骤S100中获取每两个相邻图像之间的光流，例如按照时间帧的正向顺序，获得两个相邻图像之间的第一正向光流，以及按照时间帧的反向顺序，获得两个相邻图像之间的第一正向光流。其中，假定本公开实施例的多帧图像为{P 1,P 2,...P G}，即可以包括G帧图像，如G可 以为5或者其他数值。按照时间帧的正向顺序可以获取第一正向光流{f 1→2,f 2→3,...,f G-1→G}，其中f i→i+1表示图像P i到图像P i+1的第一正向光流，i为大于或者等于1且小于G之任意整数。另外，按照时间帧的反向顺序可以获取第一反向光流{f G→G-1,f G-1→G-2,...,f 2→1}，其中，f j+1→j表示图像P j+1到图像P j的第一反向光流，j为大于或者等于1且小于G的整数。
其中，本公开实施例可以利用光流估计模型获得图像之间的第一正向光流和第一反向光流，例如光流估计模型可以为FlowNet 2.0，在本公开的其他实施例中，也可以通过其他方式获取图像之间的光流，本公开对此不进行具体限定。
另外，为了提高光流优化的精确度，本公开实施例还可以结合掩码图对图像之间的光流进行处理，得到第一正向光流和第一反向光流。图2示出根据本公开实施例的一种视频修复方法中的步骤S100的流程图。其中，所述获取连续的多帧图像中相邻图像之间的第一正向光流和第一反向光流，可以包括：
S101：获取所述多帧图像中每个图像的掩码图，所述掩码图表示对应图像的待修复区域；
本公开实施例中，获得的多帧图像中的每帧图像都可以具有对应的掩码图，该掩码图可以是人为确定的。例如，可以根据每个图像中需要执行修复的区域建立对应的掩码图，其中掩码图的尺度与对应图像的尺度相同，并且，掩码图中的像素点与相应图像的像素点是一一对应的。本公开实施例中，可以将待修复区域中的像素点在掩码图中的掩码定义为1，其余不需要修复的区域的像素点在掩码图中的掩码定义为0，或者也可以将待修复区域中的像素点在掩码图中的掩码定义为0，其余不需要修复的区域的像素点在掩码图中的掩码定义为1，本领域技术人员可以根据不同的需求设置掩码，本公开对此不进行具体限定。
S102：根据所述多帧图像的时间帧的正向顺序，确定相邻图像之间的正向光流，以及根据所述多帧图像的时间帧的反向顺序，确定相邻图像的之间的反向光流；
该过程与上述实施例中描述的光流确定过程相同，例如可以通过光流估计模型获取多帧图像中相邻图像之间的正向光流和反向光流，在此对该过程不再详细赘述。
在此需要说明的是，步骤S101和步骤S102的顺序，本公开实施例不作具体限定，步骤S101可以在步骤S102之前执行，也可以在步骤S102之后执行，或者也可以步骤S101和步骤S102同时执行。只要在执行步骤S103确定了每个图像的掩码图以及任意两个相邻图像之间的正向光流和反向光流即可以作为本公开实施例。
S103：将第i个图像和第i+1个图像之间的正向光流与第i+1个图像的掩码图相乘，得到第i个图像和第i+1个图像之间第一正向光流，以及将第i+1个图像和第i个图像之间的反向光流与第i个图像的掩码图相乘，得到第i+1个图像和第i个图像之间第一反向光流，其中i为大于或者等于1且小于G的正整数，G为多帧图像中图像的数量。
即，本公开实施例可以将正向光流与对应的掩码图相乘，即确定出正向光流中需要优化的光流区域，得到对应的第一正向光流。同样的，也可以将反向光流与对应的掩码图相乘，即可以确定出反向光流中需要优化的光流区域，得到对应的第一反向光流。
如上述实施例所述，多帧图像可以为{P 1,P 2,...P G}，得到的相邻图像之间的正向光流可以为{f 1→2,f 2→3,...,f G-1→G}，图像之间的反向光流可以为{f G→G-1,f G-1→G-2,...,f 2→1}。其中，每个图像的掩码图可以表示为{M 1，M 2，...，M G}，其中，M k表示第k个图像P k的掩码图，k为大于或者等于1且小于或者等于G的整数。
对应的，本公开实施例在得到第一个图像P 1和第二个图像P 2之间的正向光流之后，可以将该正 向光流与第二个图像P 2的掩码图M 2相乘，如对应元素相乘，得到第一个图像P 1和第二个图像P 2之间的第一正向光流。同样的，可以得到第二个图像P 2和第三个图像P 3之间的正向光流，并将该正向光流与第三个图像P 3的掩码图M 3相乘，如对应元素相乘，得到第二个图像P 2和第三个图像P 3之间的第一正向光流。同理，可以将第i个图像和第i+1个图像之间的正向光流与第i+1个图像的掩码图相乘，如对应元素相乘，得到第i个图像和第i+1个图像之间第一正向光流。通过上述方式即可以获得相邻图像之间的第一正向光流{f 1→2,f 2→3,...,f G-1→G}。
另外，同理，本公开实施例可以在得到最后一个图像P G和倒数第二个图像P G-1之间的反向光流之后，可以将该反向光流与该倒数第二个图像P G-1的掩码图M G-1相乘，如对应元素相乘，得到最后一个图像P G和倒数第二个图像P G-1之间的第一反向光流。同样的，可以得到倒数第二个图像P G-1和倒数第三个图像P G-2之间的反向光流，并将该反向光流与倒数第三个图像P G-2的掩码图M G-2相乘，如对应元素相乘，得到倒数第二个图像P G-1和倒数第三个图像P G-2之间的第一反向光流。同理，可以将第i+1个图像和第i个图像之间的反向光流与第i个图像的掩码图相乘，如对应元素相乘，得到第i+1个图像和第i个图像之间第一反向光流。通过上述方式即可以获得相邻图像之间的第一反向光流{f G→G-1,f G-1→G-2,...,f 2→1}。
本公开实施例通过将得到的正向光流与反向光流与对应图像的掩码图相乘，获得对应的第一正向光流和第一反向光流，由于得到的第一正向光流和第一反向光流中可以包括需要执行光流优化的区域，可以提高光流的优化精度。
在得到图像之间的第一正向光流和第一反向光流之后，即可以对光流执行优化处理，即可以执行步骤S200。本公开实施例中，可以对第一正向光流和第一反向光流分别执行优化，得到对应的第二正向光流和第二反向光流。本公开实施例的光流优化过程可以通过光流优化网络实现，该光流优化网络可以包括至少一个光流优化网络模块，每个光流优化网络模块相同，根据光流优化网络模块的数量可以确定光流优化的次数，本公开实施例对光流优化处理的次数不作具体限定，本领域技术人员可以根据不同的需要进行选择和设定。下面分别对第一正向光流和第一反向光流的优化过程进行详细说明。
图3示出根据本公开实施例的一种视频修复方法中步骤S200的流程图，其中，所述分别对所述第一正向光流和第一反向光流执行光流优化处理，包括：
S201：连接所述第一正向光流得到第一连接光流，以及连接所述第一反向光流得到第二连接光流；
本公开实施例，在执行光流优化之前，首先将相邻图像之间的第一正向光流分别连接，如分别级联。其中在深度方向上将第一正向光流连接，形成由多层第一正向光流构成的第一连接光流。同样的，也可以将相邻图像之间的第一反向光流分别连接，例如在深度方向上将第一反向光流连接，形成由多层第一方向光流构成的第二连接光流。
S202：分别对所述第一连接光流和第二连接光流执行N次光流优化处理，得到所述第一连接光流对应的第一优化光流，以及第二连接光流对应的第二优化光流，N为大于或者等于1的正整数；
在获得第一连接光流和第二连接光流之后，可以分别对第一连接光流和第二连接光流执行光流优化处理，如上述实施例所述，本公开实施例可以执行至少一次光流优化处理过程。例如本公开实施例中每次的光流优化处理可以利用光流优化模块执行，该光流优化模块可以由神经网络构成，或者也可 以利用对应的算法执行优化操作。对应的，在执行N次光流优化处理时，可以包括N个依次相连的光流优化网络模块，后一个光流优化网络模块的输入为前一光流优化网络模块的输出，最后一个光流优化网络模块的输出即为对第一连接光流和第二连接光流的优化结果。
在包括一个光流优化网络模块时，可以利用该光流优化网络模块对第一连接光流执行优化处理得到第一连接光流对应的第一优化子光流，以及通过光流优化网络模块对第二连接光流执行优化处理，得到第二连接光流对应的第二优化子光流。其中光流优化处理可以包括残差处理和上采样处理。即光流优化网络模块中可以进一步包括残差单元和上采样单元，通过残差单元对输入的第一连接光流或者第二连接光流执行残差处理，其中残差单元可以包括多个卷积层，每个卷积层采用的卷积核本公开实施例不作具体限定，通过残差单元残差处理后的第一连接光流的尺度变小，如减小为输入的连接光流的尺度的四分之一，本公开对此不作具体限定，可以根据需求设定。在执行残差处理之后，可以对残差处理后的第一连接光流或者第二连接光流执行上采样处理，通过上采样处理可以将输出的第一优化子光流的尺度调整成第一连接光流的尺度，以及将输出的第二优化子光流的尺度调整成第二连接光流的尺度。且通过光流优化过程可以融合多个光流的特征，同时可以提高光流精度。
在另一些实施例中，光流优化模块也可以包括多个光流优化网络模块，如N个光流优化网络模块。其中第一个光流优化网络模块可以接收第一连接光流和第二连接光流，并分别对第一连接光流和第二连接光流执行第一次光流优化处理，该第一次光流优化处理包括残差处理和升采样处理，其中具体过程与上述实施例相同，在此不再赘述。通过该第一次光流优化处理可以得到第一连接光流的第一优化子光流以及第二连接光流的第一优化子光流。
进一步的，将该第一连接子光流和第二连接子光流输入至第二个光流优化网络模块，执行第二次光流优化处理，其中第二个光流优化网络模块同样也可以包括执行残差处理的残差单元和执行升采样处理的升采样单元，通过第二个光流优化网络模块可以执行第一连接子光流和第二连接子光流的光流优化处理，其中具体过程与上述实施例相同，在此不再赘述。通过该第二次光流优化处理可以得到第一连接光流的二优化子光流以及第二连接光流的第二优化子光流。
同理，可以利用每个光流优化网络模块执行一次光流优化过程，即可以利用第i+1个光流优化网络模块对所述第一连接光流和所述第二连接光流的第i优化子光流执行第i+1次光流优化处理，得到所述第一连接光流对应的第i+1优化子光流，以及第二连接光流对应的第i+1优化子光流，其中i为大于1且小于N的正整数。最终可以通过第N个光流优化网络模块执行的第N次优化处理，得到第一连接光流的第N优化子光流以及第二连接光流的第N优化子光流，并且可以将得到的所述第一连接光流的第N优化子光流确定为所述第一优化光流，以及将得到的所述第二连接光流的第N优化子光流确定为所述第二优化光流。本公开实施例中，每个光流优化网络模块执行的光流优化处理过程可以残差处理和上采样处理。即每个光流优化网络模块可以为相同的光流优化模块。
例如，图4示出根据本公开实施例的光流优化网络的结构示意图，其中可以包括三个光流优化网络模块A、B和C。该三个光流优化网络模块可以分别通过残差单元和上采样单元组成。其中，可以通过第一个光流优化网络模块A可以执行第一连接光流f 0和第二连接光流f 1的第一次光流优化处理，得到第一连接光流的第一连接子光流f 1，以及第二连接光流的第一连接子光流f 1'。将该第一连接子光流f 1和f 1'分别输入至第二个光流优化网络模块B，执行第二次光流优化处理，得到第一连接光流的第二连接子光流f 2和第二连接光流的第二连接子光流f 2'。进一步地，将第二连接子光流f 2和f 2'分别输入至第三个光流优化网络模块C，分别执行第三次光流优化处理，得到第一连接光流对应的第三优化子光流f 3和第二连接光流对应的第三优化子光流f 3'。此时可以将最后一次光流优化处理得到的第一连接光流的第三优化子光流确定为第一优化光流，以及将最后一次光流优化处理得到的第二 连接光流的第三优化子光流确定为第二优化光流。
S203：根据所述第一优化光流得到每个第一正向光流对应的第二正向光流，以及根据所述第二优化光流得到每个第一反向光流对应的第二反向光流。
在经过N次光流优化处理后，得到第一优化光流的尺度和第一连接光流的尺度相同，可以按照深度方向将该第一优化光流拆分成G个第二正向光流，该G个第二正向光流分别对应的为第一正向光流的优化结果。同样的，在经过N次光流优化处理后，得到第二优化光流的尺度和第二连接光流的尺度相同，可以按照深度方向将该第二优化光流拆分成G个第二反向光流，该G个第二反向光流分别对应的为第一反向光流的优化结果。
通过上述实施例，即可以得到图像之间的第一正向光流优化后的第二正向光流，以及图像之间的第一反向光流优化后的第二反向光流。通过对光流进行优化可以实现对视频修复的精确度。
在得到优化后的光流之后，即可以利用该优化后的第二正向光流和第二反向光流分别执行多帧图像的正向引导和反向引导，得到最终的修复图像。下面对正向引导和反向引导的过程进行分别说明。
图5示出根据本公开实施例的一种视频修复方法的步骤S300的流程图，其中，所述利用所述第二正向光流对所述连续的多帧图像执行正向传导优化，包括：
S301：获取所述多帧图像中的每个图像的掩码图，所述掩码图表示对应图像的待修复区域；
如上所述，本公开实施例中每个图像都可以对应有掩码图，其用于表示相应图像中需要被修复的待修复区域，该区域内的像素点对应的掩码可以为1，其余区域的掩码可以为0，或者待修复区域的像素点的掩码可以为0，其余区域的像素点的掩码可以为1。本公开对此不作具体限定。
S302：按照所述多帧图像的时间帧的正向顺序，根据所述第二正向光流得到第一个图像以外的图像的第一修复图；
本公开实施例中，获得的第二正向光流表示在时间帧的正向顺序上相邻的两个图像之间的光流，即前一个图像到下一个图像的光流。因此，可以根据得到的第二正向光流对图像进行引导，即确定每个图像在对应的第二正向光流的光流变化的条件下得到的新的图像，该新的图像即为下一个图像的第一修复图。
图6示出根据本公开实施例的一种视频修复方法的步骤S302的流程图，本公开实施例中，所述按照所述多帧图像的时间帧的正向顺序，根据所述第二正向光流得到每个图像的第一修复图，可以包括：
S3021：按照所述多帧图像的时间帧的正向顺序，根据第一个第二正向光流对第一个图像执行偏移处理，得到第二个图像的第一修复图；
S3022：将第k个图像和所述第k个图像的修复图执行加和处理，得到第k个图像的第一加和图像，以及按照第k个第二正向光流对所述第k个第一加和图像执行偏移处理，得到第k+1个图像的第一修复图，其中k为大于1且小于G的正整数，G表示多帧图像中图像的数量。
其中，可以按照时间帧的正向顺序，将第二正向光流表示为{f′ 1→2,f′ 2→3,...,f′ G-1→G}，其中，f′ i-1→i表示第i-1个图像P i-1和第i个图像P i之间的第二正向光流，i为大于1且小于或者等于G的整数。对应的，第二反向光流可以表示为{f′ G→G-1,f′ G-2→G-3,...,f 2→1}，其中f′ j+1→j表示第j+1个图像P j+1与第j个图像P j之间的第二反向光流，j为大于或者等于1且小于G的整数。
对应的，可以根据第一个第二正向光流f′ 1→2的光流变化，可以对第一个图像P 1进行偏移处理，得到新的图像P′ 2，该P′ 2可以为第二个图像P 2的第一修复图。而后可以将第二图像P 2及其第一修复图P′ 2执行相加处理，如元素对应相加，得到第二个图像P 2的第一加和图。通过该加和处理，可以实现利用优化后的第二正向光流实现第二图像的待修复区域的修复，例如补充相应的像素值，或者调整像 素值等。
进而，可以利用第二个第二正向光流f′ 2→3的光流变化，对第二个图像P 1的第一加和图像进行偏移处理，得到新的图像P′ 3，该P′ 3可以为第三个图像P 3的第一修复图。而后可以将第三个图像P 3及其第一修复图P′ 3执行相加处理，如元素对应相加，得到第三个图像P 3的第一加和图像。通过该加和处理，可以实现利用优化后的第二正向光流实现第三个图像的待修复区域的修复，例如补充相应的像素值，或者调整像素值等。
同理的，可以利用第k个第二正向光流f′ k→k+1的光流变化，对第k个图像P k的第一加和图像进行偏移处理，得到新的图像P′ k，该P′ k可以为第k个图像P k的第一修复图。而后可以将第k个图像P k及其第一修复图P′ k执行相加处理，如元素对应相加，得到第k+1个图像P k+1的第一加和图像。通过该加和处理，可以实现利用优化后的第二正向光流实现第k个图像的待修复区域的修复，例如补充相应的像素值，或者调整像素值等。
通过上述实施例，即可以获得每个图像的第一修复图，该第一修复图中可以有部分待修复区域的像素值被补充(即修复)。
S303：基于得到的所述第一修复图调整对应图像的掩码图，并返回执行前一步骤，直至调整后的掩码图表示不再存在待修复区域，则确定所述多帧图像中的图像满足所述修复要求。
如上述实施例所述，在得到图像的第一修复图之后，如果第k个图像的第一修复图中的待修复区域的某像素点被修复，即修复出像素值，则可以将该第k个图像的掩码图中对应像素点的掩码值置返，如原掩码值为1，则调整为0，反之亦然。
在调整完图像的掩码图之后，可以对得到的第一修复图重复执行步骤S302，直至每个图像对应的掩码图中不再存在待修复区域，此时完成了图像的正向引导过程，即可以得到每个图像的第一优化图像。
图7示出根据本公开实施例的一种视频修复方法中步骤S300的流程图，其中所述利用所述第二反向光流对所述连续的多帧图像执行反向传导优化，可以包括：
S3001：获取所述多帧图像中的每个图像的掩码图，所述掩码图表示对应图像的待修复区域；
该过程与步骤S301相同，在此不再重复赘述。
S3002：按照所述多帧图像的时间帧的反向顺序，根据所述第二反向光流得到第一个图像以外的图像的第二修复图；
本公开实施例中，获得的第二反向光流表示在时间帧的反向顺序上相邻的两个图像之间的光流，即后一个图像到前一个图像的光流。因此，可以根据得到的第二反向光流对图像进行反向引导，即确定每个图像在对应的第二反向光流的光流变化的条件下得到的新的图像，该新的图像即为下一个图像的第一修复图。
图8示出根据本公开实施例的一种视频修复方法的步骤S3002的流程图，本公开实施例中，所述按照所述多帧图像的时间帧的反向顺序，根据所述第二反向光流得到每个图像的第二修复图，可以包括：
S30021：按照所述多帧图像的时间帧的反向顺序，根据第一个第二反向光流对第一个图像执行偏移处理，得到第二个图像的第二修复图；
S30022：将第h个图像和所述第h个图像的修复图执行加和处理，得到第h个图像的第二加和图像，以及按照第h个第二反向光流对所述第h个第二加和图像执行偏移处理，得到第h+1个图像的第二修复图，其中h为大于1且小于或等于G的正整数，G表示多帧图像中图像的数量。
其中在此需要说明的是，本公开实施例按照时间帧的正向顺序图像可以表示为{P 1,P 2,...,P k},对应的，按照时间帧的反向顺序图像可以表示为{P G,P G-1,...,P 1}。并且，反向顺序的第一个图像可以为P G，对应的第二个图像可以为P G-1，依次类推，第G个图像为P 1。
并且，可以按照时间帧的反向顺序，将第二反向光流表示为{f′ 1→2,f′ 2→3,...,f′ G-1→G}，其中，f′ i-1→i表示第i-1个图像P i-1和第i个图像P i之间的第二正向光流，i为大于1且小于或者等于G的整数。对应的，第二反向光流可以表示为{f′ G→G-1，f′ G-1→G-2,...，f′ 2→1}，其中f′ j+1→j表示第j+1个图像P j+1与第j个图像P j之间的第二反向光流，j为大于或者等于1且小于G的整数。
步骤S30021中，根据第一个第二反向光流f′ G→G-1的光流变化，可以对反向顺序中的第一个图像P G进行偏移处理，得到新的图像P″ G-1，该P″ G-1可以为反向顺序中的第二个图像P G-1的第二修复图。而后可以将反向顺序的第二个图像P G-1及其第二修复图P″ G-1执行相加处理，如元素对应相加，得到第二个图像P G-1的第二加和图像。通过该加和处理，可以实现利用优化后的第二反向光流实现倒数第二个图像的待修复区域的修复，例如补充相应的像素值，或者调整像素值等。
进而，可以利用第二个第二反向光流f′ G-1→G-2的光流变化，对反向顺序的第二个图像P G-1的第二加和图像进行偏移处理，得到新的图像P″ G-2，该P″ G-2可以为反向顺序的第三个图像P G-2的第二修复图。而后可以将反向顺序的第三个图像P G-2及其第一修复图P″ G-2执行相加处理，如元素对应相加，得到该反向顺序的第三个图像P G-2的第一加和图像。通过该加和处理，可以实现利用优化后的第二反向光流实现倒数第三个图像的待修复区域的修复，例如补充相应的像素值，或者调整像素值等。
同理的，可以利用第h个第二反向光流f′ h+1→h的光流变化，对反向顺序的第h个图像P G-h+1的第二加和图像进行偏移处理，得到新的图像P″ G-h，该P″ G-h可以为反向顺序的第h+1个图像P G-h的第二修复图。而后可以将反向顺序的第h+1个图像P G-h及其第二修复图P″ G-h执行相加处理，如元素对应相加，得到反向顺序第h+1个图像P G-h的第一加和图像。通过该加和处理，可以实现利用优化后的第二反向光流实现倒数第h+1个图像的待修复区域的修复，例如补充相应的像素值，或者调整像素值等。
通过上述实施例，即可以获得每个图像的第二修复图，该第二修复图中可以有部分待修复区域的像素值被补充(即修复)。
S3003：基于得到的所述第二修复图调整对应图像的掩码图，并返回执行前一步骤，直至调整后的掩码图表示不再存在待修复区域，则确定所述多帧图像中的图像满足所述修复要求。
如上述实施例所述，在得到图像的第二修复图之后，如果第k个图像的第二修复图中的待修复区域的某像素点被修复，即修复出像素值，则可以将该第k个图像的掩码图中对应像素点的掩码值置返，如原掩码值为1，则调整为0，反之亦然。
在调整完图像的掩码图之后，可以对得到的第二修复图重复执行步骤S3002，直至每个图像对应的掩码图中不再存在待修复区域，此时完成了图像的反向引导过程，即可以得到每个图像的第二优化图像。
在得到多帧图像的图像的第一优化图像和第二优化图像之后，可以执行步骤S400。图9示出根据本公开实施例的一种视频修复方法中步骤S400的流程图。其中所述根据正向传导优化以及反向传导优化获得的优化图像，获得所述连续的多帧图像的修复图像，可以包括：
S401：按照多帧图像的时间帧的正向顺序，获得所述正向传导优化得到的第一优化图像；
S402：按照多帧图像的时间帧的正向顺序，获得所述反向传导优化得到的第二优化图像；
S403：根据每个图像的第一优化图像和第二优化图像的均值，得到该图像的修复图像，以得到所述多帧图像中每个图像的修复图像。
如上述实施所述，可以利用在正向引导结束时图像的第一修复图像作为第一优化图像，以及利用在反向引导结束时图像的第二修复图像作为第二优化图像。
在得到图像对应的第一优化图像和第二优化图像之后，即可以将第一优化图像和第二优化图像做均值处理，即获得相同像素点的像素值的均值，得到最终的修复图像。通过该方式，可以结合正向引导和反向引导两个方向上得到的优化图像的结果，得到精确度更高的修复图像。另外，本公开实施例可以利用调整掩码并重复执行正向引导和反向引导的过程，能够实现图像像素点的完整修复。
另外，本公开实施例还提供了执行光流优化处理的光流优化网络的训练过程，其中，如上述实施例所述，光流优化网络可以包括N个光流优化网络模块，本公开实施例中的光流优化网络模块可以相同，对应的训练方式可以相同，下面通过一个光流优化网络模块的训练过程进行说明。
图10示出根据本公开实施例的光流优化模块的训练方法的流程图。其中，对所述光流优化模块中的每个光流优化网络模块进行训练的步骤，可以包括：
S501：获取训练样本，所述训练样本包括多组光流样本，以及每组光流样本对应的掩码样本和真实光流；
本公开实施例对光流优化网络模块进行训练所使用的训练样本可以包括多组光流样本，其中每组光流样本为相应的多帧图像样本中相邻图像之间的正向光流和/或反向光流，另外，训练样本中还包括每组光流样本对应的掩码样本，该掩码样本即为上述图像样本对应的掩码图。同时，训练样本中还可以包括光流样本对应的真实光流，即图像样本之间的真实光流。
S502：利用所述光流优化网络模块对所述光流样本进行光流优化处理，得到对应的优化光流；
通过本公开实施例的初始光流网络模块对组光流样本进行优化，具体的优化过程与上述步骤S200的描述相同，在此不再重复说明，通过该过程可以得到每个光流样本对应的优化光流。
S503：基于所述优化光流和所述真实光流的比较结果，从所述优化光流中选择出满足第一要求的第一像素点，并调整所述第一像素点在对应的掩码图中的掩码值，以得到新的掩码样本；
在得到光流样本的优化光流之后，本公开实施例可以获取优化光流与对应的真实光流中像素点对应的光流差，从中选出满足第一要求的第一像素点，例如可以将像素点的光流差从大到小排列，从中选取光流差较大的预设比例(如50％)的像素点作为满足第一要求的第一像素点。或者在其他实施例中，预设比例可以为其他值，本公开对此不进行具体限定。
在确定了满足第一要求的第一像素点之后，可以将所述优化光流对应掩码图中该第一像素点的掩码值确定为待修复区域对应的掩码值。即在获得的优化光流中，如果某些像素点的光流值误差较大，则可以将该对应的掩码图中的相应像素点的掩码值重新设置成待修复区域表征的掩码值(如1)，从而可以获得新的掩码图。
S504：按照预设损失函数，基于所述优化光流、原始的掩码样本以及新的掩码样本得到损失值；
其中，所述预设损失函数的表达式可以如公式(1)所示：
Figure PCTCN2019130428-appb-000005
其中，L表示光流优化处理的损失函数，M为原始的掩码样本，M h表示新的掩码样本，f表示得到的优化光流，
Figure PCTCN2019130428-appb-000006
表示真实光流，λ为权值。
本公开实施例可以根据该损失函数得到优化过程的损失值。
S505：在所述损失值满足训练要求的情况下，得到训练完成的光流优化网络模块，否则，调整所述光流优化网络模块的网络参数并重新执行光流样本的光流优化处理，直至得到的损失值满足训练要求。
例如，只有在得到的损失值小于损失阈值时，才能确定为满足训练要求，此时可以终止训练，否则调整网络模块的参数，直至得到的损失值满足训练要求。
综上所述，本公开实施例可以同时多个连续时间帧的视频图像执行修复处理，其中，可以首先对相邻帧图像之间的正向光流和反向光流进行优化，得到精度较高的光流，而后再通过优化后的正向光流以及反向光流对视频修复进行引导，可以提高视频修复的精确度，同时提高帧之间的连续性和一致性。另外，本公开实施例具有较快的修复速度，满足修复的实时性
另一方面，目前传统的方法无法处理缺失区域存在复杂的移动信息的情况，一旦缺失区域存在复杂的形状等细节信息，传统的方法则不能很好地实现视频修复，而本公开实施例可以在缺失复杂细节的情况下，也能很好的修复视频图像。
本领域技术人员可以理解，在具体实施方式的上述方法中，步骤的撰写顺序并不意味着严格的执行顺序而对实施过程构成任何限定，步骤的具体执行顺序应当以其功能和可能的内在逻辑确定。
可以理解，本公开提及的上述个方法实施例，在不违背原理逻辑的情况下，均可以彼此相互结合形成结合后的实施例，限于篇幅，本公开不再赘述。
此外，本公开还提供了视频修复装置、电子设备、计算机可读存储介质、程序，上述均可用来实现本公开提供的任一种视频修复方法，相应技术方案和描述和参见方法部分的相应记载，不再赘述。
图11示出根据本公开实施例的一种视频修复装置的框图，如图11所示，所述视频修复装置包括：
获取模块10，其用于获取连续的多帧图像中相邻图像之间的第一正向光流和第一反向光流；光流优化模块20，其用于分别对所述第一正向光流和第一反向光流执行光流优化处理，得到与所述第一正向光流对应的第二正向光流，以及与所述第一反向光流对应的第二反向光流；传导优化模块30，其用于利用所述第二正向光流对所述连续的多帧图像执行正向传导优化，以及利用所述第二反向光流对所述连续的多帧图像执行反向传导优化，直至优化后的连续的多帧图像中的图像均满足修复要求；修复模块40，其用于根据正向传导优化以及反向传导优化获得的优化图像，获得所述连续的多帧图像的修复图像。
在一些可能的实施方式中，所述获取模块还用于获取所述多帧图像中每个图像的掩码图，并根据所述多帧图像的时间帧的正向顺序，确定相邻图像之间的正向光流，以及根据所述多帧图像的时间帧的反向顺序，确定相邻图像的之间的反向光流；以及将第i个图像和第i+1个图像之间的正向光流与第i+1个图像的掩码图相乘，得到第i个图像和第i+1个图像之间第一正向光流，以及将第i+1个图像和第i个图像之间的反向光流与第i个图像的掩码图相乘，得到第i+1个图像和第i个图像之间第一反向光流，其中，所述掩码图表示对应图像的待修复区域，i为大于或者等于1且小于G的正整数，G为多帧图像中图像的数量。
在一些可能的实施方式中，所述光流优化模块还用于连接所述第一正向光流得到第一连接光流，以及连接所述第一反向光流得到第二连接光流，并分别对所述第一连接光流和第二连接光流执行N次光流优化处理，得到所述第一连接光流对应的第一优化光流，以及第二连接光流对应的第二优化光流，N为大于或者等于1的正整数；以及根据所述第一优化光流得到每个第一正向光流对应的第二正向光流， 以及根据所述第二优化光流得到每个第一反向光流对应的第二反向光流。
在一些可能的实施方式中，所述光流优化模块还用于对所述第一连接光流和第二连接光流执行第一次光流优化处理，得到所述第一连接光流对应的第一优化子光流，以及第二连接光流对应的第一优化子光流；以及分别对所述第一连接光流和所述第二连接光流的第i优化子光流执行第i+1次光流优化处理，得到所述第一连接光流对应的第i+1优化子光流，以及第二连接光流对应的第i+1优化子光流，其中i为大于1且小于N的正整数；其中，通过第N次优化处理，将得到的所述第一连接光流的第N优化子光流确定为所述第一优化光流，以及将得到的所述第二连接光流的第N优化子光流确定为所述第二优化光流，其中，各次光流优化处理包括残差处理和上采样处理。
在一些可能的实施方式中，所述传导优化模块包括第一传导优化单元，其用于获取所述多帧图像中的每个图像的掩码图，所述掩码图表示对应图像的待修复区域；按照所述多帧图像的时间帧的正向顺序，根据所述第二正向光流得到第一个图像以外的图像的第一修复图；基于得到的所述第一修复图调整对应图像的掩码图，并返回执行前一步骤，直至调整后的掩码图表示不再存在待修复区域，则确定所述多帧图像中的图像满足所述修复要求。
在一些可能的实施方式中，所述第一传导优化单元还用于按照所述多帧图像的时间帧的正向顺序，根据第一个第二正向光流对第一个图像执行偏移处理，得到第二个图像的第一修复图；以及将第k个图像和所述第k个图像的修复图执行加和处理，得到第k个图像的第一加和图像，以及按照第k个第二正向光流对所述第k个第一加和图像执行偏移处理，得到第k+1个图像的第一修复图，其中k为大于1且小于G的正整数，G表示多帧图像中图像的数量。
在一些可能的实施方式中，所述传导优化模块还包括第二传导优化单元，其用于获取所述多帧图像中的每个图像的掩码图，所述掩码图表示对应图像的待修复区域；按照所述多帧图像的时间帧的反向顺序，根据所述第二反向光流得到第一个图像以外的图像的第二修复图；基于得到的所述第二修复图调整对应图像的掩码图，并返回执行前一步骤，直至调整后的掩码图表示不再存在待修复区域，则确定所述多帧图像中的图像满足所述修复要求。
在一些可能的实施方式中，所述第二传导优化单元还用于按照所述多帧图像的时间帧的反向顺序，根据第一个第二反向光流对第一个图像执行偏移处理，得到第二个图像的第二修复图；以及将第h个图像和所述第h个图像的修复图执行加和处理，得到第h个图像的第二加和图像，以及按照第h个第二反向光流对所述第h个第二加和图像执行偏移处理，得到第h+1个图像的第二修复图，其中h为大于1且小于或等于G的正整数，G表示多帧图像中图像的数量。
在一些可能的实施方式中，所述修复模块还用于按照多帧图像的时间帧的正向顺序，获得所述正向传导优化得到的第一优化图像，并按照多帧图像的时间帧的正向顺序，获得所述反向传导优化得到的第二优化图像；以及根据每个图像的第一优化图像和第二优化图像的均值，得到该图像的修复图像，以得到所述多帧图像中每个图像的修复图像。
在一些可能的实施方式中，所述装置还包括训练模块，其用于训练所述光流优化模块，其用于向所述光流优化模块输入训练样本，所述训练样本包括多组光流样本，以及每个光流样本对应的掩码样本和真实光流；利用所述光流优化模块对所述光流样本进行光流优化处理，得到对应的优化光流；基于所述优化光流和所述真实光流的比较结果，从所述优化光流中选择出满足第一要求的第一像素点，并调整所述第一像素点在掩码图中的掩码值以得到新的掩码样本；按照预设损失函数，基于所述优化光流、原始的掩码样本以及新的掩码样本得到损失值；在所述损失值满足训练要求的情况下，得到训练完成的光流优化模块，否则，调整所述光流优化模块的网络参数并重新执行光流样本的光流优化处理，直至得到的损失值满足训练要求。
在一些可能的实施方式中，所述预设损失函数的表达式为：
Figure PCTCN2019130428-appb-000007
其中，L表示光流优化处理的损失函数，M为原始的掩码样本，M h表示新的掩码样本，f表示得到的优化光流，
Figure PCTCN2019130428-appb-000008
表示真实光流，λ为权值。
在一些实施例中，本公开实施例提供的装置具有的功能或包含的模块可以用于执行上文方法实施例描述的方法，其具体实现可以参照上文方法实施例的描述，为了简洁，这里不再赘述。
本公开实施例还提出一种计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令被处理器执行时实现上述方法。计算机可读存储介质可以是易失性或非易失性计算机可读存储介质。
本公开实施例还提出一种电子设备，包括：处理器；用于存储处理器可执行指令的存储器；其中，所述处理器被配置为上述方法。电子设备可以被提供为终端、服务器或其它形态的设备。
图12示出根据本公开实施例的一种电子设备800的框图。例如，电子设备800可以是移动电话，计算机，数字广播终端，消息收发设备，游戏控制台，平板设备，医疗设备，健身设备，个人数字助理等终端。
参照图12，电子设备800可以包括以下一个或多个组件：处理组件802，存储器804，电源组件806，多媒体组件808，音频组件810，输入/输出(I/O)的接口812，传感器组件814，以及通信组件816。
处理组件802通常控制电子设备800的整体操作，诸如与显示，电话呼叫，数据通信，相机操作和记录操作相关联的操作。处理组件802可以包括一个或多个处理器820来执行指令，以完成上述的方法的全部或部分步骤。此外，处理组件802可以包括一个或多个模块，便于处理组件802和其他组件之间的交互。例如，处理组件802可以包括多媒体模块，以方便多媒体组件808和处理组件802之间的交互。
存储器804被配置为存储各种类型的数据以支持在电子设备800的操作。这些数据的示例包括用于在电子设备800上操作的任何应用程序或方法的指令，联系人数据，电话簿数据，消息，图片，视频等。存储器804可以由任何类型的易失性或非易失性存储设备或者它们的组合实现，如静态随机存取存储器(SRAM)，电可擦除可编程只读存储器(EEPROM)，可擦除可编程只读存储器(EPROM)，可编程只读存储器(PROM)，只读存储器(ROM)，磁存储器，快闪存储器，磁盘或光盘。
电源组件806为电子设备800的各种组件提供电力。电源组件806可以包括电源管理系统，一个或多个电源，及其他与为电子设备800生成、管理和分配电力相关联的组件。
多媒体组件808包括在所述电子设备800和用户之间的提供一个输出接口的屏幕。在一些实施例中，屏幕可以包括液晶显示器(LCD)和触摸面板(TP)。如果屏幕包括触摸面板，屏幕可以被实现为触摸屏，以接收来自用户的输入信号。触摸面板包括一个或多个触摸传感器以感测触摸、滑动和触摸面板上的手势。所述触摸传感器可以不仅感测触摸或滑动动作的边界，而且还检测与所述触摸或滑动操作相关的持续时间和压力。在一些实施例中，多媒体组件808包括一个前置摄像头和/或后置摄像头。当电子设备800处于操作模式，如拍摄模式或视频模式时，前置摄像头和/或后置摄像头可以接收外部的多媒体数据。每个前置摄像头和后置摄像头可以是一个固定的光学透镜系统或具有焦距和光学变焦能力。
音频组件810被配置为输出和/或输入音频信号。例如，音频组件810包括一个麦克风(MIC)，当电子设备800处于操作模式，如呼叫模式、记录模式和语音识别模式时，麦克风被配置为接收外部音频信号。所接收的音频信号可以被进一步存储在存储器804或经由通信组件816发送。在一些实施例中，音频组件810还包括一个扬声器，用于输出音频信号。
I/O接口812为处理组件802和外围接口模块之间提供接口，上述外围接口模块可以是键盘，点击轮，按钮等。这些按钮可包括但不限于：主页按钮、音量按钮、启动按钮和锁定按钮。
传感器组件814包括一个或多个传感器，用于为电子设备800提供各个方面的状态评估。例如，传感器组件814可以检测到电子设备800的打开/关闭状态，组件的相对定位，例如所述组件为电子设备800的显示器和小键盘，传感器组件814还可以检测电子设备800或电子设备800一个组件的位置改变，用户与电子设备800接触的存在或不存在，电子设备800方位或加速/减速和电子设备800的温度变化。传感器组件814可以包括接近传感器，被配置用来在没有任何的物理接触时检测附近物体的存在。传 感器组件814还可以包括光传感器，如CMOS或CCD图像传感器，用于在成像应用中使用。在一些实施例中，该传感器组件814还可以包括加速度传感器，陀螺仪传感器，磁传感器，压力传感器或温度传感器。
通信组件816被配置为便于电子设备800和其他设备之间有线或无线方式的通信。电子设备800可以接入基于通信标准的无线网络，如WiFi，2G或3G，或它们的组合。在一个示例性实施例中，通信组件816经由广播信道接收来自外部广播管理系统的广播信号或广播相关信息。在一个示例性实施例中，所述通信组件816还包括近场通信(NFC)模块，以促进短程通信。例如，在NFC模块可基于射频识别(RFID)技术，红外数据协会(IrDA)技术，超宽带(UWB)技术，蓝牙(BT)技术和其他技术来实现。
在示例性实施例中，电子设备800可以被一个或多个应用专用集成电路(ASIC)、数字信号处理器(DSP)、数字信号处理设备(DSPD)、可编程逻辑器件(PLD)、现场可编程门阵列(FPGA)、控制器、微控制器、微处理器或其他电子元件实现，用于执行上述方法。
在示例性实施例中，还提供了一种非易失性计算机可读存储介质，例如包括计算机程序指令的存储器804，上述计算机程序指令可由电子设备800的处理器820执行以完成上述方法。
图13示出根据本公开实施例的一种电子设备1900的框图。例如，电子设备1900可以被提供为一服务器。参照图13，电子设备1900包括处理组件1922，其进一步包括一个或多个处理器，以及由存储器1932所代表的存储器资源，用于存储可由处理组件1922的执行的指令，例如应用程序。存储器1932中存储的应用程序可以包括一个或一个以上的每一个对应于一组指令的模块。此外，处理组件1922被配置为执行指令，以执行上述方法。
电子设备1900还可以包括一个电源组件1926被配置为执行电子设备1900的电源管理，一个有线或无线网络接口1950被配置为将电子设备1900连接到网络，和一个输入输出(I/O)接口1958。电子设备1900可以操作基于存储在存储器1932的操作系统，例如Windows ServerTM，Mac OS XTM，UnixTM,LinuxTM，FreeBSDTM或类似。
在示例性实施例中，还提供了一种非易失性计算机可读存储介质，例如包括计算机程序指令的存储器1932，上述计算机程序指令可由电子设备1900的处理组件1922执行以完成上述方法。
本公开可以是系统、方法和/或计算机程序产品。计算机程序产品可以包括计算机可读存储介质，其上载有用于使处理器实现本公开的各个方面的计算机可读程序指令。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是――但不限于――电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(SRAM)、便携式压缩盘只读存储器(CD-ROM)、数字多功能盘(DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本公开操作的计算机程序指令可以是汇编指令、指令集架构(ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等，以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、 部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络—包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、现场可编程门阵列(FPGA)或可编程逻辑阵列(PLA)，该电子电路可以执行计算机可读程序指令，从而实现本公开的各个方面。
这里参照根据本公开实施例的方法、装置(系统)和计算机程序产品的流程图和/或框图描述了本公开的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器，从而生产出一种机器，使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上，使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本公开的多个实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
以上已经描述了本公开的各实施例，上述说明是示例性的，并非穷尽性的，并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实施例的原理、实际应用或对市场中技术的技术改进，或者使本技术领域的其它普通技术人员能理解本文披露的各实施例。

Claims (23)

  1. 一种视频修复方法，其特征在于，包括：
    获取连续的多帧图像中相邻图像之间的第一正向光流和第一反向光流；
    分别对所述第一正向光流和第一反向光流执行光流优化处理，得到与所述第一正向光流对应的第二正向光流，以及与所述第一反向光流对应的第二反向光流；
    利用所述第二正向光流对所述连续的多帧图像执行正向传导优化，以及利用所述第二反向光流对所述连续的多帧图像执行反向传导优化，直至优化后的连续的多帧图像中的图像满足修复要求；
    根据正向传导优化以及反向传导优化获得的优化图像，获得所述连续的多帧图像的修复图像。
  2. 根据权利要求1所述的方法，其特征在于，所述获取连续的多帧图像中相邻图像之间的第一正向光流和第一反向光流，包括：
    获取所述多帧图像中每个图像的掩码图，所述掩码图表示对应图像的待修复区域；
    根据所述多帧图像的时间帧的正向顺序，确定相邻图像之间的正向光流，以及根据所述多帧图像的时间帧的反向顺序，确定相邻图像的之间的反向光流；
    将第i个图像和第i+1个图像之间的正向光流与第i+1个图像的掩码图相乘，得到第i个图像和第i+1个图像之间第一正向光流，以及将第i+1个图像和第i个图像之间的反向光流与第i个图像的掩码图相乘，得到第i+1个图像和第i个图像之间第一反向光流，其中i为大于或者等于1且小于G的正整数，G为多帧图像中图像的数量。
  3. 根据权利要求1或2所述的方法，其特征在于，所述分别对所述第一正向光流和第一反向光流执行光流优化处理，包括：
    连接所述第一正向光流得到第一连接光流，以及连接所述第一反向光流得到第二连接光流；
    分别对所述第一连接光流和第二连接光流执行N次光流优化处理，得到所述第一连接光流对应的第一优化光流，以及第二连接光流对应的第二优化光流，N为大于或者等于1的正整数；
    根据所述第一优化光流得到每个第一正向光流对应的第二正向光流，以及根据所述第二优化光流得到每个第一反向光流对应的第二反向光流。
  4. 根据权利要求3所述的方法，其特征在于，所述分别对所述第一连接光流和第二连接光流执行N次光流优化处理，包括：
    对所述第一连接光流和第二连接光流执行第一次光流优化处理，得到所述第一连接光流对应的第一优化子光流，以及第二连接光流对应的第一优化子光流；以及
    分别对所述第一连接光流和所述第二连接光流的第i优化子光流执行第i+1次光流优化处理，得到所述第一连接光流对应的第i+1优化子光流，以及第二连接光流对应的第i+1优化子光流，其中i为大于1且小于N的正整数；
    其中，通过第N次优化处理，将得到的所述第一连接光流的第N优化子光流确定为所述第一优化光流，以及将得到的所述第二连接光流的第N优化子光流确定为所述第二优化光流，其中，各次光流优化处理包括残差处理和上采样处理。
  5. 根据权利要求1-4中任意一项所述的方法，其特征在于，所述利用所述第二正向光流对所述连续的多帧图像执行正向传导优化，包括：
    获取所述多帧图像中的每个图像的掩码图，所述掩码图表示对应图像的待修复区域；
    按照所述多帧图像的时间帧的正向顺序，根据所述第二正向光流得到第一个图像以外的图像的第一修复图；
    基于得到的所述第一修复图调整对应图像的掩码图，并返回执行前一步骤，直至调整后的掩码图表示不再存在待修复区域，则确定所述多帧图像中的图像满足所述修复要求。
  6. 根据权利要求5所述的方法，其特征在于，所述按照所述多帧图像的时间帧的正向顺序，根据所述第二正向光流得到每个图像的第一修复图，包括：
    按照所述多帧图像的时间帧的正向顺序，根据第一个第二正向光流对第一个图像执行偏移处理，得到第二个图像的第一修复图；
    将第k个图像和所述第k个图像的修复图执行加和处理，得到第k个图像的第一加和图像，以及按 照第k个第二正向光流对所述第k个第一加和图像执行偏移处理，得到第k+1个图像的第一修复图，其中k为大于1且小于G的正整数，G表示多帧图像中图像的数量。
  7. 根据权利要求1-6中任意一项所述的方法，其特征在于，所述利用所述第二反向光流对所述连续的多帧图像执行反向传导优化，包括：
    获取所述多帧图像中的每个图像的掩码图，所述掩码图表示对应图像的待修复区域；
    按照所述多帧图像的时间帧的反向顺序，根据所述第二反向光流得到第一个图像以外的图像的第二修复图；
    基于得到的所述第二修复图调整对应图像的掩码图，并返回执行前一步骤，直至调整后的掩码图表示不再存在待修复区域，则确定所述多帧图像中的图像满足所述修复要求。
  8. 根据权利要求7所述的方法，其特征在于，所述按照所述多帧图像的时间帧的反向顺序，根据所述第二反向光流得到第一个图像以外的图像的第二修复图，包括：
    按照所述多帧图像的时间帧的反向顺序，根据第一个第二反向光流对第一个图像执行偏移处理，得到第二个图像的第二修复图；
    将第h个图像和所述第h个图像的修复图执行加和处理，得到第h个图像的第二加和图像，以及按照第h个第二反向光流对所述第h个第二加和图像执行偏移处理，得到第h+1个图像的第二修复图，其中h为大于1且小于或等于G的正整数，G表示多帧图像中图像的数量。
  9. 根据权利要求1-8中任意一项所述的方法，其特征在于，所述根据正向传导优化以及反向传导优化获得的优化图像，获得所述连续的多帧图像的修复图像，包括：
    按照多帧图像的时间帧的正向顺序，获得所述正向传导优化得到的第一优化图像；
    按照多帧图像的时间帧的正向顺序，获得所述反向传导优化得到的第二优化图像；
    根据每个图像的第一优化图像和第二优化图像的均值，得到该图像的修复图像，以得到所述多帧图像中每个图像的修复图像。
  10. 根据权利要求1-9中任意一项所述的方法，其特征在于，所述方法包括：通过光流优化模块执行所述分别对所述第一正向光流和第一反向光流执行光流优化处理，得到与所述第一正向光流对应的第二正向光流，以及与所述第一反向光流对应的第二反向光流；
    所述方法还包括：对所述光流优化模块进行训练的步骤，其包括：
    获取训练样本，所述训练样本包括多组光流样本，以及每个光流样本对应的掩码样本和真实光流；
    利用所述光流优化模块对所述光流样本进行光流优化处理，得到对应的优化光流；
    基于所述优化光流和所述真实光流的比较结果，从所述优化光流中选择出满足第一要求的第一像素点，并调整所述第一像素点在掩码图中的掩码值以得到新的掩码样本；
    按照预设损失函数，基于所述优化光流、原始的掩码样本以及新的掩码样本得到损失值；
    在所述损失值满足训练要求的情况下，得到训练完成的光流优化模块，否则，调整所述光流优化模块的网络参数并重新执行光流样本的光流优化处理，直至得到的损失值满足训练要求。
  11. 一种视频修复装置，其特征在于，包括：
    获取模块，其用于获取连续的多帧图像中相邻图像之间的第一正向光流和第一反向光流；
    光流优化模块，其用于分别对所述第一正向光流和第一反向光流执行光流优化处理，得到与所述第一正向光流对应的第二正向光流，以及与所述第一反向光流对应的第二反向光流；
    传导优化模块，其用于利用所述第二正向光流对所述连续的多帧图像执行正向传导优化，以及利用所述第二反向光流对所述连续的多帧图像执行反向传导优化，直至优化后的连续的多帧图像中的图像均满足修复要求；
    修复模块，其用于根据正向传导优化以及反向传导优化获得的优化图像，获得所述连续的多帧图像的修复图像。
  12. 根据权利要求11所述的装置，其特征在于，所述获取模块还用于获取所述多帧图像中每个图像的掩码图，并根据所述多帧图像的时间帧的正向顺序，确定相邻图像之间的正向光流，以及根据所述多帧图像的时间帧的反向顺序，确定相邻图像的之间的反向光流；以及
    将第i个图像和第i+1个图像之间的正向光流与第i+1个图像的掩码图相乘，得到第i个图像和第i+1个图像之间第一正向光流，以及将第i+1个图像和第i个图像之间的反向光流与第i个图像的掩码图相乘，得到第i+1个图像和第i个图像之间第一反向光流，其中，所述掩码图表示对应图像的待修复区域，i为大于或者等于1且小于G的正整数，G为多帧图像中图像的数量。
  13. 根据权利要求11或12所述的装置，其特征在于，所述光流优化模块还用于连接所述第一正向光流得到第一连接光流，以及连接所述第一反向光流得到第二连接光流，并
    分别对所述第一连接光流和第二连接光流执行N次光流优化处理，得到所述第一连接光流对应的第一优化光流，以及第二连接光流对应的第二优化光流，N为大于或者等于1的正整数；以及
    根据所述第一优化光流得到每个第一正向光流对应的第二正向光流，以及根据所述第二优化光流得到每个第一反向光流对应的第二反向光流。
  14. 根据权利要求13所述的装置，其特征在于，所述光流优化模块还用于对所述第一连接光流和第二连接光流执行第一次光流优化处理，得到所述第一连接光流对应的第一优化子光流，以及第二连接光流对应的第一优化子光流；以及
    分别对所述第一连接光流和所述第二连接光流的第i优化子光流执行第i+1次光流优化处理，得到所述第一连接光流对应的第i+1优化子光流，以及第二连接光流对应的第i+1优化子光流，其中i为大于1且小于N的正整数；
    其中，通过第N次优化处理，将得到的所述第一连接光流的第N优化子光流确定为所述第一优化光流，以及将得到的所述第二连接光流的第N优化子光流确定为所述第二优化光流，其中，各次光流优化处理包括残差处理和上采样处理。
  15. 根据权利要求11-14中任意一项所述的装置，其特征在于，所述传导优化模块包括第一传导优化单元，其用于获取所述多帧图像中的每个图像的掩码图，所述掩码图表示对应图像的待修复区域；
    按照所述多帧图像的时间帧的正向顺序，根据所述第二正向光流得到第一个图像以外的图像的第一修复图；
    基于得到的所述第一修复图调整对应图像的掩码图，并返回执行前一步骤，直至调整后的掩码图表示不再存在待修复区域，则确定所述多帧图像中的图像满足所述修复要求。
  16. 根据权利要求15所述的装置，其特征在于，所述第一传导优化单元还用于按照所述多帧图像的时间帧的正向顺序，根据第一个第二正向光流对第一个图像执行偏移处理，得到第二个图像的第一修复图；以及
    将第k个图像和所述第k个图像的修复图执行加和处理，得到第k个图像的第一加和图像，以及按照第k个第二正向光流对所述第k个第一加和图像执行偏移处理，得到第k+1个图像的第一修复图，其中k为大于1且小于G的正整数，G表示多帧图像中图像的数量。
  17. 根据权利要求11-16中任意一项所述的装置，其特征在于，所述传导优化模块还包括第二传导优化单元，其用于获取所述多帧图像中的每个图像的掩码图，所述掩码图表示对应图像的待修复区域；
    按照所述多帧图像的时间帧的反向顺序，根据所述第二反向光流得到第一个图像以外的图像的第二修复图；
    基于得到的所述第二修复图调整对应图像的掩码图，并返回执行前一步骤，直至调整后的掩码图表示不再存在待修复区域，则确定所述多帧图像中的图像满足所述修复要求。
  18. 根据权利要求17所述的装置，其特征在于，所述第二传导优化单元还用于按照所述多帧图像的时间帧的反向顺序，根据第一个第二反向光流对第一个图像执行偏移处理，得到第二个图像的第二修复图；以及
    将第h个图像和所述第h个图像的修复图执行加和处理，得到第h个图像的第二加和图像，以及按照第h个第二反向光流对所述第h个第二加和图像执行偏移处理，得到第h+1个图像的第二修复图，其中h为大于1且小于或等于G的正整数，G表示多帧图像中图像的数量。
  19. 根据权利要求11-18中任意一项所述的装置，其特征在于，所述修复模块还用于按照多帧图像的时间帧的正向顺序，获得所述正向传导优化得到的第一优化图像，并
    按照多帧图像的时间帧的正向顺序，获得所述反向传导优化得到的第二优化图像；以及
    根据每个图像的第一优化图像和第二优化图像的均值，得到该图像的修复图像，以得到所述多帧图像中每个图像的修复图像。
  20. 根据权利要求11-19中任意一项所述的装置，其特征在于，还包括训练模块，其用于训练所述光流优化模块，其用于向所述光流优化模块输入训练样本，所述训练样本包括多组光流样本，以及每个光流样本对应的掩码样本和真实光流；
    利用所述光流优化模块对所述光流样本进行光流优化处理，得到对应的优化光流；
    基于所述优化光流和所述真实光流的比较结果，从所述优化光流中选择出满足第一要求的第一像素点，并调整所述第一像素点在掩码图中的掩码值以得到新的掩码样本；
    按照预设损失函数，基于所述优化光流、原始的掩码样本以及新的掩码样本得到损失值；
    在所述损失值满足训练要求的情况下，得到训练完成的光流优化模块，否则，调整所述光流优化模块的网络参数并重新执行光流样本的光流优化处理，直至得到的损失值满足训练要求。
  21. 一种电子设备，其特征在于，包括：
    处理器；
    用于存储处理器可执行指令的存储器；
    其中，所述处理器被配置为：执行权利要求1至10中任意一项所述的方法。
  22. 一种计算机可读存储介质，其上存储有计算机程序指令，其特征在于，所述计算机程序指令被处理器执行时实现权利要求1至10中任意一项所述的方法。
  23. 一种计算机程序，其特征在于，所述计算机程序包括计算机可读代码，当所述计算机可读代码在电子设备中运行时，所述电子设备中的处理器执行用于实现权利要求1至10中任意一项所述的方法。
PCT/CN2019/130428 2019-01-31 2019-12-31 视频修复方法及装置、电子设备和存储介质 Ceased WO2020156009A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
KR1020207036725A KR102406354B1 (ko) 2019-01-31 2019-12-31 비디오 수복 방법 및 장치, 전자 기기 및 기억 매체
SG11202010145WA SG11202010145WA (en) 2019-01-31 2019-12-31 Video repair method and apparatus, electronic device, and storage medium
JP2021506038A JP7125541B2 (ja) 2019-01-31 2019-12-31 ビデオ修復方法および装置、電子機器、ならびに記憶媒体
US17/072,527 US11544820B2 (en) 2019-01-31 2020-10-16 Video repair method and apparatus, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910099762.4A CN109816611B (zh) 2019-01-31 2019-01-31 视频修复方法及装置、电子设备和存储介质
CN201910099762.4 2019-01-31

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/072,527 Continuation US11544820B2 (en) 2019-01-31 2020-10-16 Video repair method and apparatus, and storage medium

Publications (1)

Publication Number Publication Date
WO2020156009A1 true WO2020156009A1 (zh) 2020-08-06

Family

ID=66606244

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/130428 Ceased WO2020156009A1 (zh) 2019-01-31 2019-12-31 视频修复方法及装置、电子设备和存储介质

Country Status (6)

Country Link
US (1) US11544820B2 (zh)
JP (1) JP7125541B2 (zh)
KR (1) KR102406354B1 (zh)
CN (1) CN109816611B (zh)
SG (1) SG11202010145WA (zh)
WO (1) WO2020156009A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112669204A (zh) * 2021-01-04 2021-04-16 北京金山云网络技术有限公司 图像处理方法、图像处理模型的训练方法和装置
CN114494927A (zh) * 2020-11-12 2022-05-13 阿里巴巴集团控股有限公司 图像处理方法、装置、电子设备及可读存储介质
US11544820B2 (en) 2019-01-31 2023-01-03 Beijing Sensetime Technology Development Co., Ltd. Video repair method and apparatus, and storage medium
CN116777762A (zh) * 2023-04-24 2023-09-19 杭州当虹科技股份有限公司 多模块阶梯式融合学习的视频修复方法

Families Citing this family (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111105382B (zh) * 2019-12-31 2021-11-16 北京大学 视频修复方法
CN111614996B (zh) * 2020-04-07 2022-03-18 上海推乐信息技术服务有限公司 一种视频修复方法和系统
CN113660443B (zh) * 2020-05-12 2024-07-02 武汉Tcl集团工业研究院有限公司 一种视频插帧方法、终端以及存储介质
CN111612721B (zh) * 2020-05-22 2023-09-22 哈尔滨工业大学（深圳） 一种图像修复模型训练、卫星图像修复方法及装置
CN111609868A (zh) * 2020-05-29 2020-09-01 电子科技大学 一种基于改进光流法的视觉惯性里程计方法
CN111738998B (zh) * 2020-06-12 2023-06-23 深圳技术大学 病灶位置动态检测方法及装置、电子设备和存储介质
KR102803885B1 (ko) * 2020-10-08 2025-05-02 에스케이텔레콤 주식회사 영상복원 방법 및 장치
CN112272311B (zh) * 2020-10-21 2022-08-26 腾讯科技（北京）有限公司 花屏修复方法、装置、终端、服务器及介质
CN112465728B (zh) * 2020-12-07 2022-09-23 华中光电技术研究所（中国船舶重工集团公司第七一七研究所） 视频图像处理方法、系统、电子设备及存储介质
CN112862766B (zh) * 2021-01-28 2022-11-04 华北电力大学（保定） 一种基于图像数据扩充技术的绝缘子检测方法及系统
CN112862671B (zh) * 2021-02-09 2024-07-19 清华大学 视频图像编辑、修复的方法、装置及存储介质
CN114972050A (zh) * 2021-02-27 2022-08-30 华为技术有限公司 一种图像修复方法及装置
WO2022197476A1 (en) * 2021-03-19 2022-09-22 Micron Technology, Inc. Modular machine learning models for denoising images and systems and methods for using same
US12148125B2 (en) 2021-03-19 2024-11-19 Micron Technology, Inc. Modular machine learning models for denoising images and systems and methods for using same
US12277683B2 (en) * 2021-03-19 2025-04-15 Micron Technology, Inc. Modular machine learning models for denoising images and systems and methods for using same
US12086703B2 (en) 2021-03-19 2024-09-10 Micron Technology, Inc. Building units for machine learning models for denoising images and systems and methods for using same
US12272030B2 (en) 2021-03-19 2025-04-08 Micron Technology, Inc. Building units for machine learning models for denoising images and systems and methods for using same
US12373675B2 (en) 2021-03-19 2025-07-29 Micron Technology, Inc. Systems and methods for training machine learning models for denoising images
EP4080452A1 (en) * 2021-04-15 2022-10-26 Waymo LLC Unsupervised training of optical flow estimation neural networks
US12354312B2 (en) 2021-11-09 2025-07-08 Samsung Electronics Co., Ltd. Image encoding apparatus and image decoding apparatus both using artificial intelligence, and image encoding method and image decoding method performed by the image encoding apparatus and the image decoding apparatus
WO2023085759A1 (ko) * 2021-11-09 2023-05-19 삼성전자 주식회사 Ai를 이용하는 영상 부호화 장치 및 영상 복호화 장치, 및 이들에 의한 영상의 부호화 및 복호화 방법
KR102518554B1 (ko) * 2022-04-27 2023-04-06 주식회사세오 신경망을 이용한 수위 예측 방법 및 시스템
CN118118676A (zh) * 2022-11-29 2024-05-31 北京三星通信技术研究有限公司 由电子设备执行的方法、电子设备和存储介质
CN119603477B (zh) * 2024-11-29 2025-10-31 北京字跳网络技术有限公司 视频处理方法、设备、存储介质及程序产品

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105025201A (zh) * 2015-07-29 2015-11-04 武汉大学 一种时空连续的视频背景修复方法
US20170270677A1 (en) * 2014-12-08 2017-09-21 Yissum Research Development Company Of The Hebrew University Of Jerusalem Ltd. Method and system for generating adaptive fast forward of egocentric videos
CN108090919A (zh) * 2018-01-02 2018-05-29 华南理工大学 一种基于超像素光流和自适应学习因子改进的核相关滤波跟踪方法
CN109086807A (zh) * 2018-07-16 2018-12-25 哈尔滨工程大学 一种基于空洞卷积堆叠网络的半监督光流学习方法
CN109816611A (zh) * 2019-01-31 2019-05-28 北京市商汤科技开发有限公司 视频修复方法及装置、电子设备和存储介质

Family Cites Families (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR2820255A1 (fr) * 2001-01-26 2002-08-02 France Telecom Procedes de codage et de decodage d'images, dispositifs, systemes, signaux et applications correspondants
KR101418116B1 (ko) * 2007-10-24 2014-07-09 고려대학교 산학협력단 프레임 보간 장치 및 그를 포함한 프레임 속도 상향 변환장치
US9396385B2 (en) * 2010-08-26 2016-07-19 Blast Motion Inc. Integrated sensor and video motion analysis method
US10049279B2 (en) * 2016-03-11 2018-08-14 Qualcomm Incorporated Recurrent networks with motion-based attention for video understanding
JP6609505B2 (ja) * 2016-04-06 2019-11-20 Ｋｄｄｉ株式会社 画像合成装置及びプログラム
US10275892B2 (en) * 2016-06-09 2019-04-30 Google Llc Multi-view scene segmentation and propagation
CN106204567B (zh) * 2016-07-05 2019-01-29 华南理工大学 一种自然背景视频抠图方法
JP6854629B2 (ja) * 2016-11-24 2021-04-07 キヤノン株式会社 画像処理装置、画像処理方法
CN106973293B (zh) * 2017-04-21 2020-10-27 中国科学技术大学 基于视差预测的光场图像编码方法
US10547871B2 (en) * 2017-05-05 2020-01-28 Disney Enterprises, Inc. Edge-aware spatio-temporal filtering and optical flow estimation in real time
CN107292912B (zh) * 2017-05-26 2020-08-18 浙江大学 一种基于多尺度对应结构化学习的光流估计方法
US10636193B1 (en) * 2017-06-29 2020-04-28 Facebook Technologies, Llc Generating graphical representation of a user's face and body using a monitoring system included on a head mounted display
CN108229290B (zh) * 2017-07-26 2021-03-02 北京市商汤科技开发有限公司 视频物体分割方法和装置、电子设备、存储介质
CN107480646B (zh) * 2017-08-22 2020-09-25 北京工业大学 一种基于双目视觉的车载视频异常运动检测方法
CN108875900B (zh) * 2017-11-02 2022-05-24 北京旷视科技有限公司 视频图像处理方法和装置、神经网络训练方法、存储介质
CN108389217A (zh) * 2018-01-31 2018-08-10 华东理工大学 一种基于梯度域混合的视频合成方法
CN108320020A (zh) * 2018-02-07 2018-07-24 深圳市唯特视科技有限公司 一种基于双向光流的端到端无监督学习方法
CN109284720A (zh) * 2018-09-28 2019-01-29 大连民族大学 度量视频连续帧与其卷积特征图间差异在视频行为识别中的应用
KR102646695B1 (ko) * 2019-01-15 2024-03-12 포틀랜드 스테이트 유니버시티 비디오 프레임 보간을 위한 특징 피라미드 워핑
CN109819229B (zh) * 2019-01-22 2021-02-26 北京市商汤科技开发有限公司 图像处理方法及装置、电子设备和存储介质
US11526970B2 (en) * 2019-09-04 2022-12-13 Samsung Electronics Co., Ltd System and method for video processing with enhanced temporal consistency
CN112767441B (zh) * 2021-01-08 2023-06-06 南昌航空大学 一种基于残差场和位移场的图像光流优化方法及系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170270677A1 (en) * 2014-12-08 2017-09-21 Yissum Research Development Company Of The Hebrew University Of Jerusalem Ltd. Method and system for generating adaptive fast forward of egocentric videos
CN105025201A (zh) * 2015-07-29 2015-11-04 武汉大学 一种时空连续的视频背景修复方法
CN108090919A (zh) * 2018-01-02 2018-05-29 华南理工大学 一种基于超像素光流和自适应学习因子改进的核相关滤波跟踪方法
CN109086807A (zh) * 2018-07-16 2018-12-25 哈尔滨工程大学 一种基于空洞卷积堆叠网络的半监督光流学习方法
CN109816611A (zh) * 2019-01-31 2019-05-28 北京市商汤科技开发有限公司 视频修复方法及装置、电子设备和存储介质

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11544820B2 (en) 2019-01-31 2023-01-03 Beijing Sensetime Technology Development Co., Ltd. Video repair method and apparatus, and storage medium
CN114494927A (zh) * 2020-11-12 2022-05-13 阿里巴巴集团控股有限公司 图像处理方法、装置、电子设备及可读存储介质
CN112669204A (zh) * 2021-01-04 2021-04-16 北京金山云网络技术有限公司 图像处理方法、图像处理模型的训练方法和装置
CN112669204B (zh) * 2021-01-04 2024-05-03 北京金山云网络技术有限公司 图像处理方法、图像处理模型的训练方法和装置
CN116777762A (zh) * 2023-04-24 2023-09-19 杭州当虹科技股份有限公司 多模块阶梯式融合学习的视频修复方法

Also Published As

Publication number Publication date
SG11202010145WA (en) 2020-11-27
JP2021519051A (ja) 2021-08-05
KR20210018850A (ko) 2021-02-18
KR102406354B1 (ko) 2022-06-07
US20210035268A1 (en) 2021-02-04
US11544820B2 (en) 2023-01-03
JP7125541B2 (ja) 2022-08-24
CN109816611B (zh) 2021-02-12
CN109816611A (zh) 2019-05-28

Similar Documents

Publication Publication Date Title
CN109816611B (zh) 视频修复方法及装置、电子设备和存储介质
CN113569796B (zh) 关键点检测方法及装置、电子设备和存储介质
TWI724712B (zh) 圖像處理方法、電子設備和儲存介質
TWI706379B (zh) 圖像處理方法及裝置、電子設備和儲存介質
CN109886392B (zh) 数据处理方法和装置、电子设备和存储介质
CN111553864B (zh) 图像修复方法及装置、电子设备和存储介质
CN110798630B (zh) 图像处理方法及装置、电子设备和存储介质
WO2021196401A1 (zh) 图像重建方法及装置、电子设备和存储介质
TWI777112B (zh) 圖像處理方法、電子設備和儲存介質
US11314965B2 (en) Method and apparatus for positioning face feature points
WO2021012406A1 (zh) 批归一化数据的处理方法及装置、电子设备和存储介质
WO2020155711A1 (zh) 图像生成方法及装置、电子设备和存储介质
TW202127369A (zh) 網路訓練方法、圖像生成方法、電子設備及電腦可讀儲存介質
CN112862877A (zh) 用于训练图像处理网络和图像处理的方法和装置
TWI718631B (zh) 人臉圖像的處理方法及裝置、電子設備和儲存介質
WO2021035833A1 (zh) 姿态预测方法、模型训练方法及装置
TW202036476A (zh) 圖像處理方法及裝置、電子設備和儲存介質
WO2021056770A1 (zh) 图像重建方法及装置、电子设备和存储介质
CN111695682A (zh) 运算方法、装置及相关产品
CN107886515A (zh) 图像分割方法及装置
CN107895190A (zh) 神经网络模型的权值量化方法及装置
CN111583142B (zh) 图像降噪方法及装置、电子设备和存储介质
CN106875446B (zh) 相机重定位方法及装置
CN109635926A (zh) 用于神经网络的注意力特征获取方法、装置及存储介质
WO2022134416A1 (zh) 视频数据处理方法及装置、电子设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19913182

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021506038

Country of ref document: JP

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 20207036725

Country of ref document: KR

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 19.11.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19913182

Country of ref document: EP

Kind code of ref document: A1