WO2022198381A1 - 图像处理方法及图像处理装置 - Google Patents

图像处理方法及图像处理装置 Download PDF

Info

Publication number
WO2022198381A1
WO2022198381A1 PCT/CN2021/082072 CN2021082072W WO2022198381A1 WO 2022198381 A1 WO2022198381 A1 WO 2022198381A1 CN 2021082072 W CN2021082072 W CN 2021082072W WO 2022198381 A1 WO2022198381 A1 WO 2022198381A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
processed
video image
video
images
Prior art date
Application number
PCT/CN2021/082072
Other languages
English (en)
French (fr)
Inventor
段然
朱丹
吴艳红
陈冠男
Original Assignee
京东方科技集团股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司 filed Critical 京东方科技集团股份有限公司
Priority to PCT/CN2021/082072 priority Critical patent/WO2022198381A1/zh
Priority to CN202180000544.6A priority patent/CN115398469A/zh
Priority to US17/764,458 priority patent/US20240062344A1/en
Publication of WO2022198381A1 publication Critical patent/WO2022198381A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G06T5/30—Erosion or dilatation, e.g. thinning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/254—Analysis of motion involving subtraction of images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20024—Filtering details
    • G06T2207/20032—Median filtering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20224—Image subtraction

Definitions

  • the embodiments of the present disclosure relate to the field of display technologies, and in particular, to an image processing method and an image processing apparatus.
  • Video images such as film films, may suffer from scratches, dead pixels, noise or color cast due to the length of time or poor preservation. How to repair the above problems in video images to improve the display effect is an urgent problem to be solved. .
  • Embodiments of the present disclosure provide an image processing method and an image processing apparatus, which are used to perform scratch recovery, dead pixel recovery, noise removal, and/or color cast correction on a video image, so as to improve display effects.
  • an embodiment of the present disclosure provides an image processing method, including:
  • Scratch repairing step performing scratch removal processing on the video image to be processed to obtain a first image; performing a difference operation on the video image to be processed and the first image to obtain a difference map; A scratch image of scratches; a scratch repair image is obtained according to the to-be-processed video image and the scratch image;
  • the step of repairing bad pixels obtaining consecutive N 1 frames of video images, where the N 1 frames of video images include to-be-processed video images, at least one frame of video images before and at least one frame of video images after the to-be-processed video images; Perform filtering processing on the video image to be processed to obtain a dead pixel repair image according to the N 1 frames of video images; De-artifact processing is performed on the dead pixel repaired image to obtain an artifact-repaired image;
  • Step of denoising use a denoising network to denoise the video image to be processed, and the denoising network is obtained by the following training method: obtaining a target non-motion mask according to consecutive N 2 frames of video images, the N 2 frames of video images Including the video images to be denoised; according to the N 2 frames of video images and the target non-motion mask, the denoising network to be trained is trained to obtain the denoising network;
  • the color cast correction step determining the target color cast value of each RGB channel of the video image to be processed; performing color balance adjustment on the video image to be processed according to the target color cast value to obtain a first corrected image; according to the reference image, Perform color migration on the first corrected image to obtain a second corrected image.
  • an image processing apparatus including:
  • a processing module includes at least one of the following modules:
  • Scratch repair sub-module perform scratch removal processing on the video image to be processed to obtain a first image; perform a difference operation on the video image to be processed and the first image to obtain a difference map; process the difference image to obtain only Retaining the scratched image of scratches; obtaining a scratch repaired image according to the to-be-processed video image and the scratched image;
  • Dead pixel repair sub-module obtains consecutive N 1 frames of video images, the N 1 frames of video images include to-be-processed video images, at least one frame of video images before and at least one frame of video images after the to-be-processed video images ; According to the N1 frames of video images, filter the to-be-processed video images to obtain a dead pixel repair image; Performing de-artifact processing on the dead pixel repaired image to obtain an artifact-repaired image;
  • Denoising sub-module use a denoising network to denoise the video image to be processed, and the denoising network is obtained by the following training method: obtaining a target non-motion mask according to consecutive N 2 frames of video images, the N 2 frames of video The image includes a video image to be denoised; according to the N 2 frames of video images and the target non-motion mask, the denoising network to be trained is trained to obtain the denoising network;
  • Color cast correction sub-module determine the target color cast value of each RGB channel of the video image to be processed; according to the target color cast value, perform color balance adjustment on the to-be-processed video image to obtain a first corrected image; according to the reference image , performing color migration on the first corrected image to obtain a second corrected image.
  • an embodiment of the present disclosure provides an electronic device, including a processor, a memory, and a program or instruction stored on the memory and executable on the processor, where the program or instruction is processed by the processor The steps of implementing the above-mentioned image processing method when the processor is executed.
  • an embodiment of the present disclosure provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, the steps of the above image processing method are implemented.
  • scratch recovery, dead pixel recovery, noise removal, and/or color cast correction can be performed on the video image, so as to improve the display effect of the video image.
  • FIG. 1 is a schematic flowchart of a scratch repair step according to an embodiment of the disclosure
  • FIG. 2 is a schematic schematic diagram of a specific flow of a scratch repair step according to an embodiment of the disclosure
  • FIG. 3 is a schematic diagram of a to-be-processed video image and a first image according to an embodiment of the present disclosure
  • FIG. 4 is a schematic diagram of a first difference image and a second difference image according to an embodiment of the present disclosure
  • FIG. 5 is a schematic diagram of a first scratch image and a second scratch image according to an embodiment of the disclosure
  • FIG. 6 is a schematic diagram of a comparison between a video image to be processed and a scratch repaired image according to an embodiment of the present disclosure
  • FIG. 7 is a schematic flowchart of a dead pixel repair step according to an embodiment of the present disclosure.
  • FIG. 8 is a schematic diagram of a comparison between a to-be-processed video image and a dead pixel repaired image according to an embodiment of the present disclosure
  • FIG. 9 is a schematic structural diagram of a multi-scale cascaded network according to an embodiment of the present disclosure.
  • FIG. 10 is a schematic diagram of an input image of a multi-scale cascaded network according to an embodiment of the disclosure.
  • FIG. 11 is a schematic diagram of a comparison between a video image to be processed and an output image of a multi-scale cascade network according to an embodiment of the disclosure
  • FIG. 12 is a schematic structural diagram of a sub-network in an embodiment of the present disclosure.
  • FIG. 13 is a schematic diagram of a comparison between an output image and a post-processed image of a multi-scale cascade network according to an embodiment of the present disclosure
  • FIG. 14 is a schematic flowchart of a denoising step according to an embodiment of the present disclosure.
  • 15 is a schematic flowchart of obtaining a motion mask by using an optical flow network according to an embodiment of the disclosure
  • 16 is a schematic diagram of a specific flow of obtaining a motion mask by using an optical flow network according to an embodiment of the present disclosure
  • FIG. 17 is a schematic diagram of a training method of a denoising network according to an embodiment of the disclosure.
  • FIG. 18 is a schematic diagram of a method for implementing a denoising network according to an embodiment of the disclosure.
  • 19 is a schematic diagram of a video image to be processed according to an embodiment of the disclosure.
  • FIG. 20 is an enlarged view of a partial area in the video image to be processed in FIG. 19;
  • Figure 21 is the target non-motion mask M corresponding to the video image to be processed in Figure 19;
  • FIG. 22 is an image obtained by denoising the video image to be processed by using the denoising network according to an embodiment of the present disclosure
  • FIG. 23 is a schematic flowchart of a color cast correction step according to an embodiment of the present disclosure.
  • FIG. 24 is a schematic structural diagram of an image processing apparatus according to an embodiment of the disclosure.
  • Embodiments of the present disclosure provide an image processing method, including:
  • Scratch repairing step performing scratch removal processing on the video image to be processed to obtain a first image; performing a difference operation on the video image to be processed and the first image to obtain a difference image; A scratch image of scratches; a scratch repair image is obtained according to the to-be-processed video image and the scratch image;
  • the step of repairing bad pixels obtaining consecutive N 1 frames of video images, where the N 1 frames of video images include to-be-processed video images, at least one frame of video images before and at least one frame of video images after the to-be-processed video images; Perform filtering processing on the video image to be processed to obtain a dead pixel repair image according to the N 1 frames of video images; De-artifact processing is performed on the dead pixel repaired image to obtain an artifact-repaired image;
  • Step of denoising use a denoising network to denoise the video image to be processed; wherein, the denoising network is obtained by the following training method: obtaining the target non-motion mask M according to consecutive N 2 frames of video images, the N 2 The frame video image includes the video image to be denoised; according to the N 2 frame video images and the target non-motion mask M, the denoising network to be trained is trained to obtain the denoising network;
  • the color cast correction step determining the target color cast value of each RGB channel of the video image to be processed; performing color balance adjustment on the video image to be processed according to the target color cast value to obtain a first corrected image; according to the reference image, Perform color migration on the first corrected image to obtain a second corrected image.
  • At least one of the above four steps can be performed on the video image. If multiple steps need to be performed, the execution order of the multiple steps is not limited. For example, scratch repair and bad pixel repair steps need to be performed. First perform scratch repair on the video image, and then perform the dead pixel repair, or, first perform the dead pixel repair, and then perform the scratch repair.
  • At least one step of scratch repair, dead pixel repair, denoising, or color cast correction is performed on the video image, thereby improving the display effect of the video image.
  • FIG. 1 is a schematic flowchart of a scratch repair step in an embodiment of the disclosure, including:
  • Step 11 perform scratch removal processing on the video image to be processed to obtain a first image
  • the video image to be processed may be subjected to filtering processing to remove scratches, for example, the filtering processing is median filtering processing or the like.
  • Step 12 performing a difference operation on the to-be-processed video image and the first image to obtain a difference image
  • Step 13 processing the difference image to obtain a scratch image that only retains scratches
  • Step 14 Obtain a scratch repaired image according to the to-be-processed video image and the scratch image.
  • a single frame of a video image to be processed is subjected to scratch removal processing to obtain an image with scratches removed, and a difference operation is performed between the video image to be processed and the scratch-removed image to obtain scratches and image details. Then the difference image is processed again, the image details are filtered out and the scratches are retained to obtain the scratched image, and the scratched repaired image with the scratches removed is obtained according to the video image to be processed and the scratched image. , but also without affecting the image clarity.
  • performing scratch removal processing on the video image to be processed includes: performing a scratch removal process on the video image to be processed according to at least one of a filter type and a scratch type in the video image to be processed. Median filter processing to obtain a scratch-free image.
  • performing median filtering processing on the video image to be processed according to at least one of the filter type and the scratch type in the video image to be processed includes: according to the scratches in the video image to be processed.
  • Type select the corresponding filter type, and perform median filtering processing on the to-be-processed video image, wherein:
  • a median filter in the vertical direction is used to perform median filtering on the video image to be processed.
  • the median filter to be used may be determined according to the direction of the scratch in the video image to be processed.
  • the median filter may not be changed, and the video image to be processed may be rotated so that the scratches in the video image to be processed match the median filter.
  • performing median filtering processing on the video image to be processed according to at least one of the filter type and the scratch type in the video image to be processed includes: according to the filter type and the scratch type in the video image to be processed Scratch type, different preprocessing of the to-be-processed video image, median filter processing is performed on the to-be-processed video image after preprocessing, wherein:
  • the video image to be processed is rotated so that the scratches are converted into vertical scratches;
  • Vertical scratches include horizontal scratches and diagonal scratches.
  • the scratches in the video image to be processed are vertical scratches, the image to be processed does not need to be rotated.
  • Non-horizontal scratches include vertical scratches and diagonal scratches. Of course, if the scratches in the video image to be processed are horizontal scratches, the image to be processed does not need to be rotated.
  • a median filter in the horizontal direction and a median filter in the vertical direction can be used to perform median filtering on the video image to be processed. For example, the median filtering processing in the horizontal direction is performed first, and then the median filtering processing in the vertical direction is performed, or the median filtering processing in the vertical direction is performed first, and then the median filtering processing in the horizontal direction is performed.
  • performing the scratch removal processing on the video image to be processed includes: performing median filtering on the video image to be processed by using a median filter with a size of 1 ⁇ k and/or k ⁇ 1; wherein, the median filter of 1 ⁇ k The value filter is a horizontal median filter, and the k ⁇ 1 median filter is a vertical median filter.
  • a 1 ⁇ k median filter is used to filter the video image I to be processed to obtain a first image I median .
  • M 1 ⁇ k (x) represents the filtering operation performed on x using a median filter of size 1 ⁇ k.
  • the method further includes: sequentially increasing k of the median filter from a preset value, and performing a middle value on the video image to be processed. value filtering to obtain a second image; the final value of k is determined according to the filtering effect of the second image.
  • the value of k can also be directly determined according to the thickness of the scratch. For example, for an image with a resolution of 2560 ⁇ 1440, k can be set to a value below 11.
  • Fig. 3 (a) is a partial enlarged view of the video image to be processed and the video image to be processed, as can be seen from the figure, the video image to be processed has vertical scratches.
  • a horizontal median filter can be used to perform scratch removal processing on the video image to be processed to obtain a first image.
  • Fig. 3(b) is the first image and a partial enlarged view of the first image. It can be seen from the figure that the vertical scratches have been removed in the first image, but the first image is different from the first image. Compared with the image to be processed, it is more blurred.
  • performing a difference operation on the to-be-processed video image and the first image to obtain a difference image includes: performing a difference operation on the to-be-processed video image and the first image to obtain a first difference image and /or a second difference image, wherein the first difference image is obtained by subtracting the first image from the video image to be processed, and the second difference image is obtained by subtracting the video to be processed from the first image image is obtained.
  • two images are subtracted, indicating that the pixels at the corresponding positions of the two images are subtracted.
  • the first difference image is a white texture map containing image details and scratches
  • the second difference image is a black texture map containing image details and scratches.
  • the first difference image may also be referred to as a positive residual image
  • the second difference image may also be referred to as a negative residual image
  • the video image to be processed and the first image are subtracted from each other, wherein the video image to be processed is multiplied by 1 ( ⁇ 1) and the first image is multiplied by -1 ( ⁇ -1) and then added, that is, The video image to be processed is subtracted from the first image to obtain the first difference image.
  • the video image to be processed is multiplied by -1 ( ⁇ -1) and the first image is multiplied by 1 ( ⁇ 1) and then added, that is, the first image is subtracted
  • a second difference image is obtained from the video image to be processed.
  • FIG. 4 (a) in FIG. 4 is a partial enlarged view of the first difference image and the first difference image
  • (b) in FIG. 4 is a partial enlarged view of the second difference image and the second difference image.
  • the first difference image is a white texture map including image details and scratches
  • the second difference image is a black texture map including image details and scratches.
  • the first difference image and the second difference image are calculated at the same time.
  • processing the difference image to obtain a scratch image that only retains scratches includes: processing the first difference image to obtain a first scratch image that retains only scratches; and/or, to The second difference image is processed to obtain a second scratch image that only retains scratches;
  • a scratched image is an image in which image details are filtered out and only scratches remain.
  • the first scratch image may also be referred to as a positive scratch image
  • the second scratch image may also be referred to as a negative scratch image
  • processing the first difference image to obtain a first scratch image that only retains scratches includes:
  • the scratches in the to-be-processed video image are vertical scratches, subtract the first horizontally filtered image from the first vertical filtered image to obtain the first scratched image;
  • the scratches in the to-be-processed video image are horizontal scratches, subtract the first vertical filtered image from the first horizontally filtered image to obtain the first scratched image;
  • the scratches in the video image to be processed are vertical scratches
  • perform vertical median filtering and horizontal median filtering on Err white the first difference image
  • subtract the first horizontal filtered image M 1 from the first vertical filtered image M n ⁇ 1 (Err white ) ⁇ n (Err white ) respectively obtain the filtered first scratch image L white
  • the first scratch image at this time is a positive number:
  • M n ⁇ 1 (Err white ) represents performing median filtering on the first difference image in the vertical direction
  • M 1 ⁇ n (Err white ) represents performing median filtering on the first difference image in the horizontal direction.
  • Processing the second difference image to obtain a second scratch image that only retains scratches includes:
  • the scratch in the described video image to be processed is a vertical scratch
  • the scratches in the video image to be processed are horizontal scratches, subtract the second vertical filtered image from the second horizontally filtered image to obtain the second scratched image.
  • the Err black (second difference image) is subjected to vertical median filtering and horizontal median filtering to obtain a second vertical filtered image.
  • L black M n ⁇ 1 (Err black )-M 1 ⁇ n (Err black )
  • M n ⁇ 1 (Err black ) represents performing median filtering on the second difference image in the vertical direction
  • M 1 ⁇ n (Err black ) represents performing median filtering on the second difference image in the horizontal direction
  • the median filter in the vertical direction and the median filter in the horizontal direction can be combined.
  • the value of n is set relatively large, for example, it can be set as half of the average length of the scratches.
  • the maximum length of the scratches is 180, and the value of n can be set between 80-100.
  • the first difference image and the second difference image are respectively subjected to median filtering in the vertical direction and median filtering in the horizontal direction to obtain the image after vertical filtering and the image after horizontal filtering.
  • the scratches in the processed video image are vertical scratches, it is necessary to subtract the horizontally filtered image from the vertical filtered image to obtain a first scratch image and a second scratch image.
  • FIG. 5 (a) in FIG. 5 is a partial enlarged view of the first scratch image and the first scratch image, and (b) in FIG. 5 is the second scratch image and a part of the second scratch image Enlarge the image.
  • the first scratch image is an image including white scratches
  • the second scratch image is an image including black scratches.
  • Obtaining the scratch repaired image according to the to-be-processed video image and the scratch image includes: obtaining the scratch by performing an operation on the to-be-processed video image, the first scratch image and/or the second scratch image Fix images.
  • the scratch repair image can be obtained by calculating the following formula:
  • I deline is the scratch repaired image
  • I is the video image to be processed
  • L white is the first scratch image
  • L black is the second scratch image.
  • the second scratch image L black is a positive value, it needs to be multiplied by -1 to restore it to a negative value.
  • only the first scratch image may be used, or only the second scratch image may be used.
  • FIG. 7 is a schematic flowchart of a dead pixel repair step in an embodiment of the disclosure, including:
  • Step 71 Acquire consecutive N 1 frames of video images, where the N 1 frames of video images include to-be-processed video images, at least one frame of video images before and at least one frame of video images after the to-be-processed video images;
  • N 1 is a positive integer greater than or equal to 3, which can be set as required, for example, it can be 3.
  • Step 72 According to at least one frame of video image before and at least one frame of video image after the to-be-processed video image, filter the to-be-processed video image to obtain a dead pixel repair image;
  • Step 73 Perform de-artifact processing on the dead pixel repaired image according to at least one frame of video image before and at least one frame of video image after the video image to be processed to obtain an artifact repaired image.
  • dead pixel repair can include two processes: one is to remove dead pixels, and the other is to remove artifacts, which will be introduced separately below.
  • the dead pixel repair method of the embodiment of the present disclosure can be applied to the dead pixel repair of the motion picture film, of course, the dead pixel repair of other types of video images is not excluded.
  • Dead pixel is a common damage to film film. It is a white or black block-like spot formed due to the lack of gel on the surface of the film or contamination with stains during the storage process of the film.
  • the dead pixel in the film generally has the following three Features:
  • the grayscale of the dead pixel is discontinuous in the time and space domains. Since this damage is randomly distributed within a frame, it is unlikely that the dead pixels will repeat at the same position in two adjacent frames, so it appears as a kind of impulsive damage in the time domain. In a frame of image, the grayscale of the dead pixel area is generally far away from the surrounding background grayscale, so that it can be perceived by the human eye;
  • the present disclosure mainly focuses on the second characteristic of the bad pixel to achieve its repair. Since the bad pixel has discontinuity in the time domain, and the pixel values of adjacent frames at the same position are often similar, the embodiments of the present disclosure The content of the adjacent frame images before and after is used to repair the bad pixels in the current frame image.
  • median filtering is performed on the video image to be processed to obtain a dead pixel repair image.
  • N 1 is equal to 3
  • the median value is calculated pixel by pixel for the current video image It to be processed and the adjacent frame images It -1 and It +1 , because the same position of the adjacent frame images in the same scene Generally, the difference between the pixel values is not too large. Therefore, in the process of calculating the median value, the bad pixel area with a large difference with the surrounding gray value will be replaced by the pixels in the previous frame or the next frame image, thereby eliminating the intermediate frame. Bad pixels within the image.
  • FIG 8 (a) in Figure 8 is a video image to be processed with dead pixels, and (b) is a repaired image with dead pixels. It can be seen from Figure 8 that the content of the front and rear frame images can be used through median filtering. Fill the dead pixels in the intermediate frame image, but also introduce the error information of the previous and previous frames into the intermediate frame, resulting in motion artifacts.
  • the small image in the upper right corner is an enlarged image of the dead pixel area
  • the small picture in the upper right corner is an enlarged picture of the bad pixel repair area
  • the small picture in the lower right corner is an enlarged picture of the motion artifact part. Therefore, in the embodiment of the present disclosure, it is also necessary to perform de-artifact processing on the dead pixel repair image to eliminate the artifacts generated by filtering.
  • the artifact-repaired image obtained by performing de-artifact processing on the dead pixel repair image according to at least one frame of video image before and at least one frame of video image after the video image to be processed includes the following steps: :
  • N 3 -1 downsampling on the dead pixel repair image, at least one frame of video image before the to-be-processed video image, and at least one frame of video image after, respectively, to obtain N 3 -1 resolution downsampling image
  • the N 3 resolution images include: the dead pixel repair image, the to-be-processed image
  • the multi-scale cascade network includes N 3 cascaded sub-networks, so The images processed by the N cascaded sub-networks are respectively generated based on the images of the N 3 resolutions;
  • N 3 is a positive integer greater than or equal to 2.
  • the images of N 3 resolutions are input into a multi-scale cascade network for de-artifact processing to obtain an artifact-repaired image, including the artifact-repaired image:
  • the repaired image for dead pixels at least one frame of video image before the video image to be processed, and at least one frame of video image after A times down-sampling to obtain N 1 frames of first down-sampled images, respectively splicing the N 1 frames of first down-sampled images with itself to obtain a first spliced image, and inputting the first spliced image into the A sub-network to get the first output image;
  • the intermediate sub-network between the first sub-network and the last sub-network up-sampling the output image of the previous sub-network to obtain the first up-sampled image;
  • the previous at least one frame of video image and the next at least one frame of video image are down-sampled by B times, respectively, to obtain N 1 frames of second down-sampled images, and the scale of the second down-sampled image and the first up-sampled image is
  • splicing two sets of images to obtain a second spliced image and inputting the second spliced image into the intermediate sub-network to obtain a second output image, wherein one of the two sets of images is N1 frames of second down-sampled images, and the other group includes: other down-sampled images in the N1 frames of second down-sampled images except the down-sampled images corresponding to the dead pixel repair image, and the first upsampled images;
  • the third image after splicing is input into the last sub-network to obtain the artifact repaired image, wherein one of the two sets of images is the N 1 frames of video images , and the other group includes: other images in the N 1 frames of video images except the dead pixel repair image, and the second up-sampled image.
  • the foregoing sub-network may be an encoder-decoder (encoding and decoding) resblock network structure proposed in SRN.
  • encoder-decoder encode and decoding
  • other networks may also be used, which are not limited by the embodiments of the present disclosure.
  • the N 3 cascaded sub-networks have the same structure and different parameters.
  • N 3 is equal to 3
  • A is equal to 4
  • B is equal to 2.
  • the multi-scale cascade network is composed of three sub-networks with the same structure cascaded, and the three sub-networks process inputs of different scales (ie resolutions) respectively.
  • the input of the multi-scale cascade network is three consecutive frames of images I t-1 , I′ t and It t+1 , please refer to Figure 10, where (b) is I′ t , which is the image with artifacts after the above processing
  • the dead pixel repair image of (a) is It -1 , which is the previous frame image of the video image It is to be processed
  • ( c ) is It +1 , which is the next frame image of the video image It is to be processed
  • Multi-scale cascaded network outputs the artifact-removed image corresponding to I't
  • the above-mentioned splicing refers to splicing in the fourth dimension.
  • Each image is a three-dimensional array, and the array structure is H ⁇ W ⁇ C, namely height, width, channel, and the fourth dimension is the channel dimension.
  • the method of bicubic interpolation is used to downsample It -1 , I′ t and It +1 by 4 times, respectively.
  • other methods can also be used for downsampling.
  • the input image sizes of the three sub-networks are 1/4, 1/2, and 1/4 of the original input image size, respectively. 1.
  • the image is to be input into the first sub-network, so 4 times downsampling is done.
  • a bicubic interpolation method is used to convert the output of the network 1 Upsampling by a factor of 2.
  • other methods can also be used for upsampling.
  • a method of bicubic interpolation is used to downsample the images It -1 , I′ t and It +1 by a factor of 2, respectively.
  • other methods can also be used for downsampling.
  • a bicubic interpolation method is used to convert the output of the network 2 Upsampling by a factor of 2.
  • other methods can also be used for upsampling.
  • FIG. 11 is a partial area of the original video image to be processed, (b) is a partial area of the dead pixel repaired image after filtering, and the dead pixel repaired image contains Motion artifact, (c) is a partial area of the inpainted image output by the multi-scale cascaded network. It can be seen from Figure 11 that the motion artifact is eliminated.
  • the network 1, the network 2, and the network 3 may be the encoder-decoder (encoding and decoding) resblock network structure proposed in the SRN.
  • the three sub-networks in the embodiment of the present disclosure The parameters are not shared.
  • each of the sub-networks includes multiple 3D convolution layers, multiple 3 deconvolution layers, and multiple 3D average pooling layers.
  • FIG. 12 is a schematic structural diagram of a sub-network in an embodiment of the present disclosure.
  • the sub-network is an encoder-decoder resblock network, wherein Conv3d (n32f5s1) represents that the number of filters (n) is 32, the filter DeConv3d(n32f5s2) means the number of filters (n) is 32, and the filter size (f) is (1 ⁇ 5 ⁇ 5) 3D deconvolution layer with filter stride (s) of 2; AvgPool3D(f2s1) for kernel size (f) of (2 ⁇ 1 ⁇ 1) and stride (s) of 1 3D average pooling layer; floating above the arrow (B, 3, H, W, 32) The size of the feature map output by the current layer.
  • Conv3d n32f5s1
  • the filter DeConv3d(n32f5s2) means the number of filters (n) is 32
  • the filter size (f) is (1 ⁇ 5 ⁇ 5) 3D deconvolution layer with filter str
  • (B, 3, H, W, 32) refers to the size of the intermediate results output by each layer of the network.
  • the output of each layer of the network is a 5-dimensional array
  • (B, 3, H, W, 32) refers to is the structure of the array, i.e. B ⁇ 3 ⁇ H ⁇ W ⁇ 32, B is Batch_size.
  • the sub-network contains a total of 16 3D convolutional layers and 2 3D deconvolutional layers.
  • the information between adjacent frames is fused through the 3D average pooling layer.
  • the output feature maps of the product layer and the second deconvolution layer are respectively fused through the 3D average pooling layer and then added pixel by pixel, and their sum is used as the input of the 12th convolution layer.
  • the network passes a The convolutional layer with a filter number of 1 outputs the resulting image with the final artifacts removed.
  • the multi-scale cascade network is obtained by adopting the following training method:
  • Step 1 Acquire continuous N1 frames of training images, where the N1 frames of training images include a to-be-processed training image, at least one frame of training image before and at least one frame of training image after the to-be-processed training image;
  • Step 2 filtering the to-be-processed training images according to the N1 frames of training images to obtain a first training image
  • Step 3 According to the first training image, at least one frame of training image before and at least one frame of training image after the training image to be processed, train the multi-scale cascade network to be trained, and obtain the multi-scale after training. Cascaded network.
  • the total loss used includes at least one of the following: image content loss, color (Color) loss, edge loss, and perceptual loss.
  • the image content loss is mainly used to improve the fidelity of the output image.
  • the image content loss may be calculated by using an L1 loss function or a mean square error loss function.
  • the L1 loss is calculated using the following formula:
  • l content is the L1 loss
  • yi is the first training image
  • n is the number of images in a Batch.
  • the color loss function corrects the image color by performing Gaussian blurring on the texture and content of the de-artifact training image and the target image, and only saves the color information of the image.
  • the color loss is calculated by using the following formula:
  • lcolor is the color loss
  • yi is the first training image
  • n is the number of images in a Batch
  • Blur(x) is a Gaussian blur function
  • the edge loss function mainly improves the accuracy of the contour information of the de-artifact training image by calculating the difference between the edge information of the de-artifact training image and the target image.
  • the HED Hollyically-Nested Network
  • the edge loss is calculated by using the following formula:
  • H j (x) represents the image edge map extracted by the jth layer of the HED network.
  • the perceptual loss function is calculated by using the high-level features extracted by the VGG network to measure the difference between the output image and the target image at the semantic level.
  • the perceptual loss is calculated by the following formula:
  • l feature is the perceptual loss
  • y i is the first training image
  • n is the number of images in a Batch
  • the total loss is equal to a weighted sum of image content loss, color loss, edge loss, and perceptual loss.
  • the total loss is calculated using the following formula:
  • the weight of each loss may not be limited to this.
  • the training data provided by the video time-domain superdivision track of the 2020-AIM competition can be used for training.
  • the training set contains 240 sets of frame sequences in total, and each set of frame sequences contains 181 clear images of 1280 ⁇ 720 size.
  • the reasons for using this training dataset are as follows:
  • the training data set is used for the training of the video time-domain super-division track, and the objects in the image in the same scene have appropriate motion between adjacent frames, which meets the requirements of the present disclosure for simulating the training data to generate artifacts;
  • the images in the training dataset are relatively clean, free of noise, and have a larger resolution, which is beneficial for the network to generate clearer images.
  • the main purpose of network training is to remove artifacts generated by filtering. Therefore, when generating simulation data, only the The training dataset is subjected to the same filtering operation without simulating the generation of dead pixels.
  • the network model of the embodiment of the present disclosure can be trained under the ubuntu16.04 system, compiled using the python language, and based on the tensorflow deep learning framework and open-source image and video processing tools such as opencv and ffmpeg.
  • the multi-scale cascaded network to be trained is trained according to the first training image, at least one frame of training image before the to-be-processed training image, and at least one frame of training image after the training image include:
  • An image block is randomly cropped on the first training image, and an image block is cropped at the same position of at least one frame of training image before and at least one frame of training image after the to-be-processed training image, to obtain N 1 frame image block;
  • the N 1 frame image blocks are input into the multi-scale cascaded network to be trained for training.
  • the Adam optimization algorithm can be used to optimize the network parameters, and the learning rate of the Adam algorithm is set to 10-4.
  • the network training process three consecutive frames of training images are sequentially selected from the training data set as a pre-mediation filter for median filtering. processing, and then randomly crop a 512 ⁇ 512 image block in the intermediate frame, and crop the corresponding image block at the same position in the previous and previous frames as the input for one iteration of the network, when all images in the training dataset are read once, Completed for one epoch iteration. After every 10 epochs (an epoch is the process of training all training samples once), Adam's learning rate is reduced to 0.8 times the original.
  • down-sampling is performed on the cropped image block, which is a method for augmenting the data set. That is, multiple image patches can be randomly cropped for the same image multiple times for training the network, thereby increasing the number of images used to train the network. Random cropping enables different positions of the same image to be taken.
  • cropping into image blocks can also reduce the resolution of the image, reduce the amount of data processed by the network, and improve the processing speed.
  • the image after dead pixel removal and artifact removal processing not only repairs the dead pixels in the image, but also removes the artifacts caused by object motion, but the overall clarity of the image output by the network is There are still some differences with the original to-be-processed video image. Therefore, by filtering the to-be-processed video image, the dead pixel repaired image and the image repaired by the multi-scale cascade network, the details in the original to-be-processed video image are added to the image. in the repaired image to enhance the clarity of the repaired image.
  • the method further includes: filtering the artifact repaired image according to the to-be-processed video image and the dead pixel repaired image to obtain an output image Artifact repair image.
  • median filter processing is performed on the artifact repair image to obtain an output image.
  • Figure 13 is the output image of the multi-scale cascade network
  • (b) is the image after post-processing, it can be seen from Figure 13 that the definition of the post-processing image is obviously high The sharpness of the output image of the multi-scale cascade network.
  • FIG. 14 is a schematic flowchart of a denoising step according to an embodiment of the present disclosure.
  • the denoising step includes:
  • Step 141 Use a denoising network to denoise the video image to be processed; wherein, the denoising network is obtained by using the following training method: obtaining a target non-motion mask according to consecutive N 2 frames of video images, the N 2 frames of video The image includes a video image to be denoised; according to the N 2 frames of video images and the target non-motion mask, the denoising network to be trained is trained to obtain the denoising network.
  • the blind denoising technique when training the denoising network, the blind denoising technique is used, that is, no paired training data sets are required, only the video frame sequence to be denoised needs to be input, and a non-motion mask is used to Temporal denoising is performed on non-motion data as a reference image, which is suitable for denoising network training without a clear reference image.
  • a non-motion mask is used to Temporal denoising is performed on non-motion data as a reference image, which is suitable for denoising network training without a clear reference image.
  • it is suitable for a variety of video noise removal, and does not need to consider the type of noise. It only needs to use some video frames to learn the denoising network.
  • the denoising network to be trained is trained, and obtaining the denoising network includes:
  • Step 151 Obtain a reference image according to the N 2 frames of video images and the target non-motion mask
  • the reference image is equivalent to the true value of the video image to be denoised, that is, the image without noise.
  • Step 152 Input the video image to be denoised into the denoising network to be trained to obtain a first denoising map
  • Step 153 Obtain a second denoising map according to the first denoising map and the target non-motion mask
  • Step 154 Determine the loss function of the denoising network to be trained according to the reference map and the second denoising map, adjust the parameters of the denoising network to be trained according to the loss function, and obtain the denoising network .
  • an optical flow method can be used to obtain a non-motion mask of a video image, which will be described in detail below.
  • Optical flow is the instantaneous speed of pixel motion of a spatially moving object on the observation imaging plane.
  • the optical flow method uses the changes of pixels in the image sequence in the time domain and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, so as to calculate the motion of objects between adjacent frames. a method of information.
  • the motion mask refers to the motion information in the image.
  • the non-motion mask refers to the information in the image other than the motion mask, that is, the non-motion information.
  • FIG. 15 is a schematic flowchart of obtaining a motion mask by using an optical flow network according to an embodiment of the present disclosure. Input two frames of video images F and Fr into the optical flow network to obtain an optical flow diagram, and then pass the optical flow diagram through the optical flow diagram. Get the motion mask.
  • optical flow networks which can be any open-source optical flow networks, such as flownet and flownet2, or traditional optical flow algorithms (not deep learning), such as TV- L1flow, you only need to use the optical flow algorithm to obtain the optical flow graph.
  • FIG. 16 is a schematic diagram of a specific flow of obtaining a motion mask by using an optical flow network according to an embodiment of the disclosure. Assuming that the size of two frames of images input to the optical flow network is (720,576), the optical flow network outputs two frames of optical flow. At this time, the size of the optical flow graph is (720,576), and the two optical flow graphs are the first optical flow graph representing the up and down motion information in the consecutive two frames of images and the left and right motion in the consecutive two frames of images respectively.
  • the second optical flow diagram of the information subtract the last X-X1 lines and the first X-X1 lines of the first optical flow diagram (the example in the figure is the last three lines minus the first three lines) to obtain the first difference. value map; subtract the back Y-Y1 column and the front Y-Y1 column of the second optical flow map (the example in the figure is the next three columns minus the first three columns) to obtain the second difference map, and the first
  • the last X1 row of a difference map is filled with 0, and the last Y1 column of the second difference map is filled with 0, so as to obtain a map of the same size as the optical flow map.
  • the two difference plots are then added together.
  • symbol indicates the absolute value
  • >T indicates that after the absolute value operation, the corresponding pixel position value greater than T is assigned as 1, and the rest of the values less than or equal to T are assigned as 0, and T is the preset threshold.
  • the binarized image is obtained.
  • a dilation operation may also be performed on the binarized image. The specific method of the dilation operation is to find the value of the pixel in the binarized image as 1, and according to the dilation kernel, set the pixel position in the binarized image corresponding to the position of 1 in the kernel to 1.
  • the pixel position is up, down, left and right.
  • the pixel positions are set to 1.
  • other expansion cores can also be used, as long as they can play a role in expansion.
  • the purpose of the expansion operation here is to expand the range of the motion mask, and to mark all the motion positions as much as possible to reduce the error.
  • the motion mask Mask_move in the video image can be obtained, and the mask value is binarized, that is, the value is either 0 or 1, the place with a value of 1 represents the place where there is motion, and the place with a value of 0 represents the relative There is no place to exercise.
  • the following describes the training process of the denoising network in conjunction with the above-mentioned method for determining the target non-motion mask.
  • the method for determining the target non-motion mask includes:
  • Step 181 The first video image of each frame of the N 2 frames of video images and the video image to be denoised form a sample pair and input it to the optical flow network to obtain the first optical flow graph and A second optical flow diagram representing left and right motion information, the first video image is other video images in the N 2 frames of video images except the video image to be denoised, the first optical flow diagram and the The resolution of the second optical flow map is X*Y;
  • Step 182 Calculate the motion masks of each frame of the first video image and the video image to be denoised according to the first optical flow graph and the second optical flow graph, and obtain N 2 ⁇ 1 motion masks ;
  • F1 and F3 will be composed of sample pairs and input to the optical flow network to obtain the motion mask Mask_move1, and the F2 and F3 form a sample pair and input it to the optical flow network to obtain a motion mask Mask_move2, and F4 and F3 form a sample pair and input it to the optical flow network to obtain a motion mask Mask_move4, and F5 and F3 form a sample pair and input it to the optical flow network, Get the motion mask Mask_mov5.
  • Step 183 Obtain a target non-motion mask according to the N 2 ⁇ 1 motion masks.
  • obtaining the target non-motion mask according to the N 2 ⁇ 1 motion masks includes:
  • the N 2 -1 target non-motion masks are multiplied to obtain the target non-motion mask.
  • calculating the motion mask of each frame of the first video image and the video image to be denoised includes:
  • Step 191 Subtract the last X-X1 line and the first X-X1 line of the first optical flow map to obtain the first difference map, and add 0 to the last X1 line of the first difference map to obtain the processed The first difference map of ;
  • X1 is a positive integer less than X.
  • X1 can be set to 1, that is, subtract the last X-1 row and the first X-1 row to obtain the first difference map.
  • Step 192 Subtract the second Y-Y1 column and the first Y-Y1 column of the second optical flow map to obtain a second difference map, and fill the last Y1 column of the second difference map with 0 to obtain the processed The second difference map of ;
  • Y1 is a positive integer less than Y.
  • Y1 can be set to 1, that is, the second difference map is obtained by subtracting the last Y-1 column and the first Y-1 column.
  • Step 193 adding the processed first difference map and the processed second difference map to obtain a third difference map
  • Step 194 assigning a value of 1 to a pixel whose absolute value is greater than a preset threshold in the third difference map, and assigning a value of 0 to a pixel whose absolute value is less than the preset threshold, to obtain a binary map;
  • Step 195 Obtain a motion mask according to the binary image.
  • obtaining the motion mask according to the binary image includes: performing a dilation operation on the binary image to obtain the motion mask.
  • obtaining a reference image includes:
  • the N 2 products are added and averaged to obtain the reference map.
  • the N 2 products may also be weighted and added and averaged to obtain the reference image. Weights can be set as desired.
  • obtaining a second denoising map according to the first denoising map and the target non-motion mask includes: adding the first denoising map to the target non-motion mask The films are multiplied to obtain the second denoised map.
  • the N 2 is 5-9.
  • N 2 as 5 is an example to illustrate the training method of the above-mentioned denoising network.
  • F1, F2, F3, F4, and F5 are consecutive 5 frames of video images
  • F3 is the current video image to be denoised
  • DN3 is the denoising image output by the denoising network
  • corresponding to F3 M is the target non-motion mask
  • * represents the multiplication of the corresponding pixel positions
  • Ref3 represents the reference image, which can be considered as the true value corresponding to DN3.
  • the method to obtain the reference image is: multiply F1, F2, F3, F4, and F5 by M respectively, add them, and take the average value.
  • This is a principle of time-domain denoising. It uses the principle that the effective information distribution between consecutive frames is the same, but the noise is randomly and irregularly distributed. Multiple frames are added to remove the mean value, which can retain the effective information but cancel the effect of random noise.
  • the purpose of calculating the non-motion mask is to ensure that the valid information of the corresponding position pixels of the denoised image and the reference image is the same.
  • non-motion mask If there is no non-motion mask step, directly add multiple frames to remove the mean, and the valid information of the non-motion position can be retained, but the motion The position will produce serious artifacts, destroy the original effective information, and cannot be used as a reference image for training.
  • the non-motion mask only the non-motion positions are retained in the generated reference map, and the pixels at the corresponding positions are also retained in the denoising map to form training data pairs for training.
  • the method for obtaining the denoising map is: input F3, or F3 and its adjacent video images into the denoising network to obtain the first denoising map, and multiply the first denoising map by M to obtain the second denoising map.
  • Denoising map i.e. DN3
  • the denoising network may be any denoising network.
  • FIG. 18 is a schematic diagram of an implementation method of a denoising network according to an embodiment of the present disclosure.
  • the input of the denoising network is 5 frames of continuous video images.
  • the denoising network includes a plurality of filters in series, each filter including a plurality of convolution kernels in series (vertical bars in Figure 18).
  • each filter includes four convolution kernels connected in series.
  • the number of convolution kernels in the filter is not limited to four.
  • every two filters have the same resolution, and except for the last filter, the output of each of the other filters is used as the next filter and has the same resolution. input to the filter.
  • the denoising network includes 6 filters in series, wherein the first filter has the same resolution as the sixth filter, and the second filter has the same resolution as the fifth filter.
  • the resolution is the same, the third filter has the same resolution as the fourth filter, and the output of the first filter is used as the second filter and the sixth filter (same resolution as the first filter ), the output of the second filter is taken as the input of the third and fifth filter (same resolution as the second filter), the output of the third filter is taken as the fourth filter input of the filter (same resolution as the third filter).
  • the saved parameters can be used as the initialization parameters for the next video noise reduction, so that a new training can be completed only in about 100 frames of the new video.
  • FIG. 19 is a video image to be processed
  • FIG. 20 is an enlarged view of a part of the video image to be processed in FIG. 19
  • FIG. 21 is a target image corresponding to the video image to be processed in FIG. 19 Motion mask M
  • FIG. 22 is an image after denoising the video image to be processed by using the denoising network according to the embodiment of the present disclosure. It can be seen from the comparison results that the denoising effect is obvious.
  • Color digital images collected by digital imaging devices are synthesized by three channels: red (R), green (G), and blue (B).
  • RGB red
  • G green
  • B blue
  • the captured image may have a certain color deviation from the original scene due to factors such as illumination and photosensitive elements, which is called color cast.
  • the color cast image shows that the average pixel value of one or several channels in the three channels of R, G and B of the image is obviously high. Color distortion caused by color cast seriously affects the visual effect of images. Therefore, color cast correction of digital images is an important issue in the field of digital image processing. When dealing with old photos and video data, it is often necessary to deal with color cast problems due to the age and preservation issues.
  • the color cast correction steps in the embodiment of the present disclosure include:
  • Step 191 Determine the target color cast value of each RGB channel of the video image to be processed
  • Step 192 According to the target color cast value, perform color balance adjustment on the to-be-processed video image to obtain a first corrected image;
  • Step 193 Perform color migration on the first corrected image according to the reference image to obtain a second corrected image.
  • the reference image is an input image with substantially no color cast.
  • the degree of color cast of the image is automatically estimated first, the color balance adjustment of the video image to be processed is performed to initially correct the color cast, and then, according to the reference image, the color transfer process is performed on the image after the color balance adjustment, and the color cast is further adjusted. , so that the color cast correction results are more in line with the ideal expectations.
  • determining the target color cast value of each RGB channel of the video image to be processed includes:
  • Step 201 Obtain the mean value of each RGB channel of the video image to be processed
  • the calculation method of the mean (avgR, avgG, avgB) of the three RGB channels is: adding the grayscale values of all R sub-pixels in the video image to be processed, and then averaging to obtain avgR; adding all G in the video image to be processed The gray values of the sub-pixels are added, and then the average value is obtained to obtain avgG; the gray values of all B sub-pixels in the video image to be processed are added, and then the average value is obtained to obtain avgB.
  • Step 202 Convert the mean value of each RGB channel to Lab color space, and obtain each color component (1, a, b) of the Lab space corresponding to the mean value of each RGB channel;
  • Lab is a device-independent color system and a color system based on physiological characteristics. This also means that it uses a digital method to describe human visual perception.
  • the L component in the Lab color space is used to represent the brightness of the pixel, and the value range is [0,100], which means from pure black to pure white; a means the range from red to green, and the value range is [127,-128]; b represents the range from yellow to blue, and the value range is [127,-128].
  • the mean values of a and b should be close to 0. If a>0, the image is reddish, otherwise it is greenish; if b>0, it is yellowish, otherwise it is blueish.
  • Step 203 According to each color component (1, a, b) of the Lab space, determine the color cast degree (1, 0-a, 0-b) corresponding to the mean value of each RGB channel;
  • the color components a and b of the mean value of an image without color cast should be close to 0, so the color cast degree corresponding to the mean value of each RGB channel is (l, 0-a, 0-b) .
  • Step 204 Convert the color cast degree (1, 0-a, 0-b) to the RGB color space to obtain the target color cast value of each RGB channel.
  • the RGB color space cannot be directly converted to the Lab color space.
  • the XYZ color space needs to be used to convert the RGB color space to the XYZ color space, and then the XYZ color space is converted to the Lab color space.
  • converting the mean value of each channel of RGB to Lab color space includes: converting the mean value of each channel of RGB to XYZ color space to obtain the mean value of XYZ color space; Converting the mean value of XYZ color space to Lab color space;
  • converting the color cast degree (l, 0-a, 0-b) to the RGB color space includes: converting the color cast degree to the XYZ color space to obtain the color cast degree of the XYZ color space; converting the XYZ color space The degree of color cast of the color space is converted to the RGB color space.
  • RGB and XYZ may be as follows:
  • X n , Y n , and Z n are generally 0.95047, 1.0, and 1.08883 by default.
  • the concept of white balance is to define an area, take this area as a standard, and consider it to be white (18-degree gray to be precise), and the colors of other areas are color shifts based on this standard.
  • the principle of color balance adjustment is to increase or decrease its contrast color to eliminate the color cast of the picture.
  • performing color balance adjustment on the video image to be processed according to the target color cast value, and obtaining a first corrected image includes:
  • each channel of RGB perform at least one color balance adjustment processing among highlight function processing, shadow function processing and midtone function processing on the video image to be processed;
  • the highlight function and the shadow function are linear functions
  • the midtone function is an exponential function
  • y is the first corrected image
  • x is the video image to be processed
  • v is determined according to the target color cast value of each RGB channel
  • f(v), a(v), b(v), c(v), d( v) is a function of v.
  • the algorithm effect is to increase the value of R channel only, and the other two channels remain unchanged; for negative adjustment, such as R channel -50, the algorithm effect is R The channel remains unchanged, and the remaining two channel values increase.
  • the algorithm effect is that the value of R channel remains unchanged, and the other two channel values decrease; for negative adjustment, such as R channel -50, the algorithm effect is R The channel value is decreased, and the remaining two channel values are unchanged.
  • f(v) e ⁇ v .
  • v ( ⁇ R-d)-( ⁇ G-d)-( ⁇ B-d);
  • ⁇ R, ⁇ G, and ⁇ B are the target color cast values of each RGB channel, and d is the middle value after sorting ⁇ R, ⁇ G, and ⁇ B by size.
  • ⁇ R is 10
  • ⁇ G is 15, and ⁇ B is 5.
  • d 10.
  • performing color migration on the first corrected image according to the reference image to obtain the second corrected image includes:
  • Step 211 Convert the reference image and the first corrected image to Lab color space
  • Step 212 in the Lab color space, determine the mean and standard deviation of the reference image and the first corrected image;
  • Step 213 Determine the color migration result of the kth channel in the Lab color space according to the mean and standard deviation of the reference image and the first corrected image;
  • Step 214 Convert the color migration result to RGB color space to obtain the second corrected image.
  • the calculation method of the color migration result is as follows:
  • I k is the color migration result of the kth channel of the Lab color space
  • t is the reference image
  • S is the first corrected image
  • the brightness channel migration will cause the brightness of the image to change, especially for images with a large area of the same color, the change of the brightness channel will bring about visual changes. Therefore, in the embodiment of the present disclosure, only the ab channel is migrated, that is, the kth channel is at least one of the a and b channels, so that the color cast is corrected and the brightness of the image is kept unchanged.
  • an embodiment of the present disclosure further provides an image processing apparatus 200, including:
  • Processing module 201 includes at least one of the following modules:
  • Scratch repair sub-module 2011 perform scratch removal processing on the video image to be processed to obtain a first image; perform a difference operation on the video image to be processed and the first image to obtain a difference image; process the difference image to obtain a difference image Only the scratch image of the scratch is retained; the scratch repair image is obtained according to the to-be-processed video image and the scratch image;
  • Dead pixel repairing sub-module 2012 Acquire consecutive N1 frames of video images, where the N1 frames of video images include to-be-processed video images, at least one frame of video image before the to-be-processed video image, and at least one frame of video after image, wherein, N 1 is a positive integer greater than or equal to 3; according to at least one frame of video image before and at least one frame of video image after the video image to be processed, filtering processing is performed on the video image to be processed to obtain a bad image. point-repaired image; perform de-artifact processing on the dead-point repaired image according to at least one frame of video image before and at least one frame of video image after the video image to be processed to obtain an artifact-repaired image;
  • Denoising sub-module 2013 use a denoising network to denoise the to-be-processed video image, and the denoising network is obtained by the following training method: obtaining a target non-motion mask according to consecutive N 2 frames of video images, the N 2 frames The video image includes the video image to be de-noised; according to the N 2 frames of video image and the target non-motion mask, the de-noising network to be trained is trained to obtain the de-noising network;
  • Color cast correction sub-module 2014 determine the target color cast value of each RGB channel of the video image to be processed; perform color balance adjustment on the video image to be processed according to the target color cast value to obtain a first corrected image; image, and performing color migration on the first corrected image to obtain a second corrected image.
  • the scratch repairing sub-module performing scratch removal processing on the video image to be processed includes: according to at least one of a filter type and a scratch type in the video image to be processed, performing a scratch removal process on the video image to be processed.
  • the to-be-processed video image is subjected to median filtering to obtain a scratch-free image.
  • performing median filtering processing on the to-be-processed video image includes:
  • a median filter in the vertical direction is used to perform median filtering on the video image to be processed.
  • performing median filtering on the to-be-processed video image includes:
  • the video image to be processed is rotated to convert the scratches into horizontal scratches.
  • performing the scratch removal processing on the video image to be processed includes: using a median filter with a size of 1 ⁇ k and/or k ⁇ 1 to perform median filtering on the video image to be processed;
  • the scratch repair sub-module is further used for: increasing k of the median filter sequentially from a preset value, and performing median filtering on the video image to be processed to obtain a second image; according to the second image
  • performing a difference operation on the video image to be processed and the first image to obtain a difference image includes: performing a difference operation on the video image to be processed and the first image to obtain the first image.
  • a difference image and/or a second difference image wherein the first difference image is obtained by subtracting the first image from the video image to be processed, and the second difference image is obtained by subtracting the first image from the first image.
  • the described video image to be processed is obtained;
  • Processing the difference image to obtain a scratch image that only retains scratches includes: processing the first difference image to obtain a first scratch image that retains only scratches; and/or, processing the second difference image processing to obtain a second scratch image that only retains scratches;
  • Obtaining the scratch repaired image according to the to-be-processed video image and the scratch image includes: obtaining the scratch by performing an operation on the to-be-processed video image, the first scratch image and/or the second scratch image Fix images.
  • processing the first difference image to obtain a first scratch image that only retains scratches includes:
  • the scratches in the to-be-processed video image are vertical scratches, subtract the first horizontally filtered image from the first vertical filtered image to obtain the first scratched image;
  • the scratches in the to-be-processed video image are horizontal scratches, subtract the first vertical filtered image from the first horizontally filtered image to obtain the first scratched image;
  • Processing the second difference image to obtain a second scratch image that only retains scratches includes:
  • the scratches in the to-be-processed video image are vertical scratches, subtract the second horizontally filtered image from the second vertical filtered image to obtain the second scratched image;
  • the scratches in the to-be-processed video image are horizontal scratches, subtract the second vertical filtered image from the second horizontally filtered image to obtain the second scratched image.
  • I deline IL white +L black
  • I deline is the scratch repaired image
  • I is the video image to be processed
  • L white is the first scratch image
  • L black is the second scratch image
  • a dead pixel repair sub-module is configured to perform a mid-cycle processing on the video image to be processed according to at least one frame of video image before and at least one frame of video image after the video image to be processed. Value filtering to get the dead pixel repaired image.
  • the artifact-repaired image obtained by performing de-artifact processing on the dead pixel repair image according to at least one frame of video image before and at least one frame of video image after the video image to be processed includes the following steps: :
  • N 3 -1 downsampling on the dead pixel repair image, at least one frame of video image before the to-be-processed video image, and at least one frame of video image after, respectively, to obtain N 3 -1 resolution downsampling image, wherein the down-sampled image of each resolution includes N 1 down-sampled images corresponding to the dead pixel repair image, at least one frame of video image before and at least one frame of video image after the video image to be processed, respectively.
  • the N 3 resolution images include: the dead pixel restoration image, the to-be-repaired image Processing at least one frame of video image before the video image and at least one frame of video image after and the down-sampled images of the N 3 -1 resolutions, the multi-scale cascaded network includes N 3 cascaded sub-networks, The images processed by the N 3 cascaded sub-networks are respectively generated based on the images of the N 3 resolutions;
  • N 3 is a positive integer greater than or equal to 2.
  • inputting images of N 3 resolutions into a multi-scale cascade network for de-artifact processing to obtain an artifact-repaired image includes:
  • the repaired image for dead pixels at least one frame of video image before the video image to be processed, and at least one frame of video image after A times down-sampling to obtain N 1 frames of first down-sampled images, respectively splicing the N 1 frames of first down-sampled images with itself to obtain a first spliced image, and inputting the first spliced image into the A sub-network to get the first output image;
  • the intermediate sub-network between the first sub-network and the last sub-network up-sampling the output image of the previous sub-network to obtain the first up-sampled image;
  • the previous at least one frame of video image and the next at least one frame of video image are down-sampled by B times, respectively, to obtain N 1 frames of second down-sampled images, and the scale of the second down-sampled image and the first up-sampled image is
  • splicing two sets of images to obtain a second spliced image and inputting the second spliced image into the intermediate sub-network to obtain a second output image, wherein one of the two sets of images is N 1 frames of second down-sampled images, and the other group includes: other down-sampled images in the N 1 frames of second down-sampled images except the down-sampled images corresponding to the dead pixel repair image, and the first an upsampled image;
  • the third image after splicing is input into the last sub-network to obtain the artifact repaired image, wherein one of the two sets of images is the N 1 frames of video images , and the other group includes: other images in the N 1 frames of video images except the dead pixel repair image, and the second up-sampled image.
  • the N 3 cascaded sub-networks have the same structure and different parameters.
  • each of the sub-networks includes multiple 3D convolution layers, multiple 3 deconvolution layers, and multiple 3D average pooling layers.
  • N 3 is equal to 3
  • A is equal to 4
  • B is equal to 2.
  • the multi-scale cascade network is obtained by adopting the following training method:
  • N1 frames of training images include to-be-processed training images, at least one frame of training images before the to-be-processed training images, and at least one frame of training images after;
  • the first training image is obtained by filtering the to-be-processed training image according to the N1 frames of training images;
  • At least one frame of training image before and at least one frame of training image after the training image to be processed train the multi-scale cascaded network to be trained, and obtain the multi-scale cascaded network after training .
  • the total loss used includes at least one of the following: image content loss, color loss, edge loss, and perceptual loss.
  • the total loss is equal to a weighted sum of image content loss, color loss, edge loss, and perceptual loss.
  • Image content L1 loss is calculated using the following formula:
  • l content is the L1 loss
  • yi is the first training image
  • n is the number of images in a Batch.
  • the color loss is calculated by using the following formula:
  • lcolor is the color loss
  • yi is the first training image
  • n is the number of images in a Batch
  • Blur(x) is a Gaussian blur function
  • the edge loss is calculated by using the following formula:
  • H j (x) represents the image edge map extracted by the jth layer of the HED network.
  • the perceptual loss is calculated by using the following formula:
  • l feature is the perceptual loss
  • y i is the first training image
  • n is the number of images in a Batch
  • the multi-scale cascaded network to be trained is trained according to the first training image, at least one frame of training image before the to-be-processed training image, and at least one frame of training image after the training image include:
  • An image block is randomly cropped on the first training image, and an image block is cropped at the same position of at least one frame of training image before and at least one frame of training image after the to-be-processed training image, to obtain N 1 frame image block;
  • the N 1 frame image blocks are input into the multi-scale cascaded network to be trained for training.
  • the dead pixel restoration sub-module is configured to: perform filtering processing on the artifact restoration image according to the to-be-processed video image and the dead pixel restoration image to obtain an output image.
  • the dead pixel restoration sub-module is configured to: perform median filtering processing on the artifact restoration image according to the to-be-processed video image and the dead pixel restoration image to obtain an output image.
  • N 1 is equal to 3.
  • the denoising sub-module, for obtaining the target non-motion mask according to the continuous N 2 frames of video images includes:
  • a loss function of the denoising network to be trained is determined according to the reference map and the second denoising map, and parameters of the denoising network to be trained are adjusted according to the loss function to obtain the denoising network.
  • obtaining a reference image includes:
  • Each frame of the first video image in the N 2 frames of video images and the video image to be denoised form a sample pair and input it into the optical flow network to obtain a first optical flow diagram representing the up-down motion information and a left-right motion diagram representing the left and right motion.
  • the second optical flow diagram of the information, the first video image is other video images in the N 2 frames of video images except the video image to be denoised, the first optical flow diagram and the second optical flow diagram
  • the resolution of the optical flow map is X*Y;
  • the first optical flow map and the second optical flow map calculate the motion masks of each frame of the first video image and the video image to be denoised, to obtain N 2 ⁇ 1 eye motion masks;
  • the target non-motion mask is obtained according to the N 2 ⁇ 1 motion masks.
  • calculating the motion mask of each frame of the first video image and the video image to be denoised includes:
  • a motion mask is obtained from the binary image.
  • obtaining the motion mask according to the binary image includes:
  • Dilation operation is performed on the binary image to obtain the motion mask.
  • obtaining the target non-motion mask according to the N 2 ⁇ 1 motion masks includes:
  • obtaining a reference image includes:
  • the N 2 products are added and averaged to obtain the reference map.
  • obtaining a second denoising map according to the first denoising map and the target non-motion mask M includes:
  • the N 2 is 5-9.
  • an optional color cast correction sub-module configured to determine the target color cast value of each RGB channel of the video image to be processed, includes:
  • each color component (1,a,b) of described Lab space determine the color cast degree (1,0-a,0-b) corresponding to the mean value of each channel of described RGB;
  • converting the mean value of each RGB channel to the Lab color space includes: converting the mean value of each RGB channel to the XYZ color space to obtain the mean value of the XYZ color space; converting the XYZ color space to the mean value of the XYZ color space; The mean of , is converted to Lab color space;
  • Converting the color cast degree (l, 0-a, 0-b) to the RGB color space includes: converting the color cast degree to the XYZ color space to obtain the color cast degree of the XYZ color space; converting the color cast degree of the XYZ color space The degree of color cast is converted to the RGB color space.
  • performing color balance adjustment on the video image to be processed according to the target color cast value, and obtaining a first corrected image includes:
  • At least one color balance adjustment processing among highlight function processing, shadow function processing and midtone function processing is performed on the video image to be processed, wherein the highlight function and shadow function are A linear function, and the intermediate tone function is an exponential function.
  • y is the first corrected image
  • x is the video image to be processed
  • v is determined according to the target color cast value of each RGB channel
  • f(v), a(v), b(v), c(v), d( v) is a function of v.
  • f(v) e -v .
  • v ( ⁇ R-d)-( ⁇ G-d)-( ⁇ B-d);
  • ⁇ R, ⁇ G, and ⁇ B are the target color cast values of each RGB channel
  • d is the intermediate value after sorting ⁇ R, ⁇ G, and ⁇ B according to size.
  • performing color migration on the first corrected image according to the reference image, and obtaining the second corrected image includes:
  • the calculation method of the color migration result is as follows:
  • I k is the color migration result of the kth channel of the Lab color space
  • t is the reference image
  • S is the first corrected image
  • the kth channel is at least one of a channel and a channel b.
  • An embodiment of the present application further provides an electronic device, including a processor, a memory, a program or an instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, the above image processing method is implemented
  • an electronic device including a processor, a memory, a program or an instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, the above image processing method is implemented
  • Embodiments of the present application further provide a readable storage medium, where a program or an instruction is stored on the readable storage medium.
  • a program or an instruction is stored on the readable storage medium.
  • the processor is the processor in the terminal described in the foregoing embodiment.
  • the readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like.
  • the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation.
  • the technical solution of the present application can be embodied in the form of a software product in essence or in a part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of this application.
  • a storage medium such as ROM/RAM, magnetic disk, CD-ROM

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Image Processing (AREA)

Abstract

本公开提供一种图像处理方法及图像处理装置，该方法包括：对待处理视频图像执行以下步骤中的至少之一：划痕修复步骤、坏点修复步骤、去噪步骤和偏色矫正步骤。本公开实施例中，能够对视频图像进行划痕恢复、坏点恢复、噪声去除和/或偏色矫正，以提高视频图像的显示效果。

Description

图像处理方法及图像处理装置 技术领域
本公开实施例涉及显示技术领域，尤其涉及一种图像处理方法及图像处理装置。
背景技术
视频图像例如电影胶片，有可能会因时长或者保存不善，出现划痕、坏点、噪声或偏色等问题，如何对视频图像出现的上述问题进行修复，以提高显示效果，是亟待解决的问题。
发明内容
本公开实施例提供一种图像处理方法及图像处理装置，用于对视频图像进行划痕恢复、坏点恢复、噪声去除和/或偏色矫正，以提高显示效果。
为了解决上述技术问题，本公开是这样实现的：
第一方面，本公开实施例提供了一种图像处理方法，包括：
对待处理视频图像执行以下步骤中的至少之一：
划痕修复步骤：对待处理视频图像进行去划痕处理，得到第一图像；对所述待处理视频图像和所述第一图像进行差异运算得到差异图；对所述差异图像进行处理得到只保留划痕的划痕图像；根据所述待处理视频图像和所述划痕图像得到划痕修复图像；
坏点修复步骤：获取连续的N 1帧视频图像，所述N 1帧视频图像中包括待处理视频图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像；根据所述N 1帧视频图像，对所述待处理视频图像进行滤波处理得到坏点修复图像；根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像对所述坏点修复图像进行去伪影处理得到伪影修复图像；
去噪步骤：采用去噪网络对待处理视频图像进行去噪处理，所述去噪网络采用以下训练方法得到：根据连续的N 2帧视频图像得到目标非运动掩膜， 所述N 2帧视频图像包括待去噪视频图像；根据所述N 2帧视频图像和所述目标非运动掩膜，对待训练的去噪网络进行训练，得到所述去噪网络；
偏色矫正步骤：确定待处理视频图像的RGB各通道的目标偏色值；根据所述目标偏色值，对所述待处理视频图像进行色彩平衡调整，得到第一矫正图像；根据参考图像，对所述第一矫正图像进行颜色迁移，得到第二矫正图像。
第二方面，本公开实施例提供了一种图像处理装置，包括：
处理模块，所述处理模块包括以下模块中的至少之一：
划痕修复子模块：对待处理视频图像进行去划痕处理，得到第一图像；对所述待处理视频图像和所述第一图像进行差异运算得到差异图；对所述差异图像进行处理得到只保留划痕的划痕图像；根据所述待处理视频图像和所述划痕图像得到划痕修复图像；
坏点修复子模块：获取连续的N 1帧视频图像，所述N 1帧视频图像中包括待处理视频图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像；根据所述N 1帧视频图像，对所述待处理视频图像进行滤波处理得到坏点修复图像；根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像对所述坏点修复图像进行去伪影处理得到伪影修复图像；
去噪子模块：采用去噪网络对待处理视频图像进行去噪处理，所述去噪网络采用以下训练方法得到：根据连续的N 2帧视频图像得到目标非运动掩膜，所述N 2帧视频图像包括待去噪视频图像；根据所述N 2帧视频图像和所述目标非运动掩膜，对待训练的去噪网络进行训练，得到所述去噪网络；
偏色矫正子模块：确定待处理视频图像的RGB各通道的目标偏色值；根据所述目标偏色值，对所述待处理视频图像进行色彩平衡调整，得到第一矫正图像；根据参考图像，对所述第一矫正图像进行颜色迁移，得到第二矫正图像。
第三方面，本公开实施例提供了一种电子设备，包括处理器，存储器及存储在所述存储器上并可在所述处理器上运行的程序或指令，所述程序或指令被所述处理器执行时实现上述图像处理方法的步骤。
第四方面，本公开实施例提供了一种可读存储介质，所述可读存储介质上存储程序或指令，所述程序或指令被处理器执行时实现上述图像处理方法的步骤。
在本公开实施例中，能够对视频图像进行划痕恢复、坏点恢复、噪声去除和/或偏色矫正，以提高视频图像的显示效果。
附图说明
通过阅读下文优选实施方式的详细描述，各种其他的优点和益处对于本领域普通技术人员将变得清楚明了。附图仅用于示出优选实施方式的目的，而并不认为是对本公开的限制。而且在整个附图中，用相同的参考符号表示相同的部件。在附图中：
图1为本公开实施例的划痕修复步骤的流程示意图；
图2为本公开实施例的划痕修复步骤的具体流程示意图；
图3为本公开实施例的待处理视频图像和第一图像的示意图；
图4为本公开实施例的第一差异图像和第二差异图像的示意图；
图5为本公开实施例的第一划痕图像和第二划痕图像的示意图；
图6为本公开实施例的待处理视频图像和划痕修复图像的比较示意图；
图7为本公开实施例的坏点修复步骤的流程示意图；
图8为本公开实施例的待处理视频图像和坏点修复图像的比较示意图；
图9为本公开实施例的多尺度级联网络的结构示意图；
图10为本公开实施例的多尺度级联网络的输入图像示意图；
图11为本公开实施例的待处理视频图像和多尺度级联网络的输出图像的比较示意图；
图12为本公开实施例中的子网络的结构示意图；
图13为本公开实施例的多尺度级联网络的输出图像和后处理后的图像的比较示意图；
图14为本公开实施例的去噪步骤的流程示意图；
图15为本公开实施例的利用光流网络得到运动掩膜的流程示意图；
图16为本公开实施例的利用光流网络得到运动掩膜的具体流程示意图；
图17为本公开实施例的去噪网络的训练方法的示意图；
图18为本公开实施例的一种去噪网络的实现方法示意图；
图19为本公开实施例的待处理视频图像的示意图；
图20为图19中的待处理视频图像中的部分区域的放大图；
图21为图19中的待处理视频图像对应的目标非运动掩膜M；
图22为采用本公开实施例的去噪网络对待处理视频图像进行去噪处理后的图像；
图23为本公开实施例的偏色矫正步骤的流程示意图；
图24为本公开实施例的图像处理装置的结构示意图。
具体实施方式
下面将结合本公开实施例中的附图，对本公开实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本公开一部分实施例，而不是全部的实施例。基于本公开中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本公开保护的范围。
本公开实施例提供一种图像处理方法，包括：
对待处理视频图像执行以下步骤中的至少之一：
划痕修复步骤：对待处理视频图像进行去划痕处理，得到第一图像；对所述待处理视频图像和所述第一图像进行差异运算得到差异图像；对所述差异图像进行处理得到只保留划痕的划痕图像；根据所述待处理视频图像和所述划痕图像得到划痕修复图像；
坏点修复步骤：获取连续的N 1帧视频图像，所述N 1帧视频图像中包括待处理视频图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像；根据所述N 1帧视频图像，对所述待处理视频图像进行滤波处理得到坏点修复图像；根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像对所述坏点修复图像进行去伪影处理得到伪影修复图像；
去噪步骤：采用去噪网络对待处理视频图像进行去噪处理；其中，所述去噪网络采用以下训练方法得到：根据连续的N 2帧视频图像得到目标非运动 掩膜M，所述N 2帧视频图像包括待去噪视频图像；根据所述N 2帧视频图像和所述目标非运动掩膜M，对待训练的去噪网络进行训练，得到所述去噪网络；
偏色矫正步骤：确定待处理视频图像的RGB各通道的目标偏色值；根据所述目标偏色值，对所述待处理视频图像进行色彩平衡调整，得到第一矫正图像；根据参考图像，对所述第一矫正图像进行颜色迁移，得到第二矫正图像。
需要说明的是，可以对视频图像执行以上四个步骤中的至少一项，若需执行多个步骤，该多个步骤的执行顺序不限，例如需要执行划痕修复和坏点修复步骤，可以对视频图像先执行划痕修复，再执行坏点修复，或者，也可以先执行坏点修复，再执行划痕修复。
本公开实施例中，通过视频图像执行划痕修复、坏点修复、去噪或偏色校正中的至少一个步骤，从而提高视频图像的显示效果。
下面分别对上述四个步骤进行说明。
一、划痕修复
请参考图1，图1为本公开实施例中的划痕修复步骤的流程示意图，包括：
步骤11：对待处理视频图像进行去划痕处理，得到第一图像；
本公开实施例中，可以对待处理视频图像进行滤波处理，以去除划痕，所述滤波处理例如为中值滤波处理等。
步骤12：对所述待处理视频图像和所述第一图像进行差异运算得到差异图像；
步骤13：对所述差异图像进行处理得到只保留划痕的划痕图像；
步骤14：根据所述待处理视频图像和所述划痕图像得到划痕修复图像。
本公开实施例中，对单帧的待处理视频图像进行去划痕处理处理，可以得到去掉划痕的图像，将待处理视频图像和去掉划痕的图像进行差异运算得到包含划痕和图像细节的差异图像，然后对差异图像再次进行处理，滤除图像细节保留划痕得到划痕图像，并根据待处理视频图像和划痕图像得到去除划痕的划痕修复图像，在去除划痕的同时，还能够不影响图像清晰度。
下面分别对上述各步骤进行详细说明。
(1)步骤11：
本公开实施例中，可选的，对待处理视频图像进行去划痕处理包括：根据滤波器类型和所述待处理视频图像中的划痕类型中的至少一个，对所述待处理视频图像进行中值滤波处理，得到去划痕图像。
可选的，根据滤波器类型和所述待处理视频图像中的划痕类型中的至少一个，对所述待处理视频图像进行中值滤波处理包括：根据所述待处理视频图像中的划痕类型，选取对应的滤波器类型，对所述待处理视频图像进行中值滤波处理，其中：
当所述待处理视频图像中的划痕为垂直划痕时，采用水平方向的中值滤波器对所述待处理视频图像进行中值滤波；
当所述待处理视频图像中的划痕为水平划痕时，采用对垂直方向的中值滤波器所述待处理视频图像进行中值滤波。
也就是说，本公开实施例中，可以根据待处理视频图像中划痕的方向，确定采用的中值滤波器。
当然，在本公开的其他一些实施例中，也可以是不改变中值滤波器，通过对待视频图像进行旋转处理，以使得待处理视频图像中的划痕与中值滤波器匹配。
即，根据滤波器类型和所述待处理视频图像中的划痕类型中的至少一个，对所述待处理视频图像进行中值滤波处理包括：根据滤波器类型和所述待处理视频图像中的划痕类型，对所述待处理视频图像不同的预处理，对预处理之后的待处理视频图像进行中值滤波处理，其中：
当采用的是水平方向的中值滤波器，且所述待处理视频图像中的划痕为非垂直划痕时，将所述待处理视频图像进行旋转，使得划痕转换为垂直划痕；非垂直划痕包括水平划痕和斜向划痕。当然，如果待处理视频图像中的划痕为垂直划痕，则无需对待处理图像进行旋转处理。
当若采用的是垂直方向的中值滤波器，且所述视频图像中的划痕为非水平划痕时，将所述待处理视频图像进行旋转，使得划痕转换为水平划痕。非水平划痕包括垂直划痕和斜向划痕。当然，如果待处理视频图像中的划痕为水平划痕，则无需对待处理图像进行旋转处理。
此外，如果待处理视频图像中既有水平划痕，又有垂直划痕，可以同时采用水平方向的中值滤波器和垂直方向的中值滤波器对待处理视频图像进行中值滤波处理。例如，先进行水平方向的中值滤波处理，再进行垂直方向的中值滤波处理，或者，先进行垂直方向的中值滤波处理，再进行水平方向的中值滤波处理。
本公开实施例中，对待处理视频图像进行去划痕处理包括：采用尺寸为1×k和/或k×1的中值滤波器对待处理视频图像进行中值滤波；其中，1×k的中值滤波器为水平方向的中值滤波器，而，k×1的中值滤波器为垂直方向的中值滤波器。
举例来说，使用1×k的中值滤波器对待处理视频图像I进行滤波，得到一幅第一图像I median。
I median＝M 1×k(I)
其中，M 1×k(x)代表对x采用尺寸为1×k的中值滤波器进行滤波操作。
下面对如何确定滤波器的尺寸进行说明。
本公开实施例中，可选的，对待处理视频图像进行去划痕处理之前还包括：将所述中值滤波器的k从预设值开始依次增大，对所述待处理视频图像进行中值滤波，得到第二图像；根据所述第二图像的滤波效果确定k的最终值。
例如，假设采用的中值滤波器为1×k的水平方向的中值滤波器，可以首先将中值滤波器的k设置为3(预设值)，即采用1×3的中值滤波器对待处理视频图像进行滤波，得到第二图像，并观察第二图像的滤波效果，如果划痕去除效果不明显，可以将k的值设置为4(或者其他大于3的数值)，即采用1×4的中值滤波器对待处理视频图像进行滤波，得到第二图像，并观察第二图像的滤波效果，如果划痕去除效果不明显，再次增加l＝k的值，直至第二图像没有明显的划痕为止。
当然，在本公开的其他一些实施例中，也可以根据划痕的粗细程度，直接确定k的值，例如分辨率为2560×1440的图像，可以设置k为11以下的数值。
下面举例进行说明。请参考图3，图3中(a)为待处理视频图像和待处 理视频图像的局部放大图，从图中可以看出，该待处理视频图像中具有垂直划痕。请参考图2，可以采用水平方向的中值滤波器对待处理视频图像进行去划痕处理，得到第一图像。参见图3中(b)，图3中(b)为第一图像和第一图像的局部放大图，从图中可以看出，第一图像中已经去除了垂直划痕，但是第一图像与待处理图像相比，较为模糊。
(2)步骤12：
本公开实施例中，对所述待处理视频图像和所述第一图像进行差异运算得到差异图像包括：对所述待处理视频图像和所述第一图像进行差异运算，得到第一差异图像和/或第二差异图像，其中，所述第一差异图像由所述待处理视频图像减去所述第一图像得到，所述第二差异图像由所述第一图像减去所述待处理视频图像得到。
本公开实施例中，两个图像相减，指示两个图像的对应位置的像素相减。
第一差异图像是包含图像细节和划痕的白色纹理图，而，第二差异图像是包含图像细节和划痕的黑色纹理图。
本申请实施例中，第一差异图像也可以称为正残差图像，第二差异图像也可以称为负残差图像。
举例来说，用待处理视频图像I与第一图像I median互减，分别得到正残差Err white与负残差Err black，计算公式如下，其中，正残差和负残差都是正值。
Err white＝I-I median
Err black＝I median-I
仍以图2为例，将待处理视频图像和第一图像互减，其中，待处理视频图像乘以1(×1)与第一图像乘以-1(×-1)之后相加，即待处理视频图像减去第一图像，得到第一差异图像，待处理视频图像乘以-1(×-1)与第一图像乘以1(×1)之后相加，即第一图像减去待处理视频图像，得到第二差异图像。请参考图4，图4中的(a)为第一差异图像和第一差异图像的局部放大图，图4中的(b)为第二差异图像和第二差异图像的局部放大图。可以看出，第一差异图像为包含图像细节和划痕的白色纹理图，第二差异图像为包含图像细节和划痕的黑色纹理图。
上述例子中，同时计算第一差异图像和第二差异图像，当然，在本公开 的其他一些实施例中，也不排除仅计算第一差异图像或者仅计算第二差异图像。
(3)步骤13：
本公开实施例中，对所述差异图像进行处理得到只保留划痕的划痕图像包括：对所述第一差异图像进行处理得到只保留划痕的第一划痕图像；和/或，对所述第二差异图像进行处理得到只保留划痕的第二划痕图像；
划痕图像是滤除了图像细节只保留划痕的图像。
本申请实施例中，第一划痕图像也可以称为正划痕图像，第二划痕图像也可以称为负划痕图像。
本公开实施例中，可选的，对所述第一差异图像进行处理得到只保留划痕的第一划痕图像包括：
分别采用垂直方向的中值滤波器和水平方向的中值滤波器对第一差异图像进行中值滤波，得到第一垂直滤波后图像和第一水平滤波后图像；
若所述待处理视频图像中的划痕为垂直划痕，采用所述第一垂直滤波后图像减去所述第一水平滤波后图像，得到所述第一划痕图像；
若所述待处理视频图像中的划痕为水平划痕，采用所述第一水平滤波后图像减去所述第一垂直滤波后图像，得到所述第一划痕图像；
举例来说，假设待处理视频图像中的划痕为垂直划痕，对Err white(第一差异图像)做垂直方向的中值滤波处理和水平方向中值滤波处理，得到第一垂直滤波后图像M n×1(Err white)和第一水平滤波后图像M 1×n(Err white)，再将第一垂直滤波后图像M n×1(Err white)减去第一水平滤波后图像M 1×n(Err white)，分别得到滤波后的第一划痕图像L white，此时的第一划痕图像为正数表示：
L white＝M n×1(Err white)-M 1×n(Err white)
其中，M n×1(Err white)表示在垂直方向对第一差异图像进行中值滤波，M 1×n(Err white)表示在水平方向对第一差异图像进行中值滤波。
对所述第二差异图像进行处理得到只保留划痕的第二划痕图像包括：
分别采用垂直方向的中值滤波器和水平方向的中值滤波器对第二差异图像进行中值滤波，得到第二垂直滤波后图像和第二水平滤波后图像；
若所述待处理视频图像中的划痕为垂直划痕，采用所述第二垂直滤波后 图像减去所述第二水平滤波后图像，得到所述第二划痕图像；
若所述待处理视频图像中的划痕为水平划痕，采用所述第二水平滤波后图像减去所述第二垂直滤波后图像，得到所述第二划痕图像。
举例来说，假设待处理视频图像中的划痕为垂直划痕，对Err black(第二差异图像)做垂直方向的中值滤波处理和水平方向中值滤波处理，得到第二垂直滤波后图像M n×1(Err black)和第二水平滤波后图像M 1×n(Err black)，再将第二垂直滤波后图像M n×1(Err black)减去第二水平滤波后图像M 1×n(Err black)，分别得到滤波后的第二划痕图像L black，此时的第二划痕图像为正数表示：
L black＝M n×1(Err black)-M 1×n(Err black)
其中，M n×1(Err black)表示在垂直方向对第二差异图像进行中值滤波，M 1×n(Err black)表示在水平方向对第二差异图像进行中值滤波。
本公开实施例中，由于划痕的长度通常大于图像细节中的线条的长度，因而为了滤除图像细节只保留划痕，可以将垂直方向的中值滤波器和水平方向的中值滤波器的n的值设置的较大，例如，可以设置为划痕的平均的长度的一半，举例来说，划痕的最大长度为180，则可以将n的值设置为80-100之间。
仍以图2为例，分别将第一差异图像和第二差异图像进行垂直方向的中值滤波和水平方向的中值滤波，得到垂直滤波后图像和水平滤波后图像，由于图2中的待处理视频图像中的划痕为垂直划痕，则需要采用所述垂直滤波后图像减去所述水平滤波后图像，得到第一划痕图像和第二划痕图像。请参考图5，图5中的(a)为第一划痕图像和第一划痕图像的局部放大图，图5中的(b)为第二划痕图像和第二划痕图像的局部放大图。可以看出，第一划痕图像为包含白色划痕的图像，第二划痕图像为包含黑色划痕的图像。
(4)步骤14：
根据所述待处理视频图像和所述划痕图像得到划痕修复图像包括：对所述待处理视频图像、所述第一划痕图像和/或第二划痕图像进行运算得到所述划痕修复图像。
本公开实施例中，可以采用如下公式计算得到所述划痕修复图像：
I deline＝I-L white-(L black×-1)＝I-L white+L black
其中，I deline为所述划痕修复图像，I为所述待处理视频图像，L white为所述第一划痕图像，L black为所述第二划痕图像。公式中，由于第二划痕图像L black为正值，因而，需要乘以-1来恢复成负值。
本公开实施例中，计算划痕修复图像时，也可以只使用第一划痕图像，或者，只使用第二划痕图像。
将待处理图像减去划痕图像，可以保证去除划痕，同时，也保留了图像的清晰度，请参考，图6中(a)为待处理视频图像和待处理视频图像的局部放大图，(b)为划痕修复图像和划痕修复图像的局部放大图，从图6中可以看出，划痕修复图像中的划痕已经去除，且与待处理视频图像相比，图像清晰度并没有发生变化。
二、坏点修复
请参考图7，图7为本公开实施例中的坏点修复步骤的流程示意图，包括：
步骤71：获取连续的N 1帧视频图像，所述N 1帧视频图像中包括待处理视频图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像；
本公开实施例中，N 1为大于或等于3的正整数，可以根据需要进行设置，例如可以为3。
步骤72：根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像，对所述待处理视频图像进行滤波处理得到坏点修复图像；
步骤73：根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像对所述坏点修复图像进行去伪影处理得到伪影修复图像。
由于采用多帧视频图像对待处理图像进行去坏点处理，因而得到的坏点修复图像会引入运动伪影，此时，将去坏点问题转换为去伪影问题，即将滤波去坏点和多尺度级联网络去伪影结合，实现对视频图像的坏点的修复。
也就是说，坏点修复可以包括两个过程：一是去坏点，二是去伪影，下面分别进行介绍。
(1)去坏点
本公开实施例的坏点修复方法可以应用于对电影胶片的坏点修复，当然， 也不排除对其他类型的视频图像的坏点修复。
坏点是一种常见电影胶片损伤，它是在胶片存放过程中，由于胶片表面的凝胶缺失或者沾染了污点所形成的白色或黑色的块状斑点，胶片中的坏点一般具有以下三个特点：
1)坏点内部的像素灰度标准差很小，块内灰度基本保持一致；
2)坏点在时域和空域的灰度不连续。由于这种损伤在一帧内是随机分布，坏点不太可能在相邻两帧中的同一位置重复出现，因此它呈现为一种时域上的脉冲性损伤。而在一帧图像内，坏点区域的灰度一般也与周围背景灰度差距较大，从而可被人眼所感知；
3)空间的相邻性。即如果某一像素在坏点内，那么它周围的像素也很有可能属于坏点区域。
本公开主要围绕坏点的第二个特性来实现对其修复，由于坏点在时域上具有不连续性，而相邻帧在相同位置的像素值往往是相近的，所以，本公开实施例中利用相邻的前后帧图像的内容对当前帧图像内的坏点进行修复。
本公开实施例中，可选的，根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像，对所述待处理视频图像进行中值滤波得到坏点修复图像。举例来说，假设N 1等于3，对当前待处理视频图像I t以及相邻前后帧图像I t-1、I t+1逐像素计算中值，由于同场景内相邻帧图像的相同位置的像素值一般相差不会过大，因此，在计算中值过程中，与周围灰度值相差较大的坏点区域就会被前帧或后帧图像中的像素所替换，从而消除中间帧图像内的坏点。
当然，在本公开的其他一些实施例中，也不排除采用其他滤波方法，例如均值滤波对待处理图像进行去坏点处理。
如图8所示，图8中的(a)为具有坏点的待处理视频图像，(b)为坏点修复图像，从图8中可以看出通过中值滤波可以利用前后帧图像的内容填补中间帧图像内的坏点，但也会将前后帧的错误信息引入到中间帧内，产生运动伪影，图8中，(a)右上角小图为坏点区域的放大图，(b)右上角小图为坏点修复区域的放大图，右下角小图为运动伪影部分的放大图。因此，本公开实施例中，还需要对坏点修复图像进行去伪影处理以消除滤波产生的伪影。
(2)去伪影
本公开实施例中，可选的，根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像对所述坏点修复图像进行去伪影处理得到伪影修复图像包括：
对所述坏点修复图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像分别进行N 3-1次下采样，得到N 3-1种分辨率的下采样图像，将N 3种分辨率的图像输入至多尺度级联网络中进行去伪影处理得到伪影修复图像，所述N 3种分辨率的图像包括：所述坏点修复图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像以及所述N 3-1种分辨率的下采样图像，所述多尺度级联网络包括N 3个级联的子网络，所述N个级联的子网络所处理的图像分别基于所述N 3种分辨率的图像生成；
其中，N 3为大于或等于2的正整数。
进一步可选的，将N 3种分辨率的图像输入至多尺度级联网络中进行去伪影处理得到伪影修复图像包括伪影修复图像：
针对所述N 3个级联的子网络中的第一个子网络：将所述坏点修复图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像分别进行A倍下采样，得到N 1帧第一下采样图像，将所述N 1帧第一下采样图像分别与自身进行拼接，得到第一拼接后图像，将所述第一拼接后图像输入到第一个子网络，得到第一输出图像；
针对第一个子网络和最后一个子网络之间的中间子网络：将前一个子网络的输出图像进行上采样得到第一上采样图像，将所述坏点修复图像、所述待处理视频图像之前的至少一帧视频图像以和之后的至少一帧视频图像分别进行B倍下采样，得到N 1帧第二下采样图像，所述第二下采样图像和所述第一上采样图像的尺度相同，将两组图像进行拼接得到第二拼接后图像，将所述第二拼接后图像输入到所述中间子网络，得到第二输出图像，其中，所述两组图像其中一组为N 1帧第二下采样图像，另一组包括：所述N1帧第二下采样图像中除所述坏点修复图像对应的下采样图像之外的其他下采样图像，以及，所述第一上采样图像；
针对最后一个子网络：将前一个子网络的输出图像进行上采样得到第二 上采样图像，所述第二上采样图像与所述待处理视频图像的尺度相同，将两组图像进行拼接得到第三拼接后图像，将所述第三拼接后图像输入到所述最后一个子网络，得到所述伪影修复图像，其中，所述两组图像其中一组为所述N 1帧视频图像，另一组包括：所述N 1帧视频图像中除所述坏点修复图像之外的其他图像，以及，所述第二上采样图像。
本公开实施例中，可选的，上述子网络可以是SRN中提出的encoder-decoder(编码解码)resblock网络结构。当然，在本公开的其他一些实施例中，也可以采用其他网络，本公开实施例并不进行限定。
为了提升网络的效果，本公开实施例中，可选的，所述N 3个级联的子网络的结构相同，参数不同。
本公开实施例中，可选的，N 3等于3，A等于4，B等于2。
下面举例进行说明。
如图9所示，多尺度级联网络由三个相同结构的子网络级联构成，三个子网络分别处理不同尺度(即分辨率)的输入。
多尺度级联网络的输入为连续的三帧图像I t-1、I′ t和I t+1，请参考图10，其中(b)为I′ t，为经过上述处理后的带伪影的坏点修复图像，(a)为I t-1，是待处理视频图像I t的前一帧图像，(c)为I t+1，是待处理视频图像I t的后一帧图像，多尺度级联网络输出与I′ t对应的去除了伪影的图像
Figure PCTCN2021082072-appb-000001
多尺度级联网络的运算步骤如下：
1)将输入的三帧图像I t-1、I′ t和I t+1分别下采样4倍，得到分辨率缩小了4倍的第一下采样图像
Figure PCTCN2021082072-appb-000002
和
Figure PCTCN2021082072-appb-000003
将三帧第一下采样图像分别与其自身进行拼接后，得到第一拼接后图像并输入到网络1中，网络1输出一帧第一输出图像
Figure PCTCN2021082072-appb-000004
上述拼接是指在第四维度拼接。每一幅图像是一个三维数组，数组结构为H×W×C，即height，width，channel，第四维度即是channel维度。
本公开实施例中，可选的，采用通过双三次插值的方法对I t-1、I′ t和I t+1分别下采样4倍。当然，也可以采用其他方法进行下采样。
图像下采样后，其伪影也会随之缩小，更有利于网络对伪影的消除和修复，所以三个子网络的输入图像尺寸分别为原始输入图像尺寸的1/4，1/2， 和1，这里要将图像输入到第一个子网络里，所以做了4倍下采样。
2)将网络1的输出
Figure PCTCN2021082072-appb-000005
上采样2倍，得到第一上采样图像
Figure PCTCN2021082072-appb-000006
并将图像I t-1、I′ t和I t+1分别下采样2倍得到第二下采样图像
Figure PCTCN2021082072-appb-000007
和
Figure PCTCN2021082072-appb-000008
将这三帧第二下采样图像作为一组输入，另外，将
Figure PCTCN2021082072-appb-000009
和
Figure PCTCN2021082072-appb-000010
作为一组输入，将两组输入进行拼接后，得到第二拼接后图像并输入到网络2中，网络2输出一帧第二输出图像
Figure PCTCN2021082072-appb-000011
本公开实施例中，可选的，采用通过双三次插值的方法将网络1的输出
Figure PCTCN2021082072-appb-000012
上采样2倍。当然，也可以采用其他方法进行上采样。
本公开实施例中，可选的，采用通过双三次插值的方法将图像I t-1、I′ t和I t+1分别下采样2倍。当然，也可以采用其他方法进行下采样。
3)将网络2的输出
Figure PCTCN2021082072-appb-000013
上采样2倍，得到第二上采样图像
Figure PCTCN2021082072-appb-000014
将图像I t-1、I′ t和I t+1作为一组输入，另外，将I t-1、
Figure PCTCN2021082072-appb-000015
和I t+1作为一组输入，将两组输入进行拼接后，得到第三串联图像并输入到网络3中，网络3输出一帧图像
Figure PCTCN2021082072-appb-000016
即为网络的最终结果
Figure PCTCN2021082072-appb-000017
本公开实施例中，可选的，采用通过双三次插值的方法将网络2的输出
Figure PCTCN2021082072-appb-000018
上采样2倍。当然，也可以采用其他方法进行上采样。
请参考图11，图11中的(a)为原始的待处理视频图像的部分区域，(b)为经过滤波处理之后的坏点修复图像的部分区域，坏点修复图像中包含因滤波导致的运动伪影，(c)为多尺度级联网络输出的修复图像的部分区域，从图11中可以看出，运动伪影被消除。
本公开实施例中，可选的，网络1、网络2和网络3可以是SRN中提出的encoder-decoder(编码解码)resblock网络结构，为了提升网络的效果，本公开实施例中这三个子网络的参数不共享。
本公开实施例中，可选的，每个所述子网络包括多个3D卷积层、多个3的反卷积层和多个3D平均池化层。
请参考图12，图12为本公开实施例中的子网络的结构示意图，该子网络为encoder-decoder resblock网络，其中，Conv3d(n32f5s1)代表的是滤波器个数(n)为32、滤波器尺寸(f)为(1×5×5)、滤波步幅(s)为1的3D卷积层；DeConv3d(n32f5s2)表示滤波器个数(n)为32、滤波器尺寸(f)为(1×5×5)、滤波 器步幅(s)为2的3D反卷积层；AvgPool3D(f2s1)表示核尺寸(f)为(2×1×1)、步幅(s)为1的3D平均池化层；浮在箭头上方的(B,3,H,W,32)当前层输出的特征图的尺寸。这里的(B,3,H,W,32)指的是网络每一层输出的中间结果的尺寸，网络每一层输出是一个5维数组，(B,3,H,W,32)指的是数组的结构，即B×3×H×W×32，B是Batch_size。
如图12所示，子网络共含有16个3D卷积层和2个3D反卷积层，相邻帧之间的信息通过3D平均池化层进行融合，网络的第3个卷积层和第1个反卷积层的输出特征图，各自通过3D平均池化层进行信息融合后，逐像素相加求和，并将结果作为第14个卷积层的输入；网络的第6个卷积层和第2个反卷积层的输出特征图，各自通过3D平均池化层进行信息融合后逐像素相加，将它们的和作为第12个卷积层的输入，最后，网络通过一个滤波器个数为1的卷积层输出最终去除了伪影的结果图像。
本公开实施例中，可选的，多尺度级联网络采用以下训练方法得到：
步骤一：获取连续的N 1帧训练图像，所述N 1帧训练图像中包括待处理训练图像、所述待处理训练图像之前的至少一帧训练图像和之后的至少一帧训练图像；
步骤二：根据所述N 1帧训练图像对所述待处理训练图像进行滤波处理得到第一训练图像；
步骤三：根据所述第一训练图像、所述待处理训练图像之前的至少一帧训练图像和之后的至少一帧训练图像，对待训练的多尺度级联网络进行训练，得到训练后的多尺度级联网络。
本公开实施例中，可选的，对待训练的多尺度级联网络进行训练时，采用的总损失包括以下至少一项：图像内容损失、颜色(Color)损失、边缘损失和感知损失。
其中，图像内容损失主要用来提升输出图像的保真度。本公开实施例中，可选的，图像内容损失可以采用L1损失函数或者均方误差损失函数等计算。
可选的，所述L 1损失采用如下公式计算：
Figure PCTCN2021082072-appb-000019
其中，l content为L1损失，
Figure PCTCN2021082072-appb-000020
为所述去伪影训练图像，y i为所述第一训练图像，n为一个Batch里的图像数量。
颜色损失函数通过对去伪影训练图像与目标图像的纹理、内容进行高斯模糊处理，仅仅保存图像的颜色信息，来实现图像颜色的校正。本公开实施例中，可选的，所述颜色损失采用如下公式计算：
Figure PCTCN2021082072-appb-000021
其中，l color为所述颜色损失，
Figure PCTCN2021082072-appb-000022
为所述去伪影训练图像，y i为所述第一训练图像，n为一个Batch里的图像数量，Blur(x)为高斯模糊函数。
边缘损失函数主要通过计算去伪影训练图像与目标图像在边缘信息上的差值来提升去伪影训练图像轮廓信息的准确性，本公开实施例中，可以使用HED(Holistically-Nested Network)网络来提取图像的边缘信息。本公开实施例中，可选的，所述边缘损失采用如下公式计算：
Figure PCTCN2021082072-appb-000023
其中，l edge为所述边缘损失，
Figure PCTCN2021082072-appb-000024
为所述去伪影训练图像，y i为所述第一训练图像，n为一个Batch里的图像数量，H j(x)代表HED网络第j层提取的图像边缘图。
本公开实施例中，使用VGG网络提取的高层特征来计算感知损失函数，以衡量输出图像与目标图像在语义层面上的差别，可选的，所述感知损失采用如下公式计算：
Figure PCTCN2021082072-appb-000025
其中，l feature为所述感知损失，
Figure PCTCN2021082072-appb-000026
为所述去伪影训练图像，y i为所述第一训练图像，n为一个Batch里的图像数量，
Figure PCTCN2021082072-appb-000027
表示VGG网络第j层提取的图像特征图。
本公开实施例中，可选的，所述总损失等于图像内容损失、颜色损失、边缘损失和感知损失的加权和。
可选的，所述总损失采用以下公式计算得到：
L＝l content+λ 1l color+λ 2l edge+λ 3l feature
其中，λ 1＝0.5，λ 2＝10 -2，λ 3＝10 -4。当然，在本公开的其他一些实施例中，各损失的权重可以不限于此。
本公开实施例中，可以使用2020-AIM竞赛的视频时域超分赛道提供的训练数据进行训练，该训练集共包含240组帧序列，每组帧序列含有181幅1280×720的清晰图像，使用该训练数据集的原因有如下3点：
a)该数据集中每组181张图像均为同一个场景下拍摄，每次训练取同一场景内的图片，可以避免不同场景的图像内容差别过大，造成干扰；
b)该训练数据集用于视频时域超分赛道的训练，同场景下的图像内物体在相邻帧之间存在适当的运动，符合本公开模拟训练数据需要产生伪影的要求；
c)该训练数据集中的图像相对干净，不含噪声，且分辨率较大，有利于网络生成更加清晰的图像。
由于本公开实施例的待处理视频图像在输入到多尺度级联网络之前，已对坏点进行了修复，网络训练的主要目的是去除滤波产生的伪影，因此在生成仿真数据时，仅对训练数据集进行相同的滤波操作，无需模拟坏点的生成。
本公开实施例的网络模型可以在ubuntu16.04系统下进行训练，使用python语言编译，基于tensorflow深度学习框架以及opencv、ffmpeg等开源的图像、视频处理工具。
本公开实施例中，可选的，根据所述第一训练图像、所述待处理训练图像之前的至少一帧训练图像和之后的至少一帧训练图像，对待训练的多尺度级联网络进行训练包括：
在所述第一训练图像上随机裁剪出一个图像块，在所述待处理训练图像之前的至少一帧训练图像和之后的至少一帧训练图像的相同位置处分别裁剪出一个图像块，得到N 1帧图像块；
将所述N 1帧图像块输入到待训练多尺度级联网络进行训练。
本公开实施例中，可以使用Adam优化算法对网络参数进行优化，设置Adam算法的学习率为10-4，网络训练过程中，依次从训练数据集中取连续三帧训练图像做中值滤波的预处理，然后在中间帧内随机裁剪一个512×512 大小的图像块，并在前后帧相同位置裁剪对应的图像块作为网络迭代一次的输入，当训练数据集内所有图像均被读取一次后，为一轮(epoch)迭代完成。每完成10个epoch(一个Epoch就是将所有训练样本训练一次的过程)，Adam的学习率降为原来的0.8倍。
本公开实施例中，对裁剪后的图像块做下采样处理，这是扩增数据集的一种方法。也就是说，可以对同一张图像多次随机裁剪多个图像块，用于训练网络，从而增加用于训练网络的图像的数量。随机裁剪使得同一图像的不同位置可以被采用。另外，裁剪成图像块，也可以降低图像的分辨率，减少网络处理的数据量，提高处理速度。
(3)后处理
本公开实施例中，经过去坏点处理和去伪影处理后的图像既修复了图像中的坏点，又去除了物体运动带来的伪影，但网络输出的图像在整体清晰度上，与原始的待处理视频图像仍存在一定的差异，因而，通过对待处理视频图像、坏点修复图像和多尺度级联网络修复的图像做滤波操作，将原始待处理视频图像像中的细节补充到修复的图像中来，以提升修复图像的清晰度。
因此，本公开实施例中，可选的，得到伪影修复图像之后还包括：根据所述待处理视频图像和所述坏点修复图像，对所述伪影修复图像进行滤波处理，得到输出图像伪影修复图像。
进一步可选的，根据所述待处理视频图像和所述坏点修复图像，对所述伪影修复图像进行中值滤波处理，得到输出图像。
请参考图13，图13中的(a)为多尺度级联网络的输出图像，(b)为后处理之后的图像，从图13中可以看出，后处理之后的图像的清晰度明显高于多尺度级联网络的输出图像的清晰度。
三、去噪
请参考图14，图14为本公开实施例的去噪步骤的流程示意图，该去噪步骤包括：
步骤141：采用去噪网络对待处理视频图像进行去噪处理；其中，所述去噪网络采用以下训练方法得到：根据连续的N 2帧视频图像得到目标非运动掩膜，所述N 2帧视频图像包括待去噪视频图像；根据所述N 2帧视频图像和所 述目标非运动掩膜，对待训练的去噪网络进行训练，得到所述去噪网络。
本公开实施例中，在训练去噪网络时，采用的是盲去噪技术，即不需要成对的训练数据集，只需要输入需要进行去噪的视频帧序列，利用非运动掩膜，只对非运动数据进行时域去噪作为参考图，适用于无清晰参考图的去噪网络训练。同时，适用于各种不同的视频噪声去除，不需要考虑噪声类型，只需要使用部分视频帧学习去噪网络即可。
本公开实施例中，可选的，根据所述N 2帧视频图像和所述目标非运动掩膜，对待训练的去噪网络进行训练，得到所述去噪网络包括：
步骤151：根据所述N 2帧视频图像和所述目标非运动掩膜，得到参考图；
参考图相当于待去噪视频图像的真值，即没有噪声的图像。
步骤152：将所述待去噪视频图像输入至待训练去噪网络中，得到第一去噪图；
步骤153：根据所述第一去噪图和所述目标非运动掩膜，得到第二去噪图；
步骤154：根据所述参考图和所述第二去噪图确定所述待训练去噪网络的损失函数，根据所述损失函数调整所述待训练去噪网络的参数，得到所述去噪网络。
本公开实施例中，可以采用光流法得到视频图像的非运动掩膜，下面将进行详细说明。
光流(optical flow)是空间运动物体在观察成像平面上的像素运动的瞬时速度。光流法是利用图像序列中像素在时间域上的变化以及相邻帧之间的相关性来找到上一帧跟当前帧之间存在的对应关系，从而计算出相邻帧之间物体的运动信息的一种方法。
在得到非运动掩膜之前，首先需要得到运动掩膜。运动掩膜是指图像中的运动信息。而非运动掩膜是指图像中除了运动掩膜之外的信息，即非运动信息。
请参考图15，图15为本公开实施例的利用光流网络得到运动掩膜的流程示意图，将两帧视频图像F和Fr输入到光流网络中，得到光流图，再通过光流图得到运动掩膜。
本公开实施例中，对于光流网络的选择不做限制，可以是任意的目前已经开源的光流网络，如flownet、flownet2，也可以是传统的光流算法(不是深度学习)，如TV-L1flow，只需要利用光流算法得到光流图即可。
请参考图16，图16为本公开实施例的利用光流网络得到运动掩膜的具体流程示意图，假设输入光流网络的两帧图像的大小为(720,576)，光流网络输出两帧光流图，此时，光流图的大小为(720,576)，两帧光流图分别是表示连续的两帧图像中的上下运动信息的第一光流图和表示连续的两帧图像中的左右运动信息的第二光流图，将所述第一光流图的后X-X1行与前X-X1行做减法运算(图中示例是以后三行减去前三行)，得到第一差值图；将所述第二光流图的后Y-Y1列与前Y-Y1列做减法运算(图中示例是以后三列减去前三列)，得到第二差值图，将第一差值图最后X1行补0，将第二差值图最后Y1列补0，从而得到和光流图同等大小的图。之后将两个差值图相加。其中，||符号表示绝对值，>T表示绝对值运算后，对应像素位置数值大于T的像素，赋值为1，其余小于等于T的值赋值为0，T为预设阈值。经过阈值处理后得到二值化图像。可选的，还可以对该二值化图像进行膨胀运算。膨胀运算的具体方法是，找到二值化图像中像素为1的值，并根据膨胀核，将核中为1的位置对应的二值化图像中的像素位置设置为1。本公开实施例中的图16中给出的3*3的膨胀核的一个示例，即，如果二值化图像中的某一行某一列的像素位置的数值为1，则将该像素位置上下左右的像素位置均设为1。除此之外还可以是其他的膨胀核，只需要能够起到膨胀作用即可。此处膨胀运算的目的是为了扩大运动掩膜的范围，尽可能的将所有运动位置都标记出来，减小误差。
通过上述过程可以得到视频图像中的运动掩膜Mask_move，且该掩膜数值是二值化的，即数值非0即1，数值为1的地方代表有运动的地方，数值为0的地方代表相对没有运动的地方。则非运动掩膜的计算方式如下：Mask_static＝1-Mask_move。
下面结合上述目标非运动掩膜的确定方法对去噪网络的训练过程进行说明。
本公开实施例中，可选的，确定目标非运动掩膜的方法包括：
步骤181：将所述N 2帧视频图像中每一帧第一视频图像分别与所述待去噪视频图像组成样本对输入至光流网络中，得到表示上下运动信息的第一光流图和表示左右运动信息的第二光流图，所述第一视频图像为所述N 2帧视频图像中除所述待去噪视频图像之外的其他视频图像，所述第一光流图和所述第二光流图的分辨率为X*Y；
步骤182：根据所述第一光流图和所述第二光流图，计算每一帧第一视频图像与所述待去噪视频图像的运动掩膜，得到N 2-1个运动掩膜；
假设N 2帧视频图像分别为F1、F2、F3、F4、F5，F3为当前待去噪视频图像，则将分别将F1和F3组成样本对输入到光流网络，得到运动掩膜Mask_move1，将F2和F3组成样本对输入到光流网络，得到运动掩膜Mask_move2，将F4和F3组成样本对输入到光流网络，得到运动掩膜Mask_move4，将F5和F3组成样本对输入到光流网络，得到运动掩膜Mask_mov5。
步骤183：根据所述N 2-1个运动掩膜得到目标非运动掩膜。
本公开实施例中，可选的，根据所述N 2-1个运动掩膜得到目标非运动掩膜包括：
根据所述N 2-1个运动掩膜得到N 2-1个非运动掩膜；非运动掩膜＝1-运动掩膜；
将所述N 2-1个目标非运动掩膜相乘，得到所述目标非运动掩膜。
本公开实施例中，可选的，根据所述第一光流图和所述第二光流图，计算每一帧第一视频图像与所述待去噪视频图像的运动掩膜包括：
步骤191：将所述第一光流图的后X-X1行与前X-X1行做减法运算，得到第一差值图，并将第一差值图最后X1行补0，得到处理后的第一差值图；
X1为小于X的正整数。例如，X1可以取1，即将后X-1行与前X-1行做减法运算，得到第一差值图。
步骤192：将所述第二光流图的后Y-Y1列与前Y-Y1列做减法运算，得到第二差值图，并将第二差值图最后Y1列补0，得到处理后的第二差值图；
Y1为小于Y的正整数。例如，Y1可以取1，即将后Y-1列与前Y-1列做减法运算，得到第二差值图。
步骤193：将所述处理后的第一差值图和所述处理后的第二差值图相加，得到第三差值图；
步骤194：将所述第三差值图中绝对值大于预设阈值的像素赋值为1，小于所述预设阈值的像素赋值为0，得到二值图；
步骤195：根据所述二值图得到运动掩膜。
本公开实施例中，可选的，根据所述二值图得到运动掩膜包括：将所述二值图进行膨胀运算，得到所述运动掩膜。
本公开实施例中，可选的，根据所述N 2帧视频图像和所述目标非运动掩膜，得到参考图包括：
将所述N 2帧视频图像分别与所述目标非运动掩膜相乘，得到N 2个乘积；
将所述N 2个乘积相加并取均值，得到所述参考图。
可选的，也可以是将所述N 2个乘积加权相加并取均值，得到所述参考图。权重可以根据需要设置。
本公开实施例中，可选的，根据所述第一去噪图和所述目标非运动掩膜，得到第二去噪图包括：将所述第一去噪图于所述目标非运动掩膜相乘，得到所述第二去噪图。
本公开实施例中，可选的，所述N 2为5～9。
下面以N 2为5为例，对上述去噪网络的训练方法举例进行说明。
请参考图17，图17中，F1、F2、F3、F4、F5为连续的5帧视频图像，F3为当前待去噪视频图像，DN3为去噪网络输出的去噪图，与F3对应，M为目标非运动掩膜，*代表对应像素位置相乘，Ref3代表参考图，可以认为是DN3对应的真值。
首先，按照图16中的流程，分别将F1和F3输入到光流网络，得到运动掩膜Mask_move1，将F2和F3输入到光流网络，得到运动掩膜Mask_move2，将F4和F3输入到光流网络，得到运动掩膜Mask_move4，将F5和F3输入到光流网络，得到运动掩膜Mask_mov5，从而计算出非运动掩膜Mask_static1、Mask_static2、Mask_static4、Mask_static5。最终的M为4个非运动掩膜的乘积，即保留所有非运动掩膜中的非运动部分，去掉运动部分。
得到参考图的方法为：将F1、F2、F3、F4、F5分别与M相乘后，相加 取均值。这是一种时域去噪的原理，利用连续帧间有效信息分布相同，但是噪声是随机无规律分布的原理，多帧相加去均值，可以保留有效信息但是将随机噪声的影响进行抵消。计算非运动掩膜的目的是为了去噪后图像和参考图相对应位置像素有效信息相同，如果没有非运动掩膜步骤，直接多帧相加去均值，非运动位置有效信息可以保留，但是运动位置会产生严重的伪影，破坏了原有的有效信息，不能作为参考图进行训练。利用非运动掩膜，生成的参考图中仅保留非运动位置，同时去噪图中也保留相对应位置的像素，形成训练数据对，进行训练。
得到去噪图的方法为：将F3，或者，F3及其相邻的视频图像，输入到去噪网络中，得到第一去噪图，将第一去噪图与M相乘，得到第二去噪图(即DN3)。
本公开实施例中，去噪网络可以是任意的去噪网络。请参考图18，图18为本公开实施例的一种去噪网络的实现方法示意图，该去噪网络的输入为5帧连续的视频图像。该去噪网络包括：多个串联的滤波器(filter)，每个滤波器包括多个串联的卷积核(图18中的竖条)。图18所示的实施例中，每个滤波器包括四个串联的卷积核，当然，在本公开的其他一些实施例中，滤波器中的卷积核的个数也不限于4个。本公开实施例中，多个串联的滤波器中，每两个滤波器的分辨率相同，除最后一个滤波器之外，其余每个滤波器的输出作为下一个滤波器以及与其具有相同分辨率的滤波器的输入。图18所示的实施例中，去噪网络包括6个串联的滤波器，其中，第一个滤波器与第六个滤波器的分辨率相同，第二个滤波器与第五个滤波器的分辨率相同，第三个滤波器与第四个滤波器的分辨率相同，第一个滤波器的输出作为第二个滤波器和第六个滤波器(与第一个滤波器的分辨率相同)的输入，第二个滤波器的输出作为第三个滤波器和第五个滤波器(与第二个滤波器的分辨率相同)的输入，第三个滤波器的输出作为第四个滤波器(与第三个滤波器的分辨率相同)的输入。
本公开实施例中，去噪网络经过训练后，保存的参数可以作为下一次视频降噪的初始化参数，这样只需要在新的视频约100帧左右即可完成一次新的训练。
请参考图19-图22，图19为待处理视频图像，图20为图19中的待处理视频图像中的部分区域的放大图，图21为图19中的待处理视频图像对应的目标非运动掩膜M，图22为采用本公开实施例的去噪网络对待处理视频图像进行去噪处理后的图像。从对比结果可以看出，去噪效果明显。
四、偏色矫正
通过数码相机等数字成像设备采集到的彩色数字图像，是由红(R)、绿(G)、蓝(B)三个通道合成得到的。然而，数字成像设备在成像时，往往因为光照、感光元件等因素的影响，拍摄到的图像可能与原始景物有一定颜色偏差，被称为偏色。通常情况下，偏色图像表现为图像R、G、B三通道中的一个或几个通道的平均像素值明显偏高。偏色带来的颜色失真严重影响图像的视觉效果，因此，数字图像的偏色校正是数字图像处理领域的一个重要问题。在处理老旧照片、影像资料时，由于年份久远、保存等问题，经常需要处理偏色问题。
为了解决视频图像的偏色问题，请参考图23，本公开实施例的偏色矫正步骤包括：
步骤191：确定待处理视频图像的RGB各通道的目标偏色值；
步骤192：根据所述目标偏色值，对所述待处理视频图像进行色彩平衡调整，得到第一矫正图像；
步骤193：根据参考图像，对所述第一矫正图像进行颜色迁移，得到第二矫正图像。
所述参考图像为输入的基本无偏色的图像。
本公开实施例中，首先自动估算图像偏色程度，对待处理视频图像进行色彩平衡调整对偏色进行初步纠正，然后根据参考图像，对色彩平衡调整之后的图像进行颜色迁移处理，进一步调整偏色，使得偏色校正结果更符合理想的预期。
本公开实施例中，可选的，确定待处理视频图像的RGB各通道的目标偏色值包括：
步骤201：获取所述待处理视频图像的RGB各通道的均值；
RGB三通道的均值(avgR,avgG,avgB)的计算方法为：将待处理视频图 像中的所有R子像素的灰度值相加，然后求均值得到avgR；将待处理视频图像中的所有G子像素的灰度值相加，然后求均值得到avgG；将待处理视频图像中的所有B子像素的灰度值相加，然后求均值得到avgB。
步骤202：将所述RGB各通道的均值转换到Lab颜色空间，得到所述RGB各通道的均值对应的Lab空间各颜色分量(l,a,b)；
Lab是一种设备无关的颜色系统，也是一种基于生理特征的颜色系统。这也就意味着，它是用数字化的方法来描述人的视觉感应。Lab颜色空间中的L分量用于表示像素的亮度，取值范围是[0,100]，表示从纯黑到纯白；a表示从红色到绿色的范围，取值范围是[127,-128]；b表示从黄色到蓝色的范围，取值范围是[127,-128]。
通常来说，正常不偏色的图像，其均值的a和b值应该接近于0，如果a>0,则图像偏红，否则偏绿；如果b>0，则偏黄，否则偏蓝。
步骤203：根据所述Lab空间各颜色分量(l,a,b)，确定所述RGB各通道的均值对应的偏色程度(l,0-a,0-b)；
根据灰度世界假设，不偏色的图像，其均值的颜色分量a和b应该趋近于0，因此所述RGB各通道的均值对应的偏色程度为(l,0-a,0-b)。
步骤204：将所述偏色程度(l,0-a,0-b)转换到RGB颜色空间，得到RGB各通道的目标偏色值。
RGB颜色空间不能直接转换为Lab颜色空间，本公开实施例中，需要借助XYZ颜色空间，把RGB颜色空间转换到XYZ颜色空间，之后再把XYZ颜色空间转换到Lab颜色空间。
即，将所述RGB各通道的均值转换到Lab颜色空间包括：将所述RGB各通道的均值转换到XYZ颜色空间，得到XYZ颜色空间的均值；将XYZ颜色空间的均值转换到Lab颜色空间；
同样的，将所述偏色程度(l,0-a,0-b)转换到RGB颜色空间包括：将所述偏色程度转换到XYZ颜色空间，得到XYZ颜色空间的偏色程度；将XYZ颜色空间的偏色程度转换到RGB颜色空间。
本公开实施例中，RGB和XYZ的相互转换关系可以如下所示：
Figure PCTCN2021082072-appb-000028
Figure PCTCN2021082072-appb-000029
XYZ和Lab的转换关系可以如下所示：
Figure PCTCN2021082072-appb-000030
Figure PCTCN2021082072-appb-000031
Figure PCTCN2021082072-appb-000032
Figure PCTCN2021082072-appb-000033
其中，X n，Y n，Z n一般默认是0.95047，1.0，1.08883。
下面对色彩平衡调整的方法进行说明。
白平衡概念是定义一个区域，以这个区域为标准，认为它是白色(准确说是18度灰)，其他区域的色彩，是在这个标准的基础上进行色彩的偏移。色彩平衡的调整原理就是增加或降低其对比色来消除画面偏色。
本公开实施例中，可选的，根据所述目标偏色值，对所述待处理视频图像进行色彩平衡调整，得到第一矫正图像包括：
根据RGB各通道的目标偏色值，对所述待处理视频图像进行高光函数处理、阴影函数处理和中间调函数处理中的至少一项色彩平衡调整处理；
其中，所述高光函数和阴影函数为线性函数，所述中间调函数为指数函数。
本公开实施例中，可选的，所述高光函数为：y＝a(v)*x+b(v)；
所述阴影函数为：y＝c(v)*x+d(v)；
所述中间调函数为：y＝x f(v)；
y为第一矫正图像，x为所述待处理视频图像，v根据RGB各通道的目标偏色值确定，f(v),a(v),b(v),c(v),d(v)是v的函数。
本公开实施例中，进行中间调调节时，单独改动任何一个参数，都会引 起本通道朝向一个方向变化，同时另外两个通道朝向另一个方向变化。举例来说，对R通道参数+50，则R通道像素值会变大，而G,B通道像素值会变小(G-50，B-50)，两边的变化方向完全相反。
进行高光调节时，对于正向的调整，例如R通道+50，算法效果是只增加R通道的值，其他两个通道不变；对于负向的调整，例如R通道-50，算法效果是R通道不变，其余两个通道值增加。
进行阴影调节时，对于正向的调整，例如R通道+50，算法效果是R通道的值不变，其他两个通道值减少；对于负向的调整，例如R通道-50，算法效果是R通道值减少，其余两个通道值不变。
进一步可选的，f(v)＝e -v。
进一步可选的，
Figure PCTCN2021082072-appb-000034
b(v)＝0。
进一步可选的，
Figure PCTCN2021082072-appb-000035
等量的RGB三色混合，得到的是不同明度的灰色。如果将ΔR,ΔGd,ΔB都改变相同的数值，那么理论上将不会对原图做任何变化(加减灰色，颜色不应当有变化，而明度，我们需要保持明度，所以也不应该有变化)。举个例子，ΔR,ΔGd,ΔB为(+20,+35,+15)的效果等价于(+5,+20,0)的效果，也等价于(0,+15,-5)的效果。因此为了减少总的变化量，将满足条件min d|ΔR-d|+|ΔG-d|+|ΔB-d|时的(ΔR-d,ΔG-d,ΔB-d)，作为最终的目标偏色值，然后组合三个目标偏色值，得到v。
进一步可选的，对于R通道，v＝(ΔR-d)-(ΔG-d)-(ΔB-d)；
对于G通道，v＝(ΔG-d)-(ΔR-d)-(ΔB-d)；
对于B通道，v＝(ΔB-d)-(ΔR-d)-(ΔG-d)；
其中，ΔR、ΔG和ΔB为RGB各通道的目标偏色值，d为将ΔR、ΔG、ΔB按照大小进行排序后的中间值，举例来说，ΔR为10，ΔG为15，ΔB为5，则d＝10。
下面对本公开实施例中的颜色迁移的方法进行说明。
本公开实施例中，可选的，根据参考图像，对所述第一矫正图像进行颜色迁移，得到第二矫正图像包括：
步骤211：将所述参考图像和所述第一矫正图像转换到Lab颜色空间；
转换方法可以参见上述RGB与Lab的转换过程。
步骤212：在Lab颜色空间，确定所述参考图像和所述第一矫正图像的均值和标准差；
步骤213：根据所述参考图像和所述第一矫正图像的均值和标准差，确定Lab颜色空间中的第k个通道的颜色迁移结果；
步骤214：将所述颜色迁移结果转换到RGB颜色空间，得到所述第二矫正图像。
可选的，所述颜色迁移结果的计算方法如下：
Figure PCTCN2021082072-appb-000036
其中，I k为Lab颜色空间的第k个通道的颜色迁移结果，t为所述参考图像，S为所述第一矫正图像，
Figure PCTCN2021082072-appb-000037
表示所述第一矫正图像的第k个通道的均值，
Figure PCTCN2021082072-appb-000038
表示所述第一矫正图像的第k个通道的标准差，
Figure PCTCN2021082072-appb-000039
表示所述参考图像的第k个通道的均值，
Figure PCTCN2021082072-appb-000040
表示所述参考图像的第k个通道的标准差。
经过实验发现，颜色迁移过程中，亮度通道迁移会引起图像亮度变化，尤其是有大面积同一颜色的图像，亮度通道改变会带来视觉上的改变。因此，本公开实施例中，只迁移ab通道，即第k个通道为a和b通道中的至少一个，从而偏色矫正的同时保持图像亮度不变。
请参考图24，本公开实施例还提供一种图像处理装置200，包括：
处理模块201，所述处理模块包括以下模块中的至少之一：
划痕修复子模块2011：对待处理视频图像进行去划痕处理，得到第一图像；对所述待处理视频图像和所述第一图像进行差异运算得到差异图像；对所述差异图像进行处理得到只保留划痕的划痕图像；根据所述待处理视频图像和所述划痕图像得到划痕修复图像；
坏点修复子模块2012：获取连续的N 1帧视频图像，所述N 1帧视频图像中包括待处理视频图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像，其中，N 1为大于或等于3的正整数；根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像，对所述待处理视频图像进行滤波处理得到坏点修复图像；根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像对所述坏点修复图像进行去伪影处理得到伪影修复图像；
去噪子模块2013：采用去噪网络对待处理视频图像进行去噪处理，所述去噪网络采用以下训练方法得到：根据连续的N 2帧视频图像得到目标非运动掩膜，所述N 2帧视频图像包括待去噪视频图像；根据所述N 2帧视频图像和所述目标非运动掩膜，对待训练的去噪网络进行训练，得到所述去噪网络；
偏色矫正子模块2014：确定待处理视频图像的RGB各通道的目标偏色值；根据所述目标偏色值，对所述待处理视频图像进行色彩平衡调整，得到第一矫正图像；根据参考图像，对所述第一矫正图像进行颜色迁移，得到第二矫正图像。
本公开实施例中，可选的，划痕修复子模块对待处理视频图像进行去划痕处理包括：根据滤波器类型和所述待处理视频图像中的划痕类型中的至少一个，对所述待处理视频图像进行中值滤波处理，得到去划痕图像。
可选的，根据滤波器类型和所述待处理视频图像中的划痕类型中的至少一个，对所述待处理视频图像进行中值滤波处理包括：
根据所述待处理视频图像中的划痕类型，选取对应的滤波器类型，对所述待处理视频图像进行中值滤波处理，其中：
当所述待处理视频图像中的划痕为垂直划痕时，采用水平方向的中值滤波器对所述待处理视频图像进行中值滤波；
当所述待处理视频图像中的划痕为水平划痕时，采用对垂直方向的中值滤波器所述待处理视频图像进行中值滤波。
本公开实施例中，可选的，根据滤波器类型和所述待处理视频图像中的划痕类型中的至少一个，对所述待处理视频图像进行中值滤波处理包括：
根据滤波器类型和所述待处理视频图像中的划痕类型，对所述待处理视频图像不同的预处理，对预处理之后的待处理视频图像进行中值滤波处理，其中：
当采用的是水平方向的中值滤波器，且所述待处理视频图像中的划痕为非垂直划痕时，将所述待处理视频图像进行旋转，使得划痕转换为垂直划痕；
当采用的是垂直方向的中值滤波器，且所述视频图像中的划痕为非水平划痕时，将所述待处理视频图像进行旋转，使得划痕转换为水平划痕。
本公开实施例中，可选的，对待处理视频图像进行去划痕处理包括：采 用尺寸为1×k和/或k×1的中值滤波器对待处理视频图像进行中值滤波；
划痕修复子模块还用于：将所述中值滤波器的k从预设值开始依次增大，对所述待处理视频图像进行中值滤波，得到第二图像；根据所述第二图像的滤波效果确定k的最终值。
本公开实施例中，可选的，对所述待处理视频图像和所述第一图像进行差异运算得到差异图像包括：对所述待处理视频图像和所述第一图像进行差异运算，得到第一差异图像和/或第二差异图像，其中，所述第一差异图像由所述待处理视频图像减去所述第一图像得到，所述第二差异图像由所述第一图像减去所述待处理视频图像得到；
对所述差异图像进行处理得到只保留划痕的划痕图像包括：对所述第一差异图像进行处理得到只保留划痕的第一划痕图像；和/或，对所述第二差异图像进行处理得到只保留划痕的第二划痕图像；
根据所述待处理视频图像和所述划痕图像得到划痕修复图像包括：对所述待处理视频图像、所述第一划痕图像和/或第二划痕图像进行运算得到所述划痕修复图像。
本公开实施例中，可选的，对所述第一差异图像进行处理得到只保留划痕的第一划痕图像包括：
分别采用垂直方向的中值滤波器和水平方向的中值滤波器对第一差异图像进行中值滤波，得到第一垂直滤波后图像和第一水平滤波后图像；
若所述待处理视频图像中的划痕为垂直划痕，采用所述第一垂直滤波后图像减去所述第一水平滤波后图像，得到所述第一划痕图像；
若所述待处理视频图像中的划痕为水平划痕，采用所述第一水平滤波后图像减去所述第一垂直滤波后图像，得到所述第一划痕图像；
对所述第二差异图像进行处理得到只保留划痕的第二划痕图像包括：
分别采用垂直方向的中值滤波器和水平方向的中值滤波器对第二差异图像进行中值滤波，得到第二垂直滤波后图像和第二水平滤波后图像；
若所述待处理视频图像中的划痕为垂直划痕，采用所述第二垂直滤波后图像减去所述第二水平滤波后图像，得到所述第二划痕图像；
若所述待处理视频图像中的划痕为水平划痕，采用所述第二水平滤波后 图像减去所述第二垂直滤波后图像，得到所述第二划痕图像。
本公开实施例中，可选的，采用如下公式计算得到所述划痕修复图像：
I deline＝I-L white+L black
其中，I deline为所述划痕修复图像，I为所述待处理视频图像，L white为所述第一划痕图像，L black为所述第二划痕图像。
本公开实施例中，可选的，坏点修复子模块，用于根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像，对所述待处理视频图像进行中值滤波得到坏点修复图像。
本公开实施例中，可选的，根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像对所述坏点修复图像进行去伪影处理得到伪影修复图像包括：
对所述坏点修复图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像分别进行N 3-1次下采样，得到N 3-1种分辨率的下采样图像，其中，每一种分辨率的下采样图像包括与所述坏点修复图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像分别对应的N 1帧下采样图像；将N 3种分辨率的图像输入至多尺度级联网络中进行去伪影处理得到伪影修复图像，所述N 3种分辨率的图像包括：所述坏点修复图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像以及所述N 3-1种分辨率的下采样图像，所述多尺度级联网络包括N 3个级联的子网络，所述N 3个级联的子网络所处理的图像分别基于所述N 3种分辨率的图像生成；
其中，N 3为大于或等于2的正整数。
本公开实施例中，可选的，将N 3种分辨率的图像输入至多尺度级联网络中进行去伪影处理得到伪影修复图像包括：
针对所述N 3个级联的子网络中的第一个子网络：将所述坏点修复图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像分别进行A倍下采样，得到N 1帧第一下采样图像，将所述N 1帧第一下采样图像分别与自身进行拼接，得到第一拼接后图像，将所述第一拼接后图像输入到第一个子网络，得到第一输出图像；
针对第一个子网络和最后一个子网络之间的中间子网络：将前一个子网络的输出图像进行上采样得到第一上采样图像，将所述坏点修复图像、所述待处理视频图像之前的至少一帧视频图像以和之后的至少一帧视频图像分别进行B倍下采样，得到N 1帧第二下采样图像，所述第二下采样图像和所述第一上采样图像的尺度相同，将两组图像进行拼接得到第二拼接后图像，将所述第二拼接后图像输入到所述中间子网络，得到第二输出图像，其中，所述两组图像其中一组为N 1帧第二下采样图像，另一组包括：所述N 1帧第二下采样图像中除所述坏点修复图像对应的下采样图像之外的其他下采样图像，以及，所述第一上采样图像；
针对最后一个子网络：将前一个子网络的输出图像进行上采样得到第二上采样图像，所述第二上采样图像与所述待处理视频图像的尺度相同，将两组图像进行拼接得到第三拼接后图像，将所述第三拼接后图像输入到所述最后一个子网络，得到所述伪影修复图像，其中，所述两组图像其中一组为所述N 1帧视频图像，另一组包括：所述N 1帧视频图像中除所述坏点修复图像之外的其他图像，以及，所述第二上采样图像。
本公开实施例中，可选的，所述N 3个级联的子网络的结构相同，参数不同。
本公开实施例中，可选的，每个所述子网络包括多个3D卷积层、多个3的反卷积层和多个3D平均池化层。
本公开实施例中，可选的，N 3等于3，A等于4，B等于2。
本公开实施例中，可选的，多尺度级联网络采用以下训练方法得到：
获取连续的N 1帧训练图像，所述N 1帧训练图像中包括待处理训练图像、所述待处理训练图像之前的至少一帧训练图像和之后的至少一帧训练图像；
根据所述N 1帧训练图像对所述待处理训练图像进行滤波处理得到第一训练图像；
根据所述第一训练图像、所述待处理训练图像之前的至少一帧训练图像和之后的至少一帧训练图像，对待训练的多尺度级联网络进行训练，得到训练后的多尺度级联网络。
本公开实施例中，可选的，对待训练的多尺度级联网络进行训练时，采 用的总损失包括以下至少一项：图像内容损失、颜色损失、边缘损失和感知损失。
本公开实施例中，可选的，所述总损失等于图像内容损失、颜色损失、边缘损失和感知损失的加权和。图像内容L 1损失采用如下公式计算：
Figure PCTCN2021082072-appb-000041
其中，l content为L1损失，
Figure PCTCN2021082072-appb-000042
为所述去伪影训练图像，y i为所述第一训练图像，n为一个Batch里的图像数量。
本公开实施例中，可选的，所述颜色损失采用如下公式计算：
Figure PCTCN2021082072-appb-000043
其中，l color为所述颜色损失，
Figure PCTCN2021082072-appb-000044
为所述去伪影训练图像，y i为所述第一训练图像，n为一个Batch里的图像数量，Blur(x)为高斯模糊函数。
本公开实施例中，可选的，所述边缘损失采用如下公式计算：
Figure PCTCN2021082072-appb-000045
其中，l edge为所述边缘损失，
Figure PCTCN2021082072-appb-000046
为所述去伪影训练图像，y i为所述第一训练图像，n为一个Batch里的图像数量，H j(x)代表HED网络第j层提取的图像边缘图。
本公开实施例中，可选的，所述感知损失采用如下公式计算：
Figure PCTCN2021082072-appb-000047
其中，l feature为所述感知损失，
Figure PCTCN2021082072-appb-000048
为所述去伪影训练图像，y i为所述第一训练图像，n为一个Batch里的图像数量，
Figure PCTCN2021082072-appb-000049
表示VGG网络第j层提取的图像特征图。
本公开实施例中，可选的，根据所述第一训练图像、所述待处理训练图像之前的至少一帧训练图像和之后的至少一帧训练图像，对待训练的多尺度级联网络进行训练包括：
在所述第一训练图像上随机裁剪出一个图像块，在所述待处理训练图像之前的至少一帧训练图像和之后的至少一帧训练图像的相同位置处分别裁剪 出一个图像块，得到N 1帧图像块；
将所述N 1帧图像块输入到待训练多尺度级联网络进行训练。
本公开实施例中，可选的，所述坏点修复子模块，用于：根据所述待处理视频图像和所述坏点修复图像，对所述伪影修复图像进行滤波处理得到输出图像。
本公开实施例中，可选的，所述坏点修复子模块，用于：根据所述待处理视频图像和所述坏点修复图像，对所述伪影修复图像进行中值滤波处理得到输出图像。
本公开实施例中，可选的，N 1等于3。
本公开实施例中，可选的，所述去噪子模块，用于所述根据连续的N 2帧视频图像得到目标非运动掩膜包括：
根据所述N 2帧视频图像和所述目标非运动掩膜，得到参考图；
将所述待去噪视频图像输入至待训练去噪网络中，得到第一去噪图；
根据所述第一去噪图和所述目标非运动掩膜，得到第二去噪图；
根据所述参考图和所述第二去噪图确定所述待训练去噪网络的损失函数，根据所述损失函数调整所述待训练去噪网络的参数，得到所述去噪网络。
本公开实施例中，可选的，根据所述N 2帧视频图像和所述目标非运动掩膜，得到参考图包括：
将所述N 2帧视频图像中每一帧第一视频图像分别与所述待去噪视频图像组成样本对输入至光流网络中，得到表示上下运动信息的第一光流图和表示左右运动信息的第二光流图，所述第一视频图像为所述N 2帧视频图像中除所述待去噪视频图像之外的其他视频图像，所述第一光流图和所述第二光流图的分辨率为X*Y；
根据所述第一光流图和所述第二光流图，计算每一帧第一视频图像与所述待去噪视频图像的运动掩膜，得到N 2-1个目运动掩膜；
根据所述N 2-1个运动掩膜得到目标非运动掩膜。
本公开实施例中，可选的，根据所述第一光流图和所述第二光流图，计算每一帧第一视频图像与所述待去噪视频图像的运动掩膜包括：
将所述第一光流图的后X-X1行与前X-X1行做减法运算，得到第一差值 图，并将第一差值图最后X1行补0，得到处理后的第一差值图；
将所述第二光流图的后Y-Y1列与前Y-Y1列做减法运算，得到第二差值图，并将第二差值图最后Y1列补0，得到处理后的第二差值图；
将所述处理后的第一差值图和所述处理后的第二差值图相加，得到第三差值图；
将所述第三差值图中绝对值大于预设阈值的像素赋值为1，小于所述预设阈值的像素赋值为0，得到二值图；
根据所述二值图得到运动掩膜。
本公开实施例中，可选的，根据所述二值图得到运动掩膜包括：
将所述二值图进行膨胀运算，得到所述运动掩膜。
本公开实施例中，可选的，根据所述N 2-1个运动掩膜得到目标非运动掩膜包括：
根据所述N 2-1个运动掩膜得到N 2-1个非运动掩膜；其中，非运动掩膜＝1-运动掩膜；
将所述N 2-1个非运动掩膜相乘，得到目标非运动掩膜。
本公开实施例中，可选的，根据所述N 2帧视频图像和所述目标非运动掩膜M，得到参考图包括：
将所述N 2帧视频图像分别与所述目标非运动掩膜M相乘，得到N 2个乘积；
将所述N 2个乘积相加并取均值，得到所述参考图。
本公开实施例中，可选的，根据所述第一去噪图和所述目标非运动掩膜M，得到第二去噪图包括：
将所述第一去噪图于所述目标非运动掩膜M相乘，得到所述第二去噪图。
本公开实施例中，可选的，所述N 2为5～9。
本公开实施例中，可选的，偏色矫正子模块，用于确定待处理视频图像的RGB各通道的目标偏色值包括：
获取所述待处理视频图像的RGB各通道的均值；
将所述RGB各通道的均值转换到Lab颜色空间，得到所述RGB各通道的均值对应的Lab空间各颜色分量(l,a,b)；
根据所述Lab空间各颜色分量(l,a,b)，确定所述RGB各通道的均值对 应的偏色程度(l,0-a,0-b)；
将所述偏色程度(l,0-a,0-b)转换到RGB颜色空间，得到RGB各通道的目标偏色值。
本公开实施例中，可选的，将所述RGB各通道的均值转换到Lab颜色空间包括：将所述RGB各通道的均值转换到XYZ颜色空间，得到XYZ颜色空间的均值；将XYZ颜色空间的均值转换到Lab颜色空间；
将所述偏色程度(l,0-a,0-b)转换到RGB颜色空间包括：将所述偏色程度转换到XYZ颜色空间，得到XYZ颜色空间的偏色程度；将XYZ颜色空间的偏色程度转换到RGB颜色空间。
本公开实施例中，可选的，根据所述目标偏色值，对所述待处理视频图像进行色彩平衡调整，得到第一矫正图像包括：
根据RGB各通道的目标偏色值，对所述待处理视频图像进行高光函数处理、阴影函数处理和中间调函数处理中的至少一项色彩平衡调整处理，其中，所述高光函数和阴影函数为线性函数，所述中间调函数为指数函数。
本公开实施例中，可选的，所述高光函数为：y＝a(v)*x+b(v)；
所述阴影函数为：y＝c(v)*x+d(v)；
所述中间调函数为：y＝x f(v)；
y为第一矫正图像，x为所述待处理视频图像，v根据RGB各通道的目标偏色值确定，f(v),a(v),b(v),c(v),d(v)是v的函数。
本公开实施例中，可选的，f(v)＝e -v。
本公开实施例中，可选的，
Figure PCTCN2021082072-appb-000050
b(v)＝0。
本公开实施例中，可选的，
Figure PCTCN2021082072-appb-000051
本公开实施例中，可选的，对于R通道，v＝(ΔR-d)-(ΔG-d)-(ΔB-d)；
对于G通道，v＝(ΔG-d)-(ΔR-d)-(ΔB-d)；
对于B通道，v＝(ΔB-d)-(ΔR-d)-(ΔG-d)；
其中，ΔR、ΔG和ΔB为RGB各通道的目标偏色值，d为将ΔR、ΔG、ΔB按照大小进行排序后的中间值。
本公开实施例中，可选的，根据参考图像，对所述第一矫正图像进行颜 色迁移，得到第二矫正图像包括：
将所述参考图像和所述第一矫正图像转换到Lab颜色空间；
在Lab颜色空间，确定所述参考图像和所述第一矫正图像的均值和标准差；
根据所述参考图像和所述第一矫正图像的均值和标准差，确定Lab颜色空间中的第k个通道的颜色迁移结果；
将所述颜色迁移结果转换到RGB颜色空间，得到所述第二矫正图像。
本公开实施例中，可选的，所述颜色迁移结果的计算方法如下：
Figure PCTCN2021082072-appb-000052
其中，I k为Lab颜色空间的第k个通道的颜色迁移结果，t为所述参考图像，S为所述第一矫正图像，
Figure PCTCN2021082072-appb-000053
表示所述第一矫正图像的第k个通道的均值，
Figure PCTCN2021082072-appb-000054
表示所述第一矫正图像的第k个通道的标准差，
Figure PCTCN2021082072-appb-000055
表示所述参考图像的第k个通道的均值，
Figure PCTCN2021082072-appb-000056
表示所述参考图像的第k个通道的标准差。
本公开实施例中，可选的，第k个通道为a和b通道中的至少一个。
本申请实施例还提供一种电子设备，包括处理器，存储器，存储在存储器上并可在所述处理器上运行的程序或指令，该程序或指令被处理器执行时实现上述图像处理方法实施例的各个过程，且能达到相同的技术效果。
本申请实施例还提供一种可读存储介质，所述可读存储介质上存储有程序或指令，该程序或指令被处理器执行时实现上述图像处理方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。
其中，所述处理器为上述实施例中所述的终端中的处理器。所述可读存储介质，包括计算机可读存储介质，如计算机只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包 括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。此外，需要指出的是，本申请实施方式中的方法和装置的范围不限按示出或讨论的顺序来执行功能，还可包括根据所涉及的功能按基本同时的方式或按相反的顺序来执行功能，例如，可以按不同于所描述的次序来执行所描述的方法，并且还可以添加、省去、或组合各种步骤。另外，参照某些示例所描述的特征可在其他示例中被组合。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端(可以是手机，计算机，服务器，空调器，或者网络设备等)执行本申请各个实施例所述的方法。
上面结合附图对本公开的实施例进行了描述，但是本公开并不局限于上述的具体实施方式，上述的具体实施方式仅仅是示意性的，而不是限制性的，本领域的普通技术人员在本公开的启示下，在不脱离本公开宗旨和权利要求所保护的范围情况下，还可做出很多形式，均属于本公开的保护之内。

Claims (47)

  1. 一种图像处理方法，其特征在于，包括：
    对待处理视频图像执行以下步骤中的至少之一：
    划痕修复步骤：对待处理视频图像进行去划痕处理，得到第一图像；对所述待处理视频图像和所述第一图像进行差异运算得到差异图像；对所述差异图像进行处理得到只保留划痕的划痕图像；根据所述待处理视频图像和所述划痕图像得到划痕修复图像；
    坏点修复步骤：获取连续的N 1帧视频图像，所述N 1帧视频图像中包括待处理视频图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像，其中，N 1为大于或等于3的正整数；根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像，对所述待处理视频图像进行滤波处理得到坏点修复图像；根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像对所述坏点修复图像进行去伪影处理得到伪影修复图像；
    去噪步骤：采用去噪网络对待处理视频图像进行去噪处理；其中，所述去噪网络采用以下训练方法得到：根据连续的N 2帧视频图像得到目标非运动掩膜，所述N 2帧视频图像包括待去噪视频图像；根据所述N 2帧视频图像和所述目标非运动掩膜，对待训练的去噪网络进行训练，得到所述去噪网络，其中，N 2为大于或等于2的正整数；
    偏色矫正步骤：确定待处理视频图像的RGB各通道的目标偏色值；根据所述目标偏色值，对所述待处理视频图像进行色彩平衡调整，得到第一矫正图像；根据参考图像，对所述第一矫正图像进行颜色迁移，得到第二矫正图像。
  2. 根据权利要求1所述的方法，其特征在于，对待处理视频图像进行去划痕处理包括：
    根据滤波器类型和所述待处理视频图像中的划痕类型中的至少一个，对所述待处理视频图像进行中值滤波处理，得到去划痕图像。
  3. 根据权利要求2所述的方法，其特征在于，根据滤波器类型和所述待 处理视频图像中的划痕类型中的至少一个，对所述待处理视频图像进行中值滤波处理包括：
    根据所述待处理视频图像中的划痕类型，选取对应的滤波器类型，对所述待处理视频图像进行中值滤波处理，其中：
    当所述待处理视频图像中的划痕为垂直划痕时，采用水平方向的中值滤波器对所述待处理视频图像进行中值滤波；
    当所述待处理视频图像中的划痕为水平划痕时，采用对垂直方向的中值滤波器所述待处理视频图像进行中值滤波。
  4. 根据权利要求2所述的方法，其特征在于，根据滤波器类型和所述待处理视频图像中的划痕类型中的至少一个，对所述待处理视频图像进行中值滤波处理包括：
    根据滤波器类型和所述待处理视频图像中的划痕类型，对所述待处理视频图像不同的预处理，对预处理之后的待处理视频图像进行中值滤波处理，其中：
    当采用的是水平方向的中值滤波器，且所述待处理视频图像中的划痕为非垂直划痕时，将所述待处理视频图像进行旋转，使得划痕转换为垂直划痕；
    当采用的是垂直方向的中值滤波器，且所述视频图像中的划痕为非水平划痕时，将所述待处理视频图像进行旋转，使得划痕转换为水平划痕。
  5. 根据权利要求1所述的方法，其特征在于，
    对待处理视频图像进行去划痕处理包括：采用尺寸为1×k和/或k×1的中值滤波器对待处理视频图像进行中值滤波；
    对待处理视频图像进行去划痕处理之前还包括：
    将所述中值滤波器的k从预设值开始依次增大，对所述待处理视频图像进行中值滤波，得到第二图像；
    根据所述第二图像的滤波效果确定k的最终值。
  6. 根据权利要求1所述的方法，其特征在于，对所述待处理视频图像和所述第一图像进行差异运算得到差异图像包括：
    对所述待处理视频图像和所述第一图像进行差异运算，得到第一差异图像和/或第二差异图像，其中，所述第一差异图像由所述待处理视频图像减去 所述第一图像得到，所述第二差异图像由所述第一图像减去所述待处理视频图像得到；
    对所述差异图像进行处理得到只保留划痕的划痕图像包括：对所述第一差异图像进行处理得到只保留划痕的第一划痕图像；和/或，对所述第二差异图像进行处理得到只保留划痕的第二划痕图像；
    根据所述待处理视频图像和所述划痕图像得到划痕修复图像包括：对所述待处理视频图像、所述第一划痕图像和/或第二划痕图像进行运算得到所述划痕修复图像。
  7. 根据权利要求6所述的方法，其特征在于，
    对所述第一差异图像进行处理得到只保留划痕的第一划痕图像包括：
    分别采用垂直方向的中值滤波器和水平方向的中值滤波器对第一差异图像进行中值滤波，得到第一垂直滤波后图像和第一水平滤波后图像；
    若所述待处理视频图像中的划痕为垂直划痕，采用所述第一垂直滤波后图像减去所述第一水平滤波后图像，得到所述第一划痕图像；
    若所述待处理视频图像中的划痕为水平划痕，采用所述第一水平滤波后图像减去所述第一垂直滤波后图像，得到所述第一划痕图像；
    对所述第二差异图像进行处理得到只保留划痕的第二划痕图像包括：
    分别采用垂直方向的中值滤波器和水平方向的中值滤波器对第二差异图像进行中值滤波，得到第二垂直滤波后图像和第二水平滤波后图像；
    若所述待处理视频图像中的划痕为垂直划痕，采用所述第二垂直滤波后图像减去所述第二水平滤波后图像，得到所述第二划痕图像；
    若所述待处理视频图像中的划痕为水平划痕，采用所述第二水平滤波后图像减去所述第二垂直滤波后图像，得到所述第二划痕图像。
  8. 根据权利要求6所述的方法，其特征在于，采用如下公式计算得到所述划痕修复图像：
    I deline＝I-L white+L black
    其中，I deline为所述划痕修复图像，I为所述待处理视频图像，L white为所述第一划痕图像，L black为所述第二划痕图像。
  9. 根据权利要求1所述的方法，其特征在于，根据所述待处理视频图像 之前的至少一帧视频图像和之后的至少一帧视频图像，对所述待处理视频图像进行滤波处理得到坏点修复图像包括：
    根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像，对所述待处理视频图像进行中值滤波得到坏点修复图像。
  10. 根据权利要求1所述的方法，其特征在于，根据所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像对所述坏点修复图像进行去伪影处理得到伪影修复图像包括：
    对所述坏点修复图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像分别进行N 3-1次下采样，得到N 3-1种分辨率的下采样图像，其中，每一种分辨率的下采样图像包括与所述坏点修复图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像分别对应的N 1帧下采样图像；将N e种分辨率的图像输入至多尺度级联网络中进行去伪影处理得到伪影修复图像，所述N e种分辨率的图像包括：所述坏点修复图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像以及所述N e-1种分辨率的下采样图像，所述多尺度级联网络包括N 3个级联的子网络，所述N 3个级联的子网络所处理的图像分别基于所述N 3种分辨率的图像生成；
    其中，N 3为大于或等于2的正整数。
  11. 根据权利要求10所述的方法，其特征在于，将N 3种分辨率的图像输入至多尺度级联网络中进行去伪影处理得到伪影修复图像包括：
    针对所述N 3个级联的子网络中的第一个子网络：将所述坏点修复图像、所述待处理视频图像之前的至少一帧视频图像和之后的至少一帧视频图像分别进行A倍下采样，得到N 1帧第一下采样图像，将所述N 1帧第一下采样图像分别与自身进行拼接，得到第一拼接后图像，将所述第一拼接后图像输入到第一个子网络，得到第一输出图像；
    针对第一个子网络和最后一个子网络之间的中间子网络：将前一个子网络的输出图像进行上采样得到第一上采样图像，将所述坏点修复图像、所述待处理视频图像之前的至少一帧视频图像以和之后的至少一帧视频图像分别进行B倍下采样，得到N 1帧第二下采样图像，所述第二下采样图像和所述第 一上采样图像的尺度相同，将两组图像进行拼接得到第二拼接后图像，将所述第二拼接后图像输入到所述中间子网络，得到第二输出图像，其中，所述两组图像其中一组为N 1帧第二下采样图像，另一组包括：所述N 1帧第二下采样图像中除所述坏点修复图像对应的下采样图像之外的其他下采样图像，以及，所述第一上采样图像；
    针对最后一个子网络：将前一个子网络的输出图像进行上采样得到第二上采样图像，所述第二上采样图像与所述待处理视频图像的尺度相同，将两组图像进行拼接得到第三拼接后图像，将所述第三拼接后图像输入到所述最后一个子网络，得到所述伪影修复图像，其中，所述两组图像其中一组为所述N 1帧视频图像，另一组包括：所述N 1帧视频图像中除所述坏点修复图像之外的其他图像，以及，所述第二上采样图像。
  12. 根据权利要求10所述的方法，其特征在于，所述N 3个级联的子网络的结构相同，参数不同。
  13. 根据权利要求10所述的方法，其特征在于，每个所述子网络包括多个3D卷积层、多个3的反卷积层和多个3D平均池化层。
  14. 根据权利要求11所述的方法，其特征在于，N 3等于3，A等于4，B等于2。
  15. 根据权利要求1、9-14任一项所述的方法，其特征在于，多尺度级联网络采用以下训练方法得到：
    获取连续的N 1帧训练图像，所述N 1帧训练图像中包括待处理训练图像、所述待处理训练图像之前的至少一帧训练图像和之后的至少一帧训练图像；
    根据所述N 1帧训练图像对所述待处理训练图像进行滤波处理得到第一训练图像；
    根据所述第一训练图像、所述待处理训练图像之前的至少一帧训练图像和之后的至少一帧训练图像，对待训练的多尺度级联网络进行训练，得到训练后的多尺度级联网络。
  16. 根据权利要求15所述的方法，其特征在于，对待训练的多尺度级联网络进行训练时，采用的总损失包括以下至少一项：图像内容损失、颜色损失、边缘损失和感知损失。
  17. 根据权利要求16所述的方法，其特征在于，所述总损失等于图像内容损失、颜色损失、边缘损失和感知损失的加权和。
  18. 根据权利要求17所述的方法，其特征在于，所述图像内容损失采用如下公式计算：
    Figure PCTCN2021082072-appb-100001
    其中，l content为L1损失，
    Figure PCTCN2021082072-appb-100002
    为所述去伪影训练图像，y i为所述第一训练图像，n为一个Batch里的图像数量。
  19. 根据权利要求16所述的方法，其特征在于，所述颜色损失采用如下公式计算：
    Figure PCTCN2021082072-appb-100003
    其中，l color为所述颜色损失，
    Figure PCTCN2021082072-appb-100004
    为所述去伪影训练图像，y i为所述第一训练图像，n为一个Batch里的图像数量，Blur(x)为高斯模糊函数。
  20. 根据权利要求16所述的方法，其特征在于，所述边缘损失采用如下公式计算：
    Figure PCTCN2021082072-appb-100005
    其中，l edge为所述边缘损失，
    Figure PCTCN2021082072-appb-100006
    为所述去伪影训练图像，y i为所述第一训练图像，n为一个Batch里的图像数量，H j(x)代表HED网络第j层提取的图像边缘图。
  21. 根据权利要求16所述的方法，其特征在于，所述感知损失采用如下公式计算：
    Figure PCTCN2021082072-appb-100007
    其中，l feature为所述感知损失，
    Figure PCTCN2021082072-appb-100008
    为所述去伪影训练图像，y i为所述第一训练图像，n为一个Batch里的图像数量，
    Figure PCTCN2021082072-appb-100009
    表示VGG网络第j层提取的图像特征图。
  22. 根据权利要求15所述的方法，其特征在于，根据所述第一训练图像、所述待处理训练图像之前的至少一帧训练图像和之后的至少一帧训练图像， 对待训练的多尺度级联网络进行训练包括：
    在所述第一训练图像上随机裁剪出一个图像块，在所述待处理训练图像之前的至少一帧训练图像和之后的至少一帧训练图像的相同位置处分别裁剪出一个图像块，得到N 1帧图像块；
    将所述N 1帧图像块输入到待训练多尺度级联网络进行训练。
  23. 根据权利要求1所述的方法，其特征在于，得到伪影修复图像之后还包括：
    根据所述待处理视频图像和所述坏点修复图像，对所述伪影修复图像进行滤波处理，得到输出图像。
  24. 根据权利要求23所述的方法，其特征在于，根据所述待处理视频图像和所述坏点修复图像，对所述伪影修复图像进行中值滤波处理，得到输出图像。
  25. 根据权利要求1所述的方法，其特征在于，N 1等于3。
  26. 根据权利要求1所述的方法，其特征在于，根据所述N 2帧视频图像和所述目标非运动掩膜，对待训练的去噪网络进行训练，得到所述去噪网络包括：
    根据所述N 2帧视频图像和所述目标非运动掩膜，得到参考图；
    将所述待去噪视频图像输入至待训练去噪网络中，得到第一去噪图；
    根据所述第一去噪图和所述目标非运动掩膜，得到第二去噪图；
    根据所述参考图和所述第二去噪图确定所述待训练去噪网络的损失函数，根据所述损失函数调整所述待训练去噪网络的参数，得到所述去噪网络。
  27. 根据权利要求1所述的方法，其特征在于，所述待去噪视频图像为所述N 2帧视频图像中的中间帧视频图像。
  28. 根据权利要求1所述的方法，其特征在于，所述根据连续的N 2帧视频图像得到目标非运动掩膜包括：
    将所述N 2帧视频图像中每一帧第一视频图像分别与所述待去噪视频图像组成样本对输入至光流网络中，得到表示上下运动信息的第一光流图和表示左右运动信息的第二光流图，所述第一视频图像为所述N 2帧视频图像中除所述待去噪视频图像之外的其他视频图像；
    根据所述第一光流图和所述第二光流图，确定每一帧第一视频图像与所述待去噪视频图像的运动掩膜，得到N 2-1个运动掩膜；
    根据所述N 2-1个运动掩膜得到目标非运动掩膜。
  29. 根据权利要求28所述的方法，其特征在于，所述第一光流图和所述第二光流图的分辨率为X*Y，根据所述第一光流图和所述第二光流图，确定每一帧第一视频图像与所述待去噪视频图像的运动掩膜包括：
    将所述第一光流图的后X-X1行与前X-X1行做减法运算，得到第一差值图，并将第一差值图最后X1行补0，得到处理后的第一差值图；
    将所述第二光流图的后Y-Y1列与前Y-Y1列做减法运算，得到第二差值图，并将第二差值图最后Y1列补0，得到处理后的第二差值图；
    将所述处理后的第一差值图和所述处理后的第二差值图相加，得到第三差值图；
    将所述第三差值图中绝对值大于预设阈值的像素赋值为1，小于所述预设阈值的像素赋值为0，得到二值图；
    根据所述二值图得到运动掩膜。
  30. 根据权利要求29所述的方法，其特征在于，根据所述二值图得到运动掩膜包括：
    将所述二值图进行膨胀运算，得到所述运动掩膜。
  31. 根据权利要求28所述的方法，其特征在于，根据所述N 2-1个运动掩膜得到目标非运动掩膜包括：
    根据所述N 2-1个运动掩膜得到N 2-1个非运动掩膜；其中，非运动掩膜＝1-运动掩膜；
    将所述N 2-1个非运动掩膜相乘，得到目标非运动掩膜。
  32. 根据权利要求28所述的方法，其特征在于，根据所述N 2帧视频图像和所述目标非运动掩膜，得到参考图包括：
    将所述N 2帧视频图像分别与所述目标非运动掩膜相乘，得到N 2个乘积；
    将所述N 2个乘积相加并取均值，得到所述参考图。
  33. 根据权利要求26所述的方法，其特征在于，根据所述第一去噪图和所述目标非运动掩膜，得到第二去噪图包括：
    将所述第一去噪图与所述目标非运动掩膜相乘，得到所述第二去噪图。
  34. 根据权利要求1、26-33任一项所述的方法，其特征在于，所述N 2为5～9。
  35. 根据权利要求1所述的方法，其特征在于，确定待处理视频图像的RGB各通道的目标偏色值包括：
    获取所述待处理视频图像的RGB各通道的均值；
    将所述RGB各通道的均值转换到Lab颜色空间，得到所述RGB各通道的均值对应的Lab空间各颜色分量；
    根据所述Lab空间各颜色分量，确定所述RGB各通道的均值对应的偏色程度；
    将所述偏色程度转换到RGB颜色空间，得到RGB各通道的目标偏色值。
  36. 根据权利要求35所述的方法，其特征在于，
    将所述RGB各通道的均值转换到Lab颜色空间包括：将所述RGB各通道的均值转换到XYZ颜色空间，得到XYZ颜色空间的均值；将XYZ颜色空间的均值转换到Lab颜色空间；
    将所述偏色程度转换到RGB颜色空间包括：将所述偏色程度转换到XYZ颜色空间，得到XYZ颜色空间的偏色程度；将XYZ颜色空间的偏色程度转换到RGB颜色空间。
  37. 根据权利要求1所述的方法，其特征在于，根据所述目标偏色值，对所述待处理视频图像进行色彩平衡调整，得到第一矫正图像包括：
    根据RGB各通道的目标偏色值，对所述待处理视频图像进行高光函数处理、阴影函数处理和中间调函数处理中的至少一项色彩平衡调整处理，其中，所述高光函数和阴影函数为线性函数，所述中间调函数为指数函数。
  38. 根据权利要求37所述的方法，其特征在于，
    所述高光函数为：y＝a(v)*x+b(v)；
    所述阴影函数为：y＝c(v)*x+d(v)；
    所述中间调函数为：y＝x f(v)；
    y为第一矫正图像，x为所述待处理视频图像，v根据RGB各通道的目标偏色值确定，f(v),a(v),b(v),c(v),d(v)是v的函数。
  39. 根据权利要求37所述的方法，其特征在于，f(v)＝e -v。
  40. 根据权利要求37所述的方法，其特征在于，
    Figure PCTCN2021082072-appb-100010
    b(v)＝0。
  41. 根据权利要求37所述的方法，其特征在于，
    Figure PCTCN2021082072-appb-100011
  42. 根据权利要求37所述的方法，其特征在于，
    对于R通道，v＝(ΔR-d)-(ΔG-d)-(ΔB-d)；
    对于G通道，v＝(ΔG-d)-(ΔR-d)-(ΔB-d)；
    对于B通道，v＝(ΔB-d)-(ΔR-d)-(ΔG-d)；
    其中，ΔR、ΔG和ΔB为RGB各通道的目标偏色值，d为将ΔR、ΔG、ΔB按照大小进行排序后的中间值。
  43. 根据权利要求1所述的方法，其特征在于，根据参考图像，对所述第一矫正图像进行颜色迁移，得到第二矫正图像包括：
    将所述参考图像和所述第一矫正图像转换到Lab颜色空间；
    在Lab颜色空间，确定所述参考图像和所述第一矫正图像的均值和标准差；
    根据所述参考图像和所述第一矫正图像的均值和标准差，确定Lab颜色空间中的第k个通道的颜色迁移结果；
    将所述颜色迁移结果转换到RGB颜色空间，得到所述第二矫正图像。
  44. 根据权利要求43所述的方法，其特征在于，所述颜色迁移结果的计算方法如下：
    Figure PCTCN2021082072-appb-100012
    其中，I k为Lab颜色空间的第k个通道的颜色迁移结果，t为所述参考图像，S为所述第一矫正图像，
    Figure PCTCN2021082072-appb-100013
    表示所述第一矫正图像的第k个通道的均值，
    Figure PCTCN2021082072-appb-100014
    表示所述第一矫正图像的第k个通道的标准差，
    Figure PCTCN2021082072-appb-100015
    表示所述参考图像的第k个通道的均值，
    Figure PCTCN2021082072-appb-100016
    表示所述参考图像的第k个通道的标准差。
  45. 根据权利要求43或44所述的方法，其特征在于，第k个通道为a和b通道中的至少一个。
  46. 一种电子设备，其特征在于，包括处理器，存储器及存储在所述存储器上并可在所述处理器上运行的程序或指令，所述程序或指令被所述处理 器执行时实现如权利要求1至45任一项所述的图像处理方法的步骤。
  47. 一种可读存储介质，其特征在于，所述可读存储介质上存储程序或指令，所述程序或指令被处理器执行时实现如权利要求1至45任一项所述的图像处理方法的步骤。
PCT/CN2021/082072 2021-03-22 2021-03-22 图像处理方法及图像处理装置 WO2022198381A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
PCT/CN2021/082072 WO2022198381A1 (zh) 2021-03-22 2021-03-22 图像处理方法及图像处理装置
CN202180000544.6A CN115398469A (zh) 2021-03-22 2021-03-22 图像处理方法及图像处理装置
US17/764,458 US20240062344A1 (en) 2021-03-22 2021-03-22 Image processing method and image processing device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/082072 WO2022198381A1 (zh) 2021-03-22 2021-03-22 图像处理方法及图像处理装置

Publications (1)

Publication Number Publication Date
WO2022198381A1 true WO2022198381A1 (zh) 2022-09-29

Family

ID=83395037

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/082072 WO2022198381A1 (zh) 2021-03-22 2021-03-22 图像处理方法及图像处理装置

Country Status (3)

Country Link
US (1) US20240062344A1 (zh)
CN (1) CN115398469A (zh)
WO (1) WO2022198381A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI840073B (zh) 2023-01-04 2024-04-21 大陸商廣州印芯半導體技術有限公司 圖像處理裝置及包含該圖像處理裝置的駕駛監控系統
WO2024087163A1 (zh) * 2022-10-28 2024-05-02 京东方科技集团股份有限公司 坏点检测模型训练方法、坏点检测方法以及坏点修复方法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2961169A1 (en) * 2014-06-25 2015-12-30 Thomson Licensing Method and device for processing images
CN111738952A (zh) * 2020-06-22 2020-10-02 京东方科技集团股份有限公司 一种图像修复的方法、装置及电子设备
CN112102212A (zh) * 2020-11-19 2020-12-18 腾讯科技（深圳）有限公司 一种视频修复方法、装置、设备及存储介质

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2961169A1 (en) * 2014-06-25 2015-12-30 Thomson Licensing Method and device for processing images
CN111738952A (zh) * 2020-06-22 2020-10-02 京东方科技集团股份有限公司 一种图像修复的方法、装置及电子设备
CN112102212A (zh) * 2020-11-19 2020-12-18 腾讯科技（深圳）有限公司 一种视频修复方法、装置、设备及存储介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024087163A1 (zh) * 2022-10-28 2024-05-02 京东方科技集团股份有限公司 坏点检测模型训练方法、坏点检测方法以及坏点修复方法
TWI840073B (zh) 2023-01-04 2024-04-21 大陸商廣州印芯半導體技術有限公司 圖像處理裝置及包含該圖像處理裝置的駕駛監控系統

Also Published As

Publication number Publication date
CN115398469A (zh) 2022-11-25
US20240062344A1 (en) 2024-02-22

Similar Documents

Publication Publication Date Title
Mildenhall et al. Burst denoising with kernel prediction networks
CN110008817B (zh) 模型训练、图像处理方法、装置、电子设备及计算机可读存储介质
JP5543605B2 (ja) 空間画像事前確率を用いたぼけ画像修正
US11625815B2 (en) Image processor and method
US8559744B2 (en) Method for processing a digital object and related system
KR100911890B1 (ko) 이미지 모델의 컬러 성분의 복원을 위한 방법, 시스템,프로그램 모듈, 및 컴퓨터 프로그램 생성물
CN110930301B (zh) 图像处理方法、装置、存储介质及电子设备
KR20190089922A (ko) 광학 시스템 수차들의 디지털 보정
JP2001229377A (ja) 適応帰納的フィルタでディジタル画像のコントラストを調整する方法
US10628924B2 (en) Method and device for deblurring out-of-focus blurred images
US20100067818A1 (en) System and method for high quality image and video upscaling
Hou et al. Underwater image dehazing and denoising via curvature variation regularization
CN113632134B (zh) 用于生成高动态范围图像的方法、计算机可读存储介质和hdr相机
JP2001313844A (ja) 局所的な色補正を実行する方法および装置
WO2022198381A1 (zh) 图像处理方法及图像处理装置
CN113379609A (zh) 一种图像处理方法、存储介质及终端设备
US11783454B2 (en) Saliency map generation method and image processing system using the same
Chang et al. Beyond camera motion blur removing: How to handle outliers in deblurring
CN113012067A (zh) 基于Retinex理论和端到端深度网络的水下图像复原方法
Gautam et al. An advanced visibility restoration technique for underwater images
Wang et al. New insights into multi-focus image fusion: A fusion method based on multi-dictionary linear sparse representation and region fusion model
CN117499619B (zh) 一种自适应镜头解析力的图像处理方法及应用
Lee et al. Multi-image high dynamic range algorithm using a hybrid camera
WO2023044653A1 (zh) 一种显示设备系统及自适应增强画质的方法
Li et al. Chromatic aberration correction using cross-channel prior in shearlet domain

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 17764458

Country of ref document: US

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21932016

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 22.01.2024)