WO2021012965A1 - 图像处理和移动终端视频处理方法、装置、设备和介质 - Google Patents

图像处理和移动终端视频处理方法、装置、设备和介质 Download PDF

Info

Publication number
WO2021012965A1
WO2021012965A1 PCT/CN2020/101243 CN2020101243W WO2021012965A1 WO 2021012965 A1 WO2021012965 A1 WO 2021012965A1 CN 2020101243 W CN2020101243 W CN 2020101243W WO 2021012965 A1 WO2021012965 A1 WO 2021012965A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
processed
image block
block
video
Prior art date
Application number
PCT/CN2020/101243
Other languages
English (en)
French (fr)
Inventor
孟祥飞
Original Assignee
广州市百果园信息技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 广州市百果园信息技术有限公司 filed Critical 广州市百果园信息技术有限公司
Priority to US17/629,132 priority Critical patent/US20220239908A1/en
Priority to EP20844069.3A priority patent/EP4006824A4/en
Publication of WO2021012965A1 publication Critical patent/WO2021012965A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/117—Filters, e.g. for pre-processing or post-processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G06T5/30—Erosion or dilatation, e.g. thinning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/30—Noise filtering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/467—Encoded features or binary features, e.g. local binary patterns [LBP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H04N19/14—Coding unit complexity, e.g. amount of activity or edge presence estimation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
    • H04N19/86—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving reduction of coding artifacts, e.g. of blockiness
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20004—Adaptive image processing
    • G06T2207/20012—Locally adaptive
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/21—Circuitry for suppressing or minimising disturbance, e.g. moiré or halo

Definitions

  • the embodiments of the present application relate to the field of image processing, for example, to an image processing and mobile terminal video processing method, device, device, and medium.
  • coding noise In the process of video preservation and transmission, lossy compression is generally used for encoding. After the video is decoded, coding noise will be generated. Types of coding noise include blocking effect, ringing effect and sawtooth effect. For example, when a user watches a video, the mobile video application pulls the video stream from the server, decodes and plays the video stream locally. Due to the limitations of the network environment, the video stream pulled by the mobile terminal is often a highly compressed video Streaming, when a user is watching a video, there will generally be visible coding noise, which is more common when the network condition is poor.
  • the embodiments of the present application provide an image processing and mobile terminal video processing method, device, equipment and medium, which realize real-time denoising of images and adaptive filtering of each image block, thereby improving the definition of video .
  • the embodiment of the application provides an image processing method, including:
  • the embodiment of the application provides a video processing method of a mobile terminal, including:
  • the video is played.
  • An embodiment of the application provides an image processing device, including:
  • the first determining module is configured to determine the image attributes of each of the multiple image blocks in the image to be processed
  • the second determining module is configured to determine the filter model corresponding to each image block according to the image attribute of each image block;
  • the first filtering module is configured to filter each image block through the filter model corresponding to each image block to obtain a filtered image of the image to be processed.
  • the embodiment of the application provides a mobile terminal video processing device, including:
  • the decoding module is set to decode the to-be-processed video pulled from the video stream;
  • An extraction module configured to extract video frames in the to-be-processed video
  • a processing module configured to use the method described in any of the foregoing embodiments to process the video frame as the image to be processed to obtain a filtered image corresponding to the video frame;
  • the playing module is configured to play the video according to the filtered image.
  • An embodiment of the present application provides a device, which includes:
  • One or more processors are One or more processors;
  • Memory used to store one or more programs
  • the one or more processors When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method described in any embodiment of this application, or implement the image processing method described in any embodiment of this application. Mobile terminal video processing method.
  • the embodiment of the present application provides a computer-readable storage medium on which a computer program is stored.
  • the program is executed by a processor, the image processing method described in any embodiment of the present application is implemented, or the image processing method described in any embodiment of the present application is implemented.
  • Mobile terminal video processing method When the program is executed by a processor, the image processing method described in any embodiment of the present application is implemented, or the image processing method described in any embodiment of the present application is implemented.
  • FIG. 1 is a flowchart of an image processing method provided by an embodiment of the application
  • FIG. 2 is a schematic diagram of displaying image block division of a to-be-processed image provided by an embodiment of the present application
  • FIG. 3 is a flowchart of another image processing method provided by an embodiment of the application.
  • FIG. 5 is a flowchart of still another image processing method provided by an embodiment of the present application.
  • FIG. 6 is a flowchart of filtering an image block in an image processing method provided by an embodiment of the present application.
  • FIG. 7 is a flowchart of a video processing method of a mobile terminal provided by an embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of an image processing device provided by an embodiment of the application.
  • FIG. 9 is a schematic structural diagram of a video processing device for a mobile terminal according to an embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of a device provided by an embodiment of this application.
  • the embodiments of the present application may be applicable to situations in which when a user is watching a video, due to network limitations of the environment in which the user is located, visible coding noise generally appears, and the coding noise needs to be removed.
  • the embodiment of the application provides an image processing method, which implements real-time filtering of images, and adaptively filters each image block through a corresponding filtering model, thereby improving the definition of the video.
  • the filter model is trained, so that when the image processing method is executed, the corresponding filter model can be determined according to the coding noise intensity of the image to be processed and the image attributes of the image block in the image to be processed, and the image The block is filtered.
  • the process of training the filter model includes: obtaining a high-definition image of a first preset size and a low-definition image carrying coding noise, the resolution of the high-definition image and the low-definition image are the same; A plurality of image blocks of a second preset size to form a first matrix and a second matrix respectively, the second preset size is smaller than the first preset size; according to the normal equation of the least square method, and according to the first matrix Matrix and the second matrix to get the filtering model.
  • the filtering model in the embodiment of the present application may be calculated based on the least square method.
  • M and N are used to represent the size of the high-definition image, and they are all positive numbers, for example, M represents the length of the high-definition image, N represents the width of the high-definition image; L is a positive integer greater than or equal to 1.
  • d ⁇ d filtering model h refers to the size of the input original image during the filtering operation is d ⁇ d.
  • Is the vectorized representation of h ⁇ R d ⁇ d Represents a set of d 2 -dimensional real number vectors
  • h is a real number vector element in the set of real number vectors. It is a matrix consisting of image blocks of size d ⁇ d extracted from the low-resolution image y i .
  • the method of extraction is: for each pixel, extract a d ⁇ d image block centered on this pixel, and The image block of d ⁇ d is stretched into a row as a row of the matrix A i ; Represents a set of real number matrices with rows MN and 2 columns d, and A i is a real number matrix in the set of real number matrices.
  • b i ⁇ R MN , consisting of pixel expansion of x i , where each element in b i corresponds to the pixel at the center of multiple d ⁇ d image blocks in the low-resolution image y i , R MN represents MN-dimensional
  • the set of real number vectors, b i is a real number vector element in the set of real number vectors R MN .
  • the training method of the optimal filtering model can be easily extended from one picture to multiple pictures, for the convenience of description, the training method of the optimal filtering model can be limited to one picture, and the subscript i is omitted. It can be understood that the size of the filter model and the size of the image block are the same, that is, the size of the filter model is d ⁇ d, and the size of the image block is also d ⁇ d.
  • Q is a matrix of 2 xd 2 d
  • V is a vector of dimension d 2.
  • a j denote the jth row of A
  • both Q and V can be accumulated by small matrices/vectors. Represents a set of real number matrices with 1 row and d 2 columns
  • a j is a set of real number matrices A real matrix in.
  • the space complexity of formula (2) is O(d 4 ), and, in order to facilitate the statistics of the pixel value of each image block in the image, If you choose, set the value of d between 5-13. Therefore, the space complexity of formula (2) is not very large.
  • the filter model h is used to filter a low-definition image to generate a high-definition image that removes coding noise as much as possible, it is difficult for the filter model to make adaptive adjustments to the image content. Therefore, hash classification (hashing) is performed on each image block in the low-definition image, and a filtering model is trained for each bucket of the hash classification.
  • the first hash value can be represented by ⁇
  • the second hash value can be represented by ⁇
  • ⁇ can be calculated using formula (5)
  • ⁇ can be calculated using formula (6).
  • LGx and LGy represent the image gradients in the x direction and y direction of the image block; the value of i is any number in 1-d, and the value of j is in 1-d LGx 2 represents the square value of the image gradient corresponding to each image block in the x direction, LGy 2 represents the square value of the image gradient corresponding to each image block in the y direction, LGxLGy represents each image block in the x direction And the product value of the image gradient corresponding to the y direction.
  • the value range of ⁇ and ⁇ are both in the range of [0,1].
  • the interval [0,1] is equally divided into q ⁇ and q ⁇ sub-intervals respectively, q ⁇ is the number of sub-intervals corresponding to the first hash value, q ⁇ is the number of sub-intervals corresponding to the second hash value, q
  • the values of ⁇ and q ⁇ can be customized. Then, the combination of a sub-interval randomly selected from q ⁇ sub-intervals and a sub-interval randomly selected from q ⁇ sub-intervals is called a bucket, and each image block will fall in a certain Inside the barrel.
  • the training process of the filtering model includes the following steps:
  • the subintervals corresponding to the first hash value are [0,0.5) and [0.5,1]
  • the subintervals corresponding to the second hash value are [0 ,0.25), [0.25,5), [0.5,0.75) and [0.75,1]
  • select one sub-interval from the sub-interval corresponding to the first hash value and the sub-interval corresponding to the second hash value select one sub-interval from the sub-interval corresponding to the first hash value and the sub-interval corresponding to the second hash value
  • the combination of the selected two sub-intervals forms a bucket, so there can be 8 buckets, such as [0,0.5) and [0,0.25) corresponding to the bucket sequence number 1, and [0,0.5) and [0.25,5) corresponding
  • the bucket number is 2, [0,0.5) and [0.5,0.75) correspond to the bucket number 3, [0,0.5) and [0.75,1] correspond to the bucket number 4, [0.5,1] and [0, 0.25) corresponds to the bucket number 5,
  • step S20 in the process of calculating ⁇ and ⁇ of each image block, if the sequence of "traverse the image block first, and then obtain ⁇ and ⁇ according to the formula", the time complexity is O(MNd 2 ).
  • the integral graph algorithm can be used to simplify the time complexity to O(MN).
  • the process of training the filter model is to train an image quality enhancement model offline, that is, a model that upgrades low-definition images to high-definition images.
  • the model is a model file.
  • the online video can be filtered in real time through the model file to obtain a video with enhanced image quality.
  • the image processing method in this embodiment is a process of using the model file to perform real-time filtering on each video frame in the online video.
  • FIG. 1 is a flowchart of an image processing method provided by an embodiment of this application.
  • This embodiment can be applied to any server with image processing capabilities.
  • the solution of the embodiment of the present application can be applied to the problem of how to perform denoising processing on the current video frame under the condition of poor network conditions to improve the video definition.
  • the image processing method provided in this embodiment can be executed by the image processing device provided in the embodiment of the application, and the device can be implemented by software and/or hardware, and integrated into the device that executes the method.
  • the device can It is any server with image processing capabilities.
  • the method may include the following steps:
  • S110 Determine the image attribute of each image block of the multiple image blocks in the image to be processed.
  • the image attributes include: the first hash value and the second hash value; the first hash value is used to indicate the direction of the image gradient corresponding to each image block; the second hash value is used to indicate that each image block is in the first The proximity between the image gradient corresponding to one direction and the image gradient corresponding to each image block in the second direction.
  • each image block in the image to be processed may correspond to image gradients in two directions.
  • the description will be made assuming that the image block has image gradients in the x direction and the y direction.
  • the first hash value is used to indicate whether the image gradient corresponding to each image block focuses on the x direction or the y direction; the second hash value is used to indicate the corresponding image gradient of each image block in the x direction and the corresponding image gradient in the y direction.
  • the proximity between the image gradients corresponding to the directions is used to indicate whether the image gradient corresponding to each image block focuses on the x direction or the y direction.
  • determining the image attribute of each image block of the multiple image blocks in the image to be processed includes: according to the image gradient of each image block in the first direction and the second direction of each image block Image gradient, the square value of the image gradient corresponding to each image block in the first direction and the square value of the image gradient corresponding to each image block in the second direction are calculated, and each image block in the second direction
  • the product value of the image gradient corresponding to one direction and the image gradient corresponding to each image block in the second direction; the square value of the image gradient corresponding to the first direction and the square of the image gradient corresponding to the second direction are calculated by the integral graph algorithm Value, and the product value of the image gradient corresponding to the first direction and the image gradient corresponding to the second direction are calculated to obtain the first hash value and the second hash value corresponding to each image block.
  • FIG. 2 is a schematic diagram of displaying image block division of a to-be-processed image provided by an embodiment of the present application. As shown in Figure 2, suppose the size of the image to be processed is 7*7, that is, the image to be processed has 49 pixel values.
  • the size of each image block in the image to be processed is 5*5, and the image to be processed can be divided into 9 image blocks of size 5*5.
  • the image to be processed can be divided into 9 image blocks of size 5*5.
  • the number of image blocks is counted, the pattern fill area A can be recorded as the first image block of the image to be processed, and one pixel value is moved to the right or down in turn to get The new image block, and so on, the image to be processed can be divided into 9 image blocks with a size of 5*5.
  • S120 Determine a filter model corresponding to each image block according to the image attribute corresponding to each image block.
  • each image block may be determined according to the first hash value and the second hash value of each image block.
  • Filter model corresponding to each image block In one embodiment, a mapping relationship is established between the image attributes and the filter model in advance. After the image attributes of each image block are determined, the filter model corresponding to each image block can be found according to the mapping relationship between the two. .
  • S130 Filter each image block through a filter model corresponding to each image block to obtain a filtered image of the image to be processed.
  • the filter model is used to perform filtering processing on each image block to obtain the filtered image of each image block, thereby realizing that each frame of the image to be processed is
  • the adaptive filtering of the image block further improves the clarity of the image to be processed.
  • filtering the image block corresponding to the center point position (i, j) in the image to be processed is taken as an example to describe the filtering process of the image block.
  • the filter model h of the image block is determined according to the first hash value ⁇ and the second hash value ⁇ of the image block, and then the filter model h is used to Image block is filtered to get the target image
  • a k is the row vector of the image block expanded into a row, Is the transposed vector of the row vector, Is the pixel of the target image at position (i, j).
  • the filtering process of other image blocks in the image to be processed is the same as the above process, and will not be repeated here.
  • the target images corresponding to the multiple image blocks are synthesized together to obtain a filtered image of the image to be processed.
  • the image processing method provided by the embodiment of the present application determines the image attribute of each image block of the multiple image blocks in the image to be processed; determines the filter model corresponding to each image block according to the image attribute of each image block; The filter model corresponding to each image block filters each image block to obtain a filtered image of the image to be processed.
  • the embodiment of the application utilizes the rapid calculation of the image attribute of each image block in the image to be processed to speed up the filtering processing speed of the image block, thereby realizing the real-time filtering of the image, and performing the corresponding filtering model on each image block Adaptive filtering improves the clarity of the video.
  • the coding noise strength is used to indicate the strength of coding noise contained in the image to be processed.
  • the encoding noise intensity of the video to which the image to be processed belongs can be used as the encoding noise intensity of the image to be processed. Since the coding noise intensity of different videos may be different, the coding noise intensity of different images to be processed may also be different.
  • the filter model corresponding to each image block when determining the filter model corresponding to each image block, if the encoding noise intensity of the image to be processed and the image attribute corresponding to each image block in the image to be processed can be used together to determine the filter model corresponding to each image block , It will make the determined filtering model more accurate and more in line with the characteristics of each image block, so as to better realize the filtering of each image.
  • the method before determining the filter model corresponding to each image block according to the image attribute corresponding to each image block, the method further includes: determining the multiple corresponding to the image to be processed according to the coding noise intensity of the image to be processed Filter model; the determining the filter model corresponding to each image block according to the image attribute corresponding to each image block includes: according to the image attribute corresponding to each image block in the image to be processed, a plurality of corresponding to the image to be processed The filter model corresponding to each image block is determined in the filter model.
  • the method before determining the multiple filter models corresponding to the image to be processed according to the encoding noise intensity of the image to be processed, the method further includes: determining the encoding noise intensity of the image to be processed.
  • determining the encoding noise intensity of the image to be processed includes: determining the encoding noise intensity of the video to which the image to be processed belongs; determining the encoding noise intensity of the video to which the image to be processed belongs as the encoding noise intensity of the image to be processed .
  • the corresponding relationship between the encoding noise intensity level and the filter model can be established through the following filtering model training process, and then the encoding noise intensity level of the image to be processed can be determined according to the encoding noise intensity level of the image to be processed. Multiple filter models corresponding to the level of noise intensity.
  • the training process of the filtering model includes the following steps:
  • the training set may include multiple low-definition videos.
  • the multiple low-definition videos may be divided into a corresponding number of training subsets according to the level of coding noise intensity of the multiple low-definition videos.
  • the number of subintervals corresponding to the first hash values corresponding to different training subsets may be the same or different.
  • the number of subintervals corresponding to the second hash values corresponding to different training subsets may be the same or different.
  • the low-resolution images in the training set are classified according to the level of coding noise intensity.
  • the coding noise intensity can be divided into three levels: strong, medium, and weak.
  • the coding noise of the low-resolution images in the training set Intensity, the low-resolution images in the training set are divided into three training subsets, and then the low-resolution images in each training subset are used for training to obtain multiple filter models corresponding to multiple buckets in each training subset. Therefore, the coding noise intensity is also taken into consideration when training the filter model, thereby establishing the mapping relationship between the filter model and the coding noise intensity. Multiple filter models trained through the same training subset correspond to the same level of Encoding noise strength.
  • the first hash value and the second hash value of each image block in the image to be processed can be determined.
  • the bucket sequence number corresponding to the image block, and the filter model corresponding to the bucket sequence number among the multiple filter models is used as the filter model corresponding to the image block.
  • FIG. 3 is a flowchart of another image processing method provided by an embodiment of this application. This embodiment is described on the basis of the technical solution provided by the foregoing embodiment. In this embodiment, the process of determining the filtering model is mainly explained.
  • this embodiment may include the following steps:
  • S210 Determine the image attribute of each image block of the multiple image blocks in the image to be processed.
  • S220 Determine the bucket sequence number according to the first hash value and the second hash value.
  • each image block After obtaining the first hash value ⁇ and the second hash value ⁇ of each image block, the value ranges [0, 1] of the first hash value ⁇ and the second hash value ⁇ are equally divided into q ⁇ and q ⁇ subintervals. Therefore, each image block can obtain the corresponding bucket number according to its own ⁇ and ⁇ .
  • a one-to-one mapping relationship is established between the bucket number and the filtering model.
  • a bucket number corresponds to a filter model. After the bucket number of each image block is obtained, the corresponding filtering model can be found according to the mapping relationship between the two.
  • S240 Filter each image block through a filter model corresponding to each image block to obtain a filtered image of the image to be processed.
  • the technical solution provided by this embodiment finds the corresponding filter model through the bucket sequence number determined according to the first hash value and the second hash value of each image block, thereby realizing the Each image block corresponds to a filter model to perform adaptive filtering on each image block, thereby improving the clarity of the image.
  • FIG. 4 is a flowchart of another image processing method provided by an embodiment of the present application. In this embodiment, on the basis of the foregoing embodiment, the process of determining the filtering model is described.
  • the method includes the following steps:
  • S310 Determine the image attribute of each image block of the multiple image blocks in the image to be processed.
  • S320 Determine a filter model corresponding to each image block according to the image attribute of each image block.
  • S330 Determine a filter model of each image block according to the coding noise intensity of the image to be processed.
  • the coding noise strength is used to indicate the strength of coding noise contained in the image to be processed.
  • the coding noise in the entire image to be processed is not evenly distributed. If the image to be processed is divided into multiple image blocks, the intensity of coding noise contained in each image block is not the same.
  • the coding noise intensity of the more important coding noise of the multiple image blocks of the entire image to be processed can be used as the coding noise intensity of the image to be processed, such as ,
  • the encoding noise intensity of the image to be processed may be the average encoding noise intensity of all image blocks in the entire image to be processed.
  • the coding noise intensity of the entire image to be processed can also be determined according to the coding noise intensity of each image block in other ways, which is not limited.
  • the method for determining the encoding noise intensity of the image to be processed includes: obtaining the quality parameter of the video encoding corresponding to the image to be processed; and determining the encoding noise intensity of the image to be processed according to the quality parameter.
  • the quality parameter includes at least one of the following parameters: quantization parameter (Quantization Parameter, QP) and constant rate factor (Constant Ratio Factor, CRF).
  • QP Quantization Parameter
  • CRF Constant Ratio Factor
  • the distortion and quality of the image to be processed in the video are strengthened;
  • the constant rate coefficient is also a video coding parameter, similar to QP, but QP is a part of the coding process
  • CRF is a constant quality coding method, that is, from the perspective of video quality, a certain CRF corresponds to a constant video quality.
  • the value range of CRF is generally 18-28, and in x264 encoder, the default value of CRF is 23; in x265 encoder, the default value of CRF is 28.
  • the QP and CRF of the video encoding corresponding to the image to be processed can be obtained from the output log of the performance test.
  • the preset mapping relationship between the encoding noise intensity and the quality parameter is used to determine the encoding noise intensity corresponding to the video encoding. It is also possible to use one of the multiple quality parameters of the video encoding to determine the corresponding encoding noise intensity, that is, to determine the encoding noise intensity corresponding to the video encoding through QP or CRF. Exemplarily, taking CRF of 18 and 23 as an example, the range of coding noise intensity is divided.
  • the coding noise intensity is at a weak stage; 18 ⁇ CRF ⁇ 23, the coding noise strength is at a medium stage; CRF>23, the coding noise strength is at a strong stage.
  • Different coding noise intensity corresponds to different filtering models. It is also possible to establish a mapping relationship between QP and coding noise intensity, or a mapping relationship between QP and CRF and coding noise intensity. Then, according to the mapping relationship between the different quality parameters and the encoding noise intensity, the encoding noise intensity corresponding to the video code corresponding to the image to be processed is determined, that is, the encoding noise intensity of the image to be processed is determined.
  • the filter model of the image block is determined according to the overall coding noise intensity of the image to be processed.
  • the filtering model of each image block determined according to the coding noise intensity is the same.
  • S340 Filter each image block through a filter model corresponding to each image block to obtain a filtered image of the image to be processed.
  • the filter model After determining the filter model of each image block according to the coding noise intensity of the image to be processed, the filter model is used to perform adaptive filtering on the entire image to be processed. Then determine the corresponding filter model according to the image attribute of each image block, and perform adaptive filtering on each image block in the image to be processed according to the filter model, and then synthesize the filtered image of each image block to obtain Process the filtered image of the image.
  • the entire image to be processed can be adaptively filtered according to the filter model determined by the coding noise intensity of each frame of the image to be processed, and then the filter model determined according to the image attributes of each image block in the image to be processed
  • Each image block is adaptively filtered; it is also possible to perform adaptive filtering on each image block according to the filter model determined by the image attribute of each image block in the image to be processed, and then according to the coding noise intensity of each frame of image to be processed
  • the determined filtering model performs adaptive filtering on the entire image to be processed, and the order of the two steps is not limited.
  • Step S330 can be executed before step S310 or S320, as long as it is ensured that step S330 is executed before step S340.
  • the technical solution of this embodiment determines the filter model according to the coding noise intensity of the image to be processed, so as to adaptively filter each frame of image to be processed, thereby improving the definition of the video.
  • the process of adaptively filtering the image to be processed is equivalent to the process of weighted averaging the pixel values of an image block. Therefore, if the image to be processed contains white text areas such as subtitles, the white text areas will have a lightening effect. In order to minimize the impact of code denoising on the white text area, the white text area can be binarized to obtain a binarized image.
  • the method further includes: performing binarization processing on the image to be processed to obtain a binarized image of the image to be processed.
  • Performing binarization processing on the image to be processed to obtain the binarized image of the image to be processed includes steps S100-S300:
  • S100 Perform Laplacian filtering on the image to be processed to obtain a text edge image.
  • the image to be processed is the original low-resolution image, denoted as y.
  • Laplacian filtering is performed on the original low-resolution image to obtain a text edge image, which is denoted as T1.
  • T1 a text edge image
  • the RGB value 0
  • the larger the RGB value the greater the possibility that the position is the text edge.
  • the text edges of the filtered image at this time are not clear, and there are gaps and gaps, and there are some other non-text edge areas in the text edge image.
  • S200 Perform threshold segmentation on the text edge image based on a preset pixel threshold to obtain a first image.
  • the first image is an image with non-text edge regions in the text edge image removed.
  • the edges in the text area are sharper, and the pixel value of the text area obtained after Laplacian filtering is relatively large, the pixel value corresponding to the non-text area is relatively small. Therefore, at this time, you can set a pixel threshold, denoted as th, and then perform threshold segmentation on the text edge image, trying to remove the non-text edge area in the text edge image, so as to obtain an image that removes the non-text edge area in the text edge image. Denoted as the first image T2.
  • Step S200 may include steps S2002-S2008:
  • S2008 Determine the text edge image after adjusting the pixel value as the first image.
  • the text edge image after adjusting the pixel value is determined as the first image, that is, the pixel value corresponding to the pixel point greater than the pixel threshold is adjusted to the first pixel value, and the pixel corresponding to the pixel point less than or equal to the pixel threshold is adjusted The value is adjusted to the second pixel value to obtain the first image.
  • the mathematical expression of threshold segmentation can be set as:
  • T1(i,j) is the pixel value of the text edge image corresponding to the image block at position (i,j) in the image to be processed; T2(i,j) is (i,j) in the image to be processed
  • the pixel values of all pixels in the text edge image are adjusted to 0 or 255 to remove the non-text edge area to obtain the first image.
  • the first image is also a binary image, but the binary image is not clear and contains discontinuous text edges.
  • S300 Perform a morphological closing operation on the first image to obtain a binary image.
  • the morphological closing operation can eliminate narrow discontinuities and eliminate small holes.
  • the shape close operation has the function of filling small holes in objects, connecting neighboring objects and smoothing boundaries.
  • the process of performing a morphological closing operation on the first image includes: performing an expansion operation on the first image to obtain an expanded image; performing an erosion operation on the expanded image to obtain a binary image.
  • a morphological closing operation is performed on the first image T2 to obtain a binary image, denoted as T3, so that the edges of unclear and discontinuous text regions can be completely connected.
  • After obtaining the binarized image it further includes: performing average filtering on the binarized image. After the binarized image is obtained, mean filtering is performed on the binarized image, which can smooth the sharp edges of the text area and make the transition smooth.
  • FIG. 5 is a flowchart of yet another image processing method provided by an embodiment of the present application.
  • the method includes the following steps:
  • S410 Determine the image attribute of each image block in the multiple image blocks in the image to be processed.
  • S420 Determine a filter model corresponding to each image block according to the image attribute of each image block.
  • S430 Determine a filter model of each image block according to the coding noise intensity of the image to be processed.
  • S440 Filter each image block through a filter model corresponding to each image block to obtain a filtered image of the image to be processed.
  • S450 Perform Laplacian filtering on the image to be processed to obtain a text edge image.
  • step S460 Determine whether the pixel value of each pixel in the text edge image is greater than a preset pixel threshold, and in response to the judgment result that the pixel value of each pixel in the text edge image is greater than the preset pixel threshold, execute the step S470: In response to the determination result that the pixel value of each pixel in the text edge image is less than or equal to the preset pixel threshold, perform step S480.
  • S480 Adjust the pixel value corresponding to the pixel point less than or equal to the pixel threshold to the second pixel value.
  • S490 Determine the text edge image after adjusting the pixel value as the first image.
  • S4100 Perform a morphological closing operation on the first image to obtain a binary image.
  • the synthesis formula of the denoising image may be: Among them, y represents the pixel value of the original low-resolution image, T is the pixel value of the binarized image T3, Is the pixel value of the denoised image of the image to be processed, Is the first weight, It is the second weight.
  • T is the pixel value of the binarized image T3
  • Is the pixel value of the denoised image of the image to be processed Is the first weight
  • It is the second weight.
  • the pixel value of the binarized image is 255, it means that the denoised image of the image to be processed is the original low-definition image; when the pixel value of the binarized image is 0, it means that the denoised image of the image to be processed is adaptive The filtered image obtained by filtering.
  • the step of performing Laplacian filtering on the image to be processed to obtain a binarized image may be performed simultaneously with the step of obtaining a filtered image of the image to be processed, namely steps S410-S440. It can be understood that, as long as steps S410-S440 and S450-S4100 are executed before step S41110, the execution order of steps S410-S440 and S450-S4100 is not limited.
  • the technical solution of this embodiment reduces the impact of coding denoising on the text area as much as possible by binarizing the text area in the image to be processed, and avoids the phenomenon of the text area being faded, thereby ensuring that the image in the video is clear At the same time, it can also ensure that the video subtitles are clean and clear.
  • FIG. 6 is a flowchart of filtering processing on image blocks in an image processing method provided by an embodiment of the present application. On the basis of the foregoing embodiments, the process of filtering each image block in the image processing method is explained.
  • the image processing method is applied to a mobile terminal as an example for description.
  • the mobile terminal may be a smart phone, an ipad, a notebook computer, and the like.
  • the method includes the following steps:
  • S530 Find a filter model corresponding to the bucket number according to the first hash value and the second hash value of each image block.
  • S550 Synthesize the target images of each image block together to obtain a filtered image of the image to be processed.
  • the technical solution of this embodiment improves the calculation efficiency of the first hash value and the second hash value of the image block through the definition of the new image local features and the fast calculation method, so that the mobile terminal can be more portable
  • the encoding noise of the current video frame in the video can be removed in real time, as well as any type of encoding noise in the video.
  • the intensity of coding noise will change with changes in network conditions.
  • coding noises of different intensities can be processed.
  • training the filter model training sets with different coding noise intensities can be constructed and trained separately to obtain multiple filter models. Therefore, in the actual operation of image processing, different filtering models can be used for filtering according to the quality parameters of the video encoding (for example, QP and/or CRF).
  • the embodiment of the present application provides a video processing method for a mobile terminal, and this embodiment can be applied to any mobile terminal with a video processing function.
  • the mobile terminal video processing method provided in this embodiment can be executed by the mobile terminal video processing device provided in this embodiment of the application.
  • the device can be implemented in software and/or hardware, and is integrated in the device that executes the method.
  • the device can be any mobile terminal with video processing capabilities.
  • FIG. 7 is a flowchart of a method for processing a video on a mobile terminal according to an embodiment of the present application. As shown in FIG. 7, the method includes:
  • S620 Extract video frames from the video to be processed.
  • S630 Process the video frame as the image to be processed to obtain a filtered image corresponding to the video frame.
  • S640 Perform video playback according to the filtered image.
  • an application (APP) on the mobile terminal needs to pull the video stream from the server, obtain and decode the video to be processed in the video stream, and then identify and extract the video to be processed
  • the current video frame, the current video frame is used as the image to be processed, and the image processing method described in any of the above embodiments is executed to filter and denoise the current video frame to obtain a filtered image of the current video frame, and then denoise
  • the image updates the corresponding current video frame in the video to be processed, and plays the video, which improves the clarity of the video.
  • the image processing method of the foregoing embodiment is fast in calculation and has high denoising efficiency, it can be used on mobile terminals. Real-time processing requirements for video frames and playback have improved the functionality of mobile terminals and the viewing experience of users.
  • the technical solution of this embodiment by adaptively filtering the video frames in the video stream, can not only eliminate the coding noise better, but also adapt to the current network conditions.
  • the subtitles are binarized to keep the subtitles clean and clear, which improves the user's viewing experience.
  • the embodiments of the present application improve the denoising efficiency of the video, realize the real-time denoising of the video, and improve the clarity when watching the video.
  • FIG. 8 is a schematic structural diagram of an image processing device provided by an embodiment of the application. As shown in FIG. 8, the device may include: a first determining module 710, a second determining module 720, and a first filtering module 730.
  • the first determining module 710 is configured to determine the image attribute of each image block of the multiple image blocks in the image to be processed; the second determining module 720 is configured to determine the corresponding image block of each image block according to the image attribute of each image block The first filtering module 730 is configured to filter each image block through the filter model corresponding to each image block to obtain a filtered image of the image to be processed.
  • the technical solution provided by this embodiment determines the image attribute of each image block of the multiple image blocks in the image to be processed; determines the filter model corresponding to each image block according to the image attribute of each image block; The filter model corresponding to the image block filters each image block to obtain a filtered image of the image to be processed.
  • the embodiment of the application utilizes the rapid calculation of the image attribute of each image block in the image to be processed to speed up the filtering processing speed of the image block, thereby realizing the real-time filtering of the image, and performing the corresponding filtering model on each image block Adaptive filtering improves the clarity of the video.
  • the image attributes include: a first hash value and a second hash value; the first hash value is used to indicate the focus direction of the image gradient corresponding to each image block; the second hash value is used Yu represents the proximity of each image block in the first direction and the corresponding image gradient in the second direction.
  • the first determining module 710 is configured to determine the image attribute of each image block of the multiple image blocks in the image to be processed in the following manner, including: according to the first direction and the first direction of each image block Image gradients in two directions, the square value of the image gradient corresponding to each image block in the first direction, the square value of the image gradient corresponding to each image block in the second direction, and each image The product value of the image gradient corresponding to the block in the first direction and the image gradient corresponding to each image block in the second direction; the square value of the image gradient corresponding to the first direction, the second The square value of the image gradient corresponding to the direction, and the product value of the image gradient corresponding to the first direction and the image gradient corresponding to the second direction are calculated to obtain the corresponding first hash value and second hash value .
  • the second determining module 720 includes: a first determining unit configured to determine a bucket sequence number according to the first hash value and the second hash value corresponding to each image block; The second determining unit is configured to determine the filter model corresponding to each image block according to the bucket serial number.
  • the image processing device further includes: a third determining module configured to determine the filtering model of each image block according to the coding noise intensity of the image to be processed.
  • the first filtering module 730 is configured to filter each image block by the target model corresponding to each image block to obtain a filtered image of the image to be processed, wherein each image
  • the target model corresponding to the block includes the filter model of each image block determined according to the coding noise intensity of the image to be processed and the filter model corresponding to each image block determined according to the image attribute of each image block .
  • the method for determining the encoding noise intensity of the image to be processed includes: acquiring the quality parameter of the video encoding corresponding to the image to be processed; and determining the encoding noise intensity of the image to be processed according to the quality parameter.
  • the image processing device further includes: a processing module configured to perform a binarization process on the image to be processed to obtain a binarized image of the image to be processed.
  • the processing module includes: a filtering unit, which is configured to perform Laplacian filtering on the image to be processed to obtain a text edge image; and the segmentation unit, which is configured to perform processing on the text edge image based on a preset pixel threshold Perform threshold segmentation to obtain a first image, the first image is an image with non-text edge regions in the text edge image removed; the closing operation unit is set to perform a morphological closing operation on the first image to obtain a binary image.
  • the image processing device further includes: a second filtering module, configured to, after obtaining the binarized image, further includes: performing average filtering on the binarized image.
  • the closed operation unit includes: a first operation subunit configured to perform an expansion operation on the first image to obtain an expanded image; and a second operation subunit configured to perform an erosion operation on the expanded image to Obtain a binary image.
  • the segmentation unit includes: a comparison subunit configured to compare the pixel value corresponding to each pixel in the text edge image with a preset pixel threshold; a first adjustment subunit, Set to adjust the pixel value corresponding to the pixel point greater than the pixel threshold value to the first pixel value; the second adjustment sub-unit is set to adjust the pixel value corresponding to the pixel point less than or equal to the pixel threshold value to the second pixel value; The unit is set to determine the text edge image after adjusting the pixel value as the first image.
  • the image processing device further includes: a first weight determination module configured to use the ratio between the first pixel value and the pixel value of the binary image as the first pixel value A weight; the second weight determination module is set to use the ratio between the pixel value of the binary image and the first pixel value as the second weight; the denoising module is set to the product of the pixel value of the filtered image and the first weight The value is added to the product value of the pixel value of the image to be processed and the second weight to obtain a denoised image.
  • a first weight determination module configured to use the ratio between the first pixel value and the pixel value of the binary image as the first pixel value A weight
  • the second weight determination module is set to use the ratio between the pixel value of the binary image and the first pixel value as the second weight
  • the denoising module is set to the product of the pixel value of the filtered image and the first weight The value is added to the product value of the pixel value of the image to be processed and the second weight to obtain a denoised image
  • the process of training the filter model includes: acquiring a high-definition image of a first preset size and a low-definition image carrying coding noise, the resolution of the high-definition image and the low-definition image are the same; And low-definition images extract multiple image blocks of the second preset size to form a first matrix and a second matrix respectively, the second preset size is smaller than the first preset size; based on the normal equation of the least squares method, and according to The first matrix and the second matrix obtain the filtering model.
  • the size of the filter model is the same as the size of the image block.
  • the image processing device further includes: a fourth determining module configured to determine multiple filter models corresponding to the image to be processed according to the coding noise intensity of the image to be processed; the second determining module 720 is It is configured to determine the filter model corresponding to each image block from the multiple filter models corresponding to the image to be processed according to the image attribute of each image block in the image to be processed.
  • the image processing device provided in this embodiment may be applicable to the image processing method provided in any embodiment of this application.
  • FIG. 9 is a schematic structural diagram of a video processing device for a mobile terminal according to an embodiment of the present application. As shown in FIG. 9, the device includes: a decoding module 810, an extraction module 820, a processing module 830, and a playback module 840.
  • the decoding module 810 is set to decode the to-be-processed video pulled from the video stream; the extraction module 820 is set to extract the video frames in the to-be-processed video; the processing module 830 is set to use the image processing described in any of the above embodiments
  • the method processes the video frame as the image to be processed to obtain the corresponding filtered image; the playback module 840 is configured to perform video playback according to the filtered image.
  • the technical solution provided by this embodiment improves the denoising efficiency of the video by performing denoising processing on the current video frame in the video, realizes the real-time denoising of the video, and improves the clarity of the video when watching, thereby improving The user’s viewing experience.
  • the mobile terminal video processing device provided in this embodiment may be applicable to the mobile terminal video processing method provided in any embodiment of this application.
  • FIG. 10 is a schematic structural diagram of a device provided by an embodiment of the application.
  • the device includes a processor 910, a storage device 920, an input device 930, and an output device 940.
  • the processor 910 in the device may be one or more. In FIG. 10, one processor 910 is taken as an example.
  • the processor 910, the memory 920, the input device 930, and the output device 940 in the device may be connected by a bus or other means. In Figure 10, the bus connection is taken as an example.
  • the memory 920 in the device can be used to store one or more programs.
  • the programs can be software programs, computer-executable programs, and modules, such as those corresponding to the image processing method provided in the embodiments of this application.
  • Program instructions/modules (for example, the modules in the image processing device shown in FIG. 8 include: a first determining module 710, a second determining module 720, and a first filtering module 730).
  • the processor 910 executes various functional applications and data processing of the device by running software programs, instructions, and modules stored in the memory 920, that is, implements the image processing method in the foregoing method embodiment.
  • the memory 920 may include a program storage area and a data storage area.
  • the program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created according to the use of the device and the like.
  • the memory 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.
  • the memory 920 may further include a memory remotely provided with respect to the processor 910, and these remote memories may be connected to the device through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
  • the input device 930 may be configured to receive numeric or character information input by the user to generate key signal input related to user settings and function control of the terminal device.
  • the output device 940 may include a display device such as a display screen.
  • the program when one or more programs included in the above-mentioned device are executed by one or more processors 910, the program performs the following operations: Determine the image attribute of each of the multiple image blocks in the image to be processed; The image attribute of the image block determines the filter model corresponding to each image block; each image block is filtered through the filter model corresponding to each image block to obtain a filtered image of the image to be processed.
  • the device provided in this embodiment can be configured to execute the image processing method provided in any embodiment of this application.
  • the device provided in this embodiment can be configured to execute the video processing method of a mobile terminal provided in any embodiment of this application.
  • the embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored.
  • the image processing method or the mobile terminal video processing method in any embodiment of the present application can be implemented.
  • the image processing method may include: determining the image attribute of each image block of the plurality of image blocks in the image to be processed; determining the filter model corresponding to each image block according to the image attribute of each image block; The corresponding filtering model filters each image block to obtain the filtered image of the image to be processed
  • the mobile terminal video processing method includes: decoding a to-be-processed video pulled from a video stream; extracting video frames from the to-be-processed video; processing the video frame as a to-be-processed image by using the image processing method described in any of the above embodiments , Obtain the filtered image corresponding to the video frame; Perform video playback according to the filtered image of the video frame.
  • the embodiment of the application provides a storage medium containing computer executable instructions.
  • the computer executable instructions are not limited to the method operations described above, and can also execute the image processing method or mobile terminal video provided by any embodiment of the application. Related operations in the processing method.
  • the present application can be implemented by software and necessary general-purpose hardware, or can be implemented by hardware, but the former is a better implementation in many cases.
  • the technical solution of this application can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), Random Access Memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute this application.
  • ROM read-only memory
  • RAM Random Access Memory
  • FLASH flash memory
  • hard disk or optical disk etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

本申请公开了一种图像处理和移动终端视频处理方法、装置、设备和介质，该图像处理方法包括：确定待处理图像中多个图像块的每个图像块的图像属性；根据每个图像块的图像属性确定每个图像块对应的滤波模型；通过每个图像块对应的滤波模型对所述每个图像块进行滤波，得到待处理图像的滤波图像。

Description

图像处理和移动终端视频处理方法、装置、设备和介质
本申请要求在2019年07月23日提交中国专利局、申请号为201910667696.6的中国专利申请，以及在2020年04月28日提交中国专利局、申请号为202010348909.1的中国专利申请的优先权，上述申请的全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及图像处理领域，例如涉及一种图像处理和移动终端视频处理方法、装置、设备和介质。
背景技术
视频在保存和传输过程中，一般都会使用有损压缩的手段进行编码。在视频解码后，会产生编码噪声。编码噪声的类型包括块效应、振铃效应和锯齿效应。例如，当用户观看视频时，移动端视频应用程序从服务器拉取视频流，在本地对视频流解码并播放，由于网络环境的限制，移动端拉取到的视频流往往是经过高度压缩的视频流，在用户在观看视频时，一般会出现可见的编码噪声，这种编码噪声在网络条件较差时更为常见。
在图像/视频去噪领域，常见的方法是去除由摄像头感光器产生的高斯白噪声。也有一些方法用于去除编码噪声，但往往只针对编码噪声中的指定类型如块效应进行处理；此外，近年来业内也有探索基于神经网络的编码噪声去除算法，虽然对编码噪声去除有一定效果，但神经网络的计算量非常大，实时性差。
发明内容
本申请实施例提供了一种图像处理和移动终端视频处理方法、装置、设备和介质，实现了对图像的实时去噪，以及对每个图像块的自适应滤波，进而提高了视频的清晰度。
本申请实施例提供了一种图像处理方法，包括：
确定待处理图像中多个图像块的每个图像块的图像属性；
根据每个图像块的图像属性确定所述每个图像块对应的滤波模型；
通过每个图像块对应的所述滤波模型对所述每个图像块进行滤波，得到所述待处理图像的滤波图像。
本申请实施例提供了一种移动终端视频处理方法，包括：
解码从视频流中拉取的待处理视频；
抽取所述待处理视频中的视频帧；
采用上述任意实施例所述的方法处理作为待处理图像的所述视频帧，得到所述视频帧对应的滤波图像；
根据所述滤波图像，进行视频播放。
本申请实施例提供了一种图像处理装置，包括：
第一确定模块，设置为确定待处理图像中多个图像块的每个图像块的图像属性；
第二确定模块，设置为根据每个图像块的图像属性确定所述每个图像块对应的滤波模型；
第一滤波模块，设置为通过每个图像块对应的所述滤波模型对所述每个图像块进行滤波，得到所述待处理图像的滤波图像。
本申请实施例提供了一种移动终端视频处理装置，包括：
解码模块，设置为解码从视频流中拉取的待处理视频；
抽取模块，设置为抽取所述待处理视频中的视频帧；
处理模块，设置为采用上述任意实施例所述的方法处理作为待处理图像的所述视频帧，得到所述视频帧对应的滤波图像；
播放模块，设置为根据所述滤波图像，进行视频播放。
本申请实施例提供了一种设备，该设备包括：
一个或多个处理器；
存储器，用于存储一个或多个程序；
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现本申请任意实施例所述的图像处理方法，或者实现本申请任意实施例所述的移动终端视频处理方法。
本申请实施例提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现本申请任意实施例所述的图像处理方法，或者实现本申请任意实施例所述的移动终端视频处理方法。
附图说明
图1为本申请实施例提供的一种图像处理方法的流程图；
图2是本申请实施例提供的一种待处理图像进行图像块划分的显示示意图；
图3为本申请实施例提供的另一种图像处理方法的流程图；
图4是本申请实施例提供的又一种图像处理方法的流程图；
图5是本申请实施例提供的再一种图像处理方法的流程图；
图6是本申请实施例提供的一种图像处理方法中对图像块进行滤波处理的流程图；
图7是本申请实施例提供的一种移动终端视频处理方法的流程图；
图8为本申请实施例提供的一种图像处理装置的结构示意图；
图9是本申请实施例提供的一种移动终端视频处理装置的结构示意图；
图10为本申请实施例提供的一种设备的结构示意图。
具体实施方式
下面结合附图和实施例对本申请进行说明。此处所描述的实施例仅仅用于解释本申请，而非对本申请的限定。另外为了便于描述，附图中仅示出了与本申请相关的部分而非全部结构。
本申请实施例可适用于在用户观看视频时，由于所处环境的网络限制，一般会出现可见的编码噪声，需要去除编码噪声的情况。本申请实施例提供了一种图像处理方法，实现了对图像的实时滤波，以及通过对应的滤波模型对每个图像块进行自适应滤波，提高了视频的清晰度。
在执行图像处理方法之前，对滤波模型进行训练，从而在执行图像处理方法时，可根据待处理图像的编码噪声强度以及待处理图像中图像块的图像属性，确定对应的滤波模型，并对图像块进行滤波处理。对滤波模型进行训练的过程包括：获取第一预设尺寸的高清图像和携带编码噪声的低清图像，高清图像和低清图像的分辨率相同；分别从所述高清图像和低清图像中抽取多个第二预设尺寸的图像块，以分别组成第一矩阵和第二矩阵，所述第二预设尺寸小于所述第一预设尺寸；按照最小二乘法的正规方程，并根据第一矩阵和第二矩阵，得到滤波模型。
可选的，本申请实施例中的滤波模型可以基于最小二乘法计算得到。假设有L张MxN的高清图像，并用x i表示第i张高清图像，i＝1,2,...,L，M和N用 于表示高清图像的尺寸，并均为正数，比如，M表示高清图像的长度，N表示高清图像的宽度；L为大于等于1的正整数。同时，获取与所述高清图像等分辨率但携带编码噪声的图像(记为低清图像)，并用y i表示第i张低清图像，i＝1,2,...,L。同时，假设训练一个d×d的滤波模型h，将低清图像y i中的像素值，调整为高清图像x i中对应的像素值。d×d用于表示滤波模型的尺寸。在实施例中，d×d的滤波模型h，指的是滤波操作过程中，输入的原始图像的尺寸为d×d。
训练滤波模型的原则是求解以下最小二乘问题，需求解公式为：
Figure PCTCN2020101243-appb-000001
公式(1)中，
Figure PCTCN2020101243-appb-000002
是h∈R d×d的向量化表示，
Figure PCTCN2020101243-appb-000003
表示d 2维的实数向量的集合，h为实数向量的集合中的一个实数向量元素。
Figure PCTCN2020101243-appb-000004
是一个矩阵，由低清图像y i中抽取出的尺寸为d×d的图像块组成，抽取的方式是：针对每一个像素，抽取以这个像素为中心的d×d的一个图像块，把d×d的图像块拉伸为一行，作为矩阵A i的一行；
Figure PCTCN2020101243-appb-000005
表示MN行，d 2列的实数矩阵的集合，A i为实数矩阵的集合中的一个实数矩阵。b i∈R MN，由x i的像素展开构成，其中，b i中的每个元素对应低清图像y i中多个d×d的图像块的中心位置的像素，R MN表示MN维的实数向量的集合，b i为实数向量的集合R MN中的一个实数向量元素。A i和b i均对应第i张低清图像，i＝1,2,...,L，L为大于等于1的正整数。由于该最优滤波模型的训练方法由一张图可以很容易扩展到多张图，为了便于说明，可将最优滤波模型的训练方法限定在一张图内，并省略角标i。可以理解为，滤波模型的尺寸和图像块的尺寸相同，即滤波模型的尺寸为d×d，则图像块的尺寸也为d×d。
由公式(1)可知，A是一个非常大的矩阵，空间复杂度将达到O(MNd 2)。因此，利用最小二乘法的正规方程对公式(1)进行转变，即可将对公式(1)的求解转变为对公式(2)的求解
Figure PCTCN2020101243-appb-000006
公式(2)中，Q＝A TA，V＝A Tb。可以得出，Q是一个d 2xd 2的矩阵，V是一个d 2维向量。令A j表示A的第j行，
Figure PCTCN2020101243-appb-000007
则Q和V都可以通过小的矩阵/向量累加而成。
Figure PCTCN2020101243-appb-000008
表示1行，d 2列的实数矩阵的集合，A j为实数矩阵的集合
Figure PCTCN2020101243-appb-000009
中的一个实数矩阵。
Figure PCTCN2020101243-appb-000010
Figure PCTCN2020101243-appb-000011
可以得到，利用公式(3)和公式(4)的累加方法，公式(2)的空间复杂度是O(d 4)，并且，为了便于对图像中每个图像块的像素值进行统计，可选的，将d的取值设置在5-13之间。因此，公式(2)的空间复杂度也不是很大。
利用最小二乘法的原理，可将公式(2)转换为求解方程：Qh＝V，方程的解h是最小二乘意义下的滤波模型。虽然，利用滤波模型h对一张低清图像进行滤波，可以生成尽可能去除编码噪声的高清图像，但该滤波模型难以针对图像内容做出自适应调整。因此，对低清图像中的每个图像块进行哈希分类(hashing)，并对哈希分类的每个桶(bucket)，各自训练一个滤波模型。从而，在对待处理图像进行滤波处理的实际操作中，首先对当前将要进行滤波操作的低清图像(即待处理图像)的图像块，计算出对应的第一哈希值和第二哈希值，根据第一哈希值和第二哈希值找到对应的桶序号，用该桶序号所对应的滤波模型进行滤波。
示例性地，针对一个d*d的图像块，对图像块的第一哈希值和第二哈希值的计算过程进行说明。第一哈希值可以用θ来表示，第二哈希值可以用μ来表示。并且，θ可以采用公式(5)进行计算得到，μ可以采用公式(6)进行计算得到。
Figure PCTCN2020101243-appb-000012
Figure PCTCN2020101243-appb-000013
公式(5)和公式(6)中，LGx和LGy表示图像块的x方向和y方向的图像梯度；i的取值为1-d中的任意一个数，j的取值为1-d中的任意一个数，LGx 2表示每个图像块在x方向所对应图像梯度的平方值，LGy 2表示每个图像块在y方向所对应图像梯度的平方值，LGxLGy表示每个图像块在x方向和y方向所对应图像梯度的乘积值。θ和μ的取值范围都在[0,1]范围内。将区间[0,1]分别平均划分为q θ和q μ个子区间，q θ为第一哈希值对应的子区间个数，q μ为第二哈希值对应的子区间个数，q θ和q μ的值可以自定义设置。然后，将从q θ个子区间任意取的一个子区间和从q μ个子区间中任意取的一个子区间组成的组合称为一个桶 (bucket)，并且，每个图像块均会落在一个确定的桶内。为了保证落在每个桶内的所有图像块均对应有一个滤波模型，在对滤波模型进行训练时，需对每个桶内的所有图像块进行滤波模型的训练，最终可以得到q θ*q μ个滤波模型。相应地，在求解过程中，需构造q θ*q μ个矩阵Q和向量V，并求解对应的q θ*q μ个最小二乘方程。。
在实施例中，滤波模型的训练过程包括如下步骤：
S10、对训练集中的低清图像，遍历所有尺寸为dxd的图像块。
S20、计算出低清图像中每个图像块的第一哈希值θ和第二哈希值μ，以得到对应的桶序号。
一实施例中，例如q θ＝2，q μ＝4，则第一哈希值对应的子区间为[0,0.5)和[0.5,1]，第二哈希值对应的子区间[0,0.25)，[0.25,5)，[0.5,0.75)和[0.75,1]，则从第一哈希值对应的子区间和第二哈希值对应的子区间中各选取一个子区间，选择的两个子区间的组合构成一个桶，因此可以有8个桶，如[0,0.5)和[0,0.25)对应的桶序号为1，[0,0.5)和[0.25,5)对应的桶序号为2，[0,0.5)和[0.5,0.75)对应的桶序号为3，[0,0.5)和[0.75,1]对应的桶序号为4，[0.5,1]和[0,0.25)对应的桶序号为5，[0.5,1]和[0.25,5)对应的桶序号为6，[0.5,1]和[0.5,0.75)对应的桶序号为7，[0.5,1]和[0.75,1]对应的桶序号为8。则若一个图像块的第一哈希值θ落在子区间[0.5,1]和第二哈希值μ落在子区间[0,0.25)，则该图像块对应的桶序号为5。
S30、利用公式(3)，将当前图像块的像素值展开成一行(即A j)，将
Figure PCTCN2020101243-appb-000014
累加到该桶序号对应的矩阵Q上，并根据图像块的位置，查找到对应高清图像对应位置的像素值b j，将
Figure PCTCN2020101243-appb-000015
累加到该桶序号对应的矩阵V上。
S40、在遍历完所有低清图像的图像块后，得到每个桶最终的Q和V，解方程Qh＝V，即可得到该桶对应的最优滤波模型h。
步骤S20中，对每个图像块的θ和μ进行计算的过程，若按照“先遍历图像块，再按照公式求θ和μ”的顺序，则时间复杂度为O(MNd 2)。但由于θ和μ的计算公式中，对图像块的每个元素进行的操作都是无差别的相加，此时，可以利用积分图算法进行简化，将时间复杂度简化到O(MN)。第一哈希值和第二哈希值的计算步骤如下：
首先，利用待处理图像的原图，求得每张待处理图像在x方向和y方向的图像梯度，即梯度图Gx和Gy；然后，根据梯度图Gx和Gy求得每张待处理图像在x方向所对应图像梯度的平方值Gx 2(Gx 2表示Gx的平方，即(Gx) 2的简写)，在y方向所对应图像梯度的平方值Gy 2(Gy 2表示Gy的平方，即(Gy) 2的简写)， 以及在x方向和y方向所对应图像梯度的乘积值Gxy(Gxy表示Gx和Gy的乘积值，即GxGy的简写)；然后利用Gx 2、Gy 2和Gxy的积分图算法，得到待处理图像的积分图(积分图指的是用二维矩阵表示的一个图，用于表示图像的积分)
Figure PCTCN2020101243-appb-000016
Figure PCTCN2020101243-appb-000017
和I Gxy，使得
Figure PCTCN2020101243-appb-000018
Figure PCTCN2020101243-appb-000019
通过
Figure PCTCN2020101243-appb-000020
和I Gxy计算待处理图像的Gx 2、Gy 2和Gxy的局部和
Figure PCTCN2020101243-appb-000021
和LS Gxy，使得
Figure PCTCN2020101243-appb-000022
Figure PCTCN2020101243-appb-000023
LS Gxy(i,j)＝I Gxy(i+l)-I Gxy(i-l)，其中
Figure PCTCN2020101243-appb-000024
计算出以(i,j)为中心位置的待处理图像中每个dxd的图像块的第一哈希值θ和第二哈希值μ：
Figure PCTCN2020101243-appb-000025
对滤波模型进行训练的过程，是离线地训练一个画质增强的模型，即将低清图像提升为高清图像的模型。该模型是一个模型文件。然后，可通过该模型文件对线上视频进行实时滤波，以得到画质增强的视频。本实施例中的图像处理方法即为采用该模型文件对线上视频中的每个视频帧进行实时滤波的过程。
图1为本申请实施例提供的一种图像处理方法的流程图，本实施例可应用于任一种具备图像处理能力的服务端中。本申请实施例的方案可以适用于如何在网络条件较差的情况下，对当前视频帧进行去噪处理，以提高视频清晰度的问题。本实施例提供的一种图像处理方法可以由本申请实施例提供的图像处理装置来执行，该装置可以通过软件和/或硬件的方式来实现，并集成在执行本方法的设备中，该设备可以是任一具备图像处理能力的服务器。
参考图1，该方法可以包括如下步骤：
S110、确定待处理图像中多个图像块的每个图像块的图像属性。
图像属性包括：第一哈希值和第二哈希值；第一哈希值用于表示每个图像块所对应图像梯度的侧重方向；第二哈希值用于表示每个图像块在第一方向所对应图像梯度和所述每个图像块在第二方向所对应图像梯度之间的接近度。
一实施例中，待处理图像中的每个图像块可以对应有两个方向的图像梯度。在本实施例中，以图像块具备x方向和y方向的图像梯度进行说明。第一哈希值用于每个图像块所对应图像梯度侧重x方向还是y方向；第二哈希值用于表示每个图像块在x方向所对应图像梯度和所述每个图像块在y方向所对应图像梯度之间的接近度。本实施例中，通过公式(5)，可以得到，当θ越大时，表明该图像块的图像梯度更侧重x方向；当θ越小时，表明该图像块的图像梯度 更侧重y方向。通过公式(6)，可以得到，当μ越接近1时，表明该图像块在x方向和y方向的接近度越高，即x方向和y方向的图像梯度越接近。
在一实施例中，确定待处理图像中多个图像块的每个图像块的图像属性，包括：根据每个图像块在第一方向的图像梯度和所述每个图像块在第二方向的图像梯度，计算得到所述每个图像块在第一方向所对应图像梯度的平方值和所述每个图像块在第二方向所对应图像梯度的平方值，以及所述每个图像块在第一方向所对应图像梯度和所述每个图像块在第二方向所对应图像梯度的乘积值；通过积分图算法对第一方向所对应图像梯度的平方值、第二方向所对应图像梯度的平方值，以及第一方向所对应图像梯度和第二方向所对应图像梯度的乘积值进行计算，以得到所述每个图像块对应的第一哈希值和第二哈希值。
一实施例中，为了能够确定待处理图像中每个图像块的滤波模型，需先对待处理图像中图像块的图像属性进行统计，以查找到对应的滤波模型。从待处理图像抽取预设尺寸的图像块，并通过上述实施例中图像块的图像属性的确定方式，得到待处理图像中每个图像块的第一哈希值和第二哈希值。图2是本申请实施例提供的一种待处理图像进行图像块划分的显示示意图。如图2所示，假设待处理图像的尺寸为7*7，即该待处理图像具备49个像素值，以滤波模型的尺寸为5*5为例，由于需要保证图像块的尺寸和滤波模型的尺寸相同，则待处理图像中每个图像块的尺寸为5*5，可以得到待处理图像可以划分为9个尺寸为5*5的图像块。比如，从待处理图像的左上角开始，对图像块个数进行统计，图案填充区域A可以记为待处理图像的第一个图像块，依次向右或向下移动一个像素值，即可得到新的图像块，依次类推，该待处理图像可以划分为9个尺寸为5*5的图像块。
在将一整张待处理图像进行图像块的抽取之后，遍历该待处理图像中所有图像块，然后利用第一哈希值和第二哈希值的计算公式，计算出每个图像块的第一哈希值和第二哈希值，即确定了待处理图像中每个图像块的图像属性。
S120、根据每个图像块对应的图像属性确定所述每个图像块对应的滤波模型。
在确定待处理图像中多个图像块的每个图像块的第一哈希值和第二哈希值之后，可根据每个图像块的第一哈希值和第二哈希值，确定每个图像块对应的滤波模型。一实施例中，预先在图像属性和滤波模型之间建立有映射关系，在确定每个图像块的图像属性之后，可根据两者之间的映射关系，查找到每个图像块对应的滤波模型。
S130、通过每个图像块对应的滤波模型对所述每个图像块进行滤波，得到待处理图像的滤波图像。
在实施例中，在确定每个图像块的滤波模型之后，利用滤波模型对每个图像块进行滤波处理，以得到每个图像块的滤波图像，从而实现了对每帧待处理图像中每个图像块的自适应滤波，进而提高了待处理图像的清晰度。
为了便于对图像块的滤波过程进行解释。在实施例中，以对待处理图像中的中心点位置(i,j)所对应图像块进行滤波为例，对图像块的滤波过程进行说明。假设该图像块的中心点位置为(i,j)，根据该图像块的第一哈希值θ和第二哈希值μ，确定该图像块的滤波模型h，然后利用滤波模型h对该图像块进行滤波，以得到目标图像
Figure PCTCN2020101243-appb-000026
A k是图像块的像素展开成一行的行向量，
Figure PCTCN2020101243-appb-000027
是该行向量的转置向量，
Figure PCTCN2020101243-appb-000028
是目标图像在(i,j)位置的像素。对待处理图像中的其它图像块的滤波过程同上述过程，在此不再一一赘述。在对待处理图像中的每个图像块进行滤波之后，将多个图像块对应的目标图像合成到一起，以得到待处理图像的滤波图像。
本申请实施例提供的图像处理方法，通过确定待处理图像中多个图像块的每个图像块的图像属性；根据每个图像块的图像属性确定所述每个图像块对应的滤波模型；通过每个图像块对应的滤波模型对所述每个图像块进行滤波，得到待处理图像的滤波图像。本申请实施例利用待处理图像中每个图像块的图像属性的快速计算，加快了图像块的滤波处理速度，进而实现了图像的实时滤波，并且，通过对应的滤波模型对每个图像块进行自适应滤波，提高了视频的清晰度。
编码噪声强度用于表示待处理图像中所包含编码噪声的强弱程度。在实际操作过程中，以待处理图像为视频中的待处理图像为例，可将待处理图像所属视频的编码噪声强度作为待处理图像的编码噪声强度。由于不同视频的编码噪声强度可能是不同的，所以不同待处理图像的编码噪声强度也可能是不同的。因此，在确定每个图像块对应的滤波模型时，如果能根据待处理图像的编码噪声强度和待处理图像中每个图像块对应的图像属性共同来确定所述每个图像块对应的滤波模型，则会使确定出的滤波模型更加精准，更符合每个图像块的特性，从而更好地实现对每个图像的滤波。
在一实施例中，在根据每个图像块对应的图像属性确定所述每个图像块对应的滤波模型之前，还包括：根据待处理图像的编码噪声强度确定所述待处理图像对应的多个滤波模型；所述根据每个图像块对应的图像属性确定所述每个图像块对应的滤波模型包括：根据待处理图像中每个图像块对应的图像属性从所述待处理图像对应的多个滤波模型中确定所述每个图像块对应的滤波模型。
在一实施例中，在根据待处理图像的编码噪声强度确定所述待处理图像对应的多个滤波模型之前还包括：确定待处理图像的编码噪声强度。
在一实施例中，确定待处理图像的编码噪声强度包括：确定待处理图像所属视频的编码噪声强度；将所述待处理图像所属视频的编码噪声强度确定为所 述待处理图像的编码噪声强度。
在一实施例中，可以通过如下滤波模型的训练过程建立编码噪声强度等级与滤波器模型之间的对应关系，进而可以根据待处理图像的编码噪声强度的等级确定与所述待处理图像的编码噪声强度的等级对应的多个滤波器模型。
滤波模型的训练过程包括如下步骤：
S50、将训练集中的低清图像按照编码噪声强度的等级分成对应数量的训练子集。
一实施例中，训练集中可包括多个低清视频，在划分训练子集时，可根据多个低清视频的编码噪声强度的等级，将多个低清视频分成对应数量的训练子集。
S60、对每个训练子集中的低清图像，遍历所有尺寸为dxd的图像块。
S70、计算出所述每个训练子集中每个图像块的第一哈希值θ和第二哈希值μ，以得到对应的桶序号。
S80、利用公式(3)，将当前图像块的像素值展开成一行(即A j)，将
Figure PCTCN2020101243-appb-000029
累加到该桶序号对应的矩阵Q上，并根据当前图像块的位置，查找到对应高清图像对应位置的像素值b j，将
Figure PCTCN2020101243-appb-000030
累加到该桶序号对应的矩阵V上；遍历所述每个训练子集中所有低清图像的图像块，得到所述每个训练子集中每个桶最终的Q和V，解方程Qh＝V，即可得到该桶对应的最优滤波模型h。
S90、采用S60～S80的步骤处理所有训练子集，得到每个训练子集中每个桶对应的最优滤波模型。
本实施例中，不同训练子集对应的第一哈希值对应的子区间个数可以相同，也可以不同。不同训练子集对应的第二哈希值对应的子区间个数可以相同，也可以不同。
通过在滤波模型的训练过程中将训练集中的低清图像按照编码噪声强度的等级分类，例如编码噪声强度可以分为强、中、弱三个等级，则根据训练集中的低清图像的编码噪声强度，将训练集中的低清图像对应分为三个训练子集，然后分别采用每个训练子集中的低清图像进行训练，得到每个训练子集中多个桶对应的多个滤波模型，由此在训练滤波模型时也将编码噪声强度这一因素考虑在内，从而建立起了滤波模型和编码噪声强度之间的映射关系，通过同一训练子集训练得到的多个滤波模型对应同一等级的编码噪声强度。在确定待处理图像中每个图像块的滤波模型时，可以先根据待处理图像的编码噪声强度的等级，确定对应的一组(多个)滤波模型，进而再根据每个图像块的图像属性从这一组滤波模型中确定出最终的滤波模型。由此确定出的每个图像块的滤波模型可以更好地对每个图像块进行滤波。
本实施例中，在根据待处理图像的编码噪声强度的等级，确定对应的多个滤波模型后，可以根据待处理图像中每个图像块的第一哈希值和第二哈希值确定该图像块对应的桶序号，进而将多个滤波模型中与该桶序号对应的滤波模型作为该图像块对应的滤波模型。
图3为本申请实施例提供的另一种图像处理方法的流程图，本实施例在上述实施例提供的技术方案的基础上进行说明。本实施例中主要对滤波模型的确定过程进行解释说明。
如图3所示，本实施例中可以包括如下步骤：
S210、确定待处理图像中多个图像块的每个图像块的图像属性。
S220、根据第一哈希值和第二哈希值确定桶序号。
在得到每个图像块的第一哈希值θ和第二哈希值μ之后，将第一哈希值θ和第二哈希值μ的取值范围[0,1]，分别平均划分为q θ和q μ个子区间。从而，每个图像块根据自身的θ和μ均可以得到对应的桶序号。
S230、根据桶序号确定每个图像块对应的滤波模型。
一实施例中，在桶序号与滤波模型之间建立有一对一的映射关系。一个桶序号对应有一个滤波模型。在得到每个图像块的桶序号之后，按照两者之间的映射关系可查找到对应的滤波模型。
S240、通过每个图像块对应的滤波模型对所述每个图像块进行滤波，得到待处理图像的滤波图像。
本实施例提供的技术方案，在上述实施例的基础上，通过根据每个图像块的第一哈希值和第二哈希值确定的桶序号，查找到对应的滤波模型，从而实现了对每个图像块均对应有滤波模型，以对每个图像块进行自适应滤波，进而提高了图像的清晰度。
图4是本申请实施例提供的又一种图像处理方法的流程图。本实施例在上述实施例的基础上，对滤波模型的确定过程作说明。
如图4所示，该方法包括如下步骤：
S310、确定待处理图像中多个图像块的每个图像块的图像属性。
S320、根据每个图像块的图像属性确定所述每个图像块对应的滤波模型。
S330、根据待处理图像的编码噪声强度确定每个图像块的滤波模型。
编码噪声强度用于表示待处理图像中所包含编码噪声的强弱程度。在实际操作过程中，整张待处理图像中的编码噪声并不是均匀分布的。若将待处理图像划分为多个图像块，则每个图像块中所包含的编码噪声强度并不是相同的。 为了便于根据待处理图像的编码噪声强度确定图像块的滤波模型，可将整张待处理图像的多个图像块的编码噪声强度中比较偏重的编码噪声强度作为待处理图像的编码噪声强度，比如，待处理图像的编码噪声强度可以为整张待处理图像中所有图像块的平均编码噪声强度。也可以按照其它的方式根据每个图像块的编码噪声强度确定整张待处理图像的编码噪声强度，对此并不进行限定。
在一实施例中，待处理图像的编码噪声强度的确定方式，包括：获取待处理图像所对应视频编码的质量参数；根据质量参数确定待处理图像对应的编码噪声强度。质量参数至少包括如下参数之一：量化参数(Quantization Parameter，QP)和恒定速率系数(Constant Ratio Factor，CRF)。量化参数是视频编码参数的一种，用于描述视频编码过程中细节的损失程度。并且，QP越高，表示细节损失越多，相应的，视频中的待处理图像失真加强和质量下降；恒定速率系数，也是一种视频编码参数，类似于QP，但QP是编码过程中的一个技术指标，而CRF是恒定质量的编码方式，即从视频质量的角度描述，一个确定的CRF对应一个恒定的视频质量。CRF的取值范围一般为18-28，并且在x264编码器中，CRF的默认值为23；在x265编码器中，CRF的默认值为28。
在一实施例中，在对视频进行性能测试时，可通过性能测试的输出日志中获取待处理图像所对应视频编码的QP和CRF。在得到视频编码的QP和CRF之后，利用预先设定的编码噪声强度和质量参数的映射关系，确定视频编码对应的编码噪声强度。也可以利用视频编码的多个质量参数中的一个确定对应的编码噪声强度，即通过QP或CRF确定视频编码对应的编码噪声强度。示例性地，以CRF为18和23为例，对编码噪声强度的范围进行划分。假设CRF<18，编码噪声强度处于弱阶段；18＜CRF＜23，编码噪声强度处于中等阶段；CRF＞23，编码噪声强度处于强阶段。不同的编码噪声强度对应不同的滤波模型。也可建立QP与编码噪声强度的映射关系，或者建立QP、CRF共同与编码噪声强度的映射关系。然后，根据不同的质量参数与编码噪声强度之间的映射关系，确定待处理图像所对应视频编码对应的编码噪声强度，即确定待处理图像的编码噪声强度。
在确定待处理图像的编码噪声强度之后，按照待处理图像整体的编码噪声强度确定图像块的滤波模型。在此情况下，按照编码噪声强度确定的每个图像块的滤波模型是相同的。
S340、通过每个图像块对应的滤波模型对所述每个图像块进行滤波，得到待处理图像的滤波图像。
根据待处理图像的编码噪声强度确定每个图像块的滤波模型之后，采用该滤波模型对待处理图像的整体进行自适应滤波。然后再根据每个图像块的图像 属性确定对应的滤波模型，并按照该滤波模型对待处理图像中每个图像块进行自适应滤波，然后对每个图像块滤波之后的图像进行合成，以得到待处理图像的滤波图像。
在实际操作过程中，可先按照每帧待处理图像的编码噪声强度确定的滤波模型对待处理图像整体进行自适应滤波，然后再按照待处理图像中每个图像块的图像属性确定的滤波模型对每个图像块进行自适应滤波；也可先按照待处理图像中每个图像块的图像属性确定的滤波模型对每个图像块进行自适应滤波，然后再按照每帧待处理图像的编码噪声强度确定的滤波模型对待处理图像整体进行自适应滤波，对两个步骤的顺序不作限定。
步骤S330可在步骤S310或S320之前执行，只要保证步骤S330在步骤S340之前执行即可。
本实施例的技术方案，在上述实施例的基础上，根据待处理图像的编码噪声强度确定滤波模型，以对每帧待处理图像进行自适应滤波，提高了视频的清晰度。
对待处理图像进行自适应滤波的过程，相当于对一个图像块的像素值进行加权平均的过程。因此，若待处理图像中包含字幕等白色文字区域时，会使得白色文字区域出现变淡的效果。为了尽可能降低编码去噪对白色文字区域的影响，可以对白色文字区域进行二值化处理，以得到一张二值化图像。在上述实施例的基础上，所述方法，还包括：对待处理图像进行二值化处理，得到待处理图像的二值化图像。
对待处理图像进行二值化处理，得到待处理图像的二值化图像，包括步骤S100-S300：
S100、对待处理图像进行拉普拉斯滤波，得到文字边缘图像。
待处理图像为原低清图像，记为y。对该原低清图像进行拉普拉斯滤波，以得到文字边缘图像，记为T1。在文字边缘图像中，若RGB值为0，则表示该区域为平坦区域，几乎不可能是文字；而当RGB值越大，表示该位置是文字边缘的可能性就越大。但此时的滤波图像的文字边缘不清晰，并且存在缺口和缝隙，以及文字边缘图像中还存在一些其它的非文字边缘的区域。
S200、基于预先设定的像素阈值，对文字边缘图像进行阈值分割，得到第一图像。
第一图像为去掉文字边缘图像中非文字边缘区域的图像。在此需要理解的是，由于文字区域内的边缘较为锐利，且在经过拉普拉斯滤波之后得到的文字区域的像素值比较大，而非文字区域对应的像素值比较小。所以，此时，可以 设置一个像素阈值，记为th，然后对文字边缘图像进行阈值分割，力图去掉文字边缘图像中非文字边缘的区域，以得到去掉文字边缘图像中非文字边缘区域的图像，记为第一图像T2。
步骤S200可以包括步骤S2002-S2008：
S2002、将文字边缘图像中每个像素点对应的像素值与预先设定的像素阈值进行比对。
S2004、将大于像素阈值的像素点对应的像素值调整为第一像素值。
S2006、将小于或等于像素阈值的像素点对应的像素值调整为第二像素值。
S2008、将调整像素值后的文字边缘图像确定为第一图像。
本实施例中，将调整像素值后的文字边缘图像确定为第一图像，即将大于像素阈值的像素点对应的像素值调整为第一像素值，将小于或等于像素阈值的像素点对应的像素值调整为第二像素值，得到第一图像。
在一实施例中，可以将阈值分割的数学表达式设置为：
Figure PCTCN2020101243-appb-000031
上式中，T1(i,j)为待处理图像中(i,j)位置的图像块所对应的文字边缘图像的像素值；T2(i，j)为待处理图像中(i,j)位置的图像块所对应的文字边缘图像进行阈值分割后的像素值；第一像素值为255；第二像素值为0。本实施例中，将文字边缘图像中所有像素点的像素值均调整为0或255，以去掉非文字边缘的区域，得到第一图像。第一图像也是一个二值图像，但该二值图像不清晰，且包含不连续的文字边缘。
S300、对第一图像进行形态学闭操作，得到二值化图像。
形态学闭操作可以消弭狭窄的间断，消除小的孔洞。形体学闭操作具有填充物体内细小空洞，连接邻近物体和平滑边界的作用。对第一图像进行形态学闭操作的过程，包括：对第一图像进行膨胀操作，得到膨胀图像；对膨胀图像进行腐蚀操作，以得到二值化图像。在一实施例中，对第一图像T2进行形态学闭操作，以得到二值化图像，记为T3，从而可以将不清晰、不连续的文字区域的边缘连接完整。
在得到二值化图像之后，还包括：对二值化图像进行均值滤波。在得到二值化图像之后，对二值化图像进行均值滤波，可以平滑文字区域的锐利边缘，使其平滑过渡。
在上述实施例的基础上，对待处理图像的去噪图像的合成过程进行解释。 图5是本申请实施例提供的再一种图像处理方法的流程图。
如图5所示，该方法包括如下步骤：
S410、确定待处理图像中多个图像块中每个图像块的图像属性。
S420、根据每个图像块的图像属性确定所述每个图像块对应的滤波模型。
S430、根据待处理图像的编码噪声强度确定所述每个图像块的滤波模型。
S440、通过每个图像块对应的滤波模型对所述每个图像块进行滤波，得到待处理图像的滤波图像。
S450、对待处理图像进行拉普拉斯滤波，得到文字边缘图像。
S460、判断文字边缘图像中每个像素点的像素值是否大于预先设定的像素阈值，响应文字边缘图像中所述每个像素点的像素值大于预先设定的像素阈值的判断结果，执行步骤S470；响应文字边缘图像中所述每个像素点的像素值小于或等于预先设定的像素阈值的判断结果，执行步骤S480。
S470、将大于像素阈值的像素点对应的像素值调整为第一像素值。
S480、将小于或等于像素阈值的像素点对应的像素值调整为第二像素值。
S490、将调整像素值后的文字边缘图像确定为第一图像。
S4100、对第一图像进行形态学闭操作，得到二值化图像。
S4110、将第一像素值和二值化图像的像素值的差值，与第一像素值之间的比值作为第一权重。
S4120、将二值化图像的像素值与第一像素值之间的比值作为第二权重。
S4130、对滤波图像的像素值和第一权重的乘积值，与待处理图像的像素值和第二权重的乘积值进行相加，以得到去噪图像。
在一实施例中，去噪图像的合成公式可以为：
Figure PCTCN2020101243-appb-000032
其中，y表示原低清图像的像素值，T为二值化图像T3的像素值，
Figure PCTCN2020101243-appb-000033
为待处理图像的去噪图像的像素值，
Figure PCTCN2020101243-appb-000034
为第一权重，
Figure PCTCN2020101243-appb-000035
为第二权重。当二值化图像的像素值为255，则表示待处理图像的去噪图像为原低清图像；当二值化图像的像素值为0，则表示待处理图像的去噪图像为通过自适应滤波得到的滤波图像。
对待处理图像进行拉普拉斯滤波，以得到二值化图像的步骤，即步骤S450-S4100，可以与得到待处理图像的滤波图像的步骤，即步骤S410-S440，同时进行。可以理解为，只要步骤S410-S440和S450-S4100在步骤S41110之前执行即可，对步骤S410-S440和S450-S4100的执行顺序不作限定。
本实施例的技术方案，通过对待处理图像中文字区域的二值化处理，尽可能降低了编码去噪对文字区域的影响，避免文字区域出现变淡效果的现象，从而在保证视频中图像清晰度的同时，还可以保证视频字幕的干净清晰。
图6是本申请实施例提供的一种图像处理方法中对图像块进行滤波处理的流程图。在上述实施例的基础上，对图像处理方法中每个图像块进行滤波处理的过程进行解释说明。在本实施例中，以图像处理方法应用于移动终端为例进行说明，移动终端可以为智能手机、ipad、笔记本电脑等。
如图6所示，该方法包括如下步骤：
S510、获取一张含编码噪声的图片，并作为待处理图像。
S520、按照图像块的图像属性的计算公式，得到每个图像块的第一哈希值和第二哈希值。
S530、根据每个图像块的第一哈希值和第二哈希值查找到对应桶序号的滤波模型。
S540、利用滤波模型对图像块进行滤波，得到目标图像。
S550、将每个图像块的目标图像合成到一起，以得到待处理图像的滤波图像。
本实施例的技术方案，通过新的图像局部特征的定义和快速计算方式，提高了图像块的第一哈希值和第二哈希值的计算效率，从而在移动终端拉取到携带较强编码噪声的视频后，在视频解码之后，播放视频之前，可实时去除视频中的当前视频帧的编码噪声，以及去除视频中任意类型的编码噪声。
在视频直播、短视频等应用领域，编码噪声的强度会随着网络情况的变化而变化，此时，可以针对不同强度的编码噪声进行处理。在对滤波模型进行训练时，可以构建不同编码噪声强度的训练集，并分别进行训练，得到多个滤波模型。从而在图像处理的实际操作过程中，可以根据视频编码的质量参数(比如，QP和/或CRF)，来使用不同的滤波模型进行滤波。
本申请实施例提供了一种移动终端视频处理方法，本实施例可应用于任一种具有视频处理功能的移动终端中。本实施例提供的一种移动终端视频处理方法可以由本申请实施例提供的移动终端视频处理装置来执行，该装置可以通过软件和/或硬件的方式来实现，并集成在执行本方法的设备中，该设备可以是任一具备视频处理能力的移动终端。
图7是本申请实施例提供的一种移动终端视频处理方法的流程图，如图7所示，该方法包括：
S610、解码从视频流中拉取的待处理视频。
S620、抽取待处理视频中的视频帧。
S630、对作为待处理图像的视频帧进行处理，得到视频帧对应的滤波图像。
S640、根据滤波图像，进行视频播放。
在实施例中，在用户观看视频时，移动终端上的应用程序(Application，APP)需要从服务器上拉取视频流，获取并解码视频流中的待处理视频，然后识别抽取待处理视频中的当前视频帧，并将当前视频帧作为待处理图像，并执行上述任一实施例所述的图像处理方法，对当前视频帧进行滤波去噪，以得到当前视频帧的滤波图像，然后通过去噪图像更新待处理视频中对应的当前视频帧，并进行视频的播放，提高了视频的清晰度，另外因前述实施例的图像处理方法计算快并且去噪效率高，因而应用在移动终端上能够满足视频帧及播放的实时处理需求，从而提升了移动终端的功能性和用户的观看体验。
本实施例的技术方案，通过对视频流中的视频帧进行自适应滤波，在网络受限的情况下，不仅可以较好地消除编码噪声，还能适应当前的网络状况，对视频帧上的字幕进行二值化处理，以保持字幕干净清晰，提升了用户的观看体验。
本申请实施例提高了视频的去噪效率，实现了视频的实时去噪，以及，提高了视频观看时的清晰度。
图8为本申请实施例提供的一种图像处理装置的结构示意图，如图8所示，该装置可以包括：第一确定模块710、第二确定模块720和第一滤波模块730。
第一确定模块710，设置为确定待处理图像中多个图像块的每个图像块的图像属性；第二确定模块720，设置为根据每个图像块的图像属性确定所述每个图像块对应的滤波模型；第一滤波模块730，设置为通过每个图像块对应的滤波模型对所述每个图像块进行滤波，得到待处理图像的滤波图像。
本实施例提供的技术方案，通过确定待处理图像中多个图像块的每个图像块的图像属性；根据每个图像块的图像属性确定所述每个图像块对应的滤波模型；通过每个图像块对应的滤波模型对所述每个图像块进行滤波，得到待处理图像的滤波图像。本申请实施例利用待处理图像中每个图像块的图像属性的快速计算，加快了图像块的滤波处理速度，进而实现了图像的实时滤波，并且，通过对应的滤波模型对每个图像块进行自适应滤波，提高了视频的清晰度。
在上述实施例的基础上，图像属性包括：第一哈希值和第二哈希值；第一哈希值用于表示每个图像块所对应图像梯度的侧重方向；第二哈希值用于表示每个图像块在第一方向和第二方向所对应图像梯度之间的接近度。
在上述实施例的基础上，第一确定模块710是设置为通过如下方式确定待处理图像中多个图像块的每个图像块的图像属性，包括：根据每个图像块在第一方向和第二方向的图像梯度，计算得到所述每个图像块在第一方向所对应图像梯度的平方值、所述每个图像块在第二方向所对应图像梯度的平方值，以及所述每个图像块在第一方向所对应图像梯度和所述每个图像块在第二方向所对应图像梯度的乘积值；通过积分图算法对所述第一方向所对应图像梯度的平方值、所述第二方向所对应图像梯度的平方值，以及所述第一方向所对应图像梯度和所述第二方向所对应图像梯度的乘积值进行计算，以得到对应的第一哈希值和第二哈希值。
在上述实施例的基础上，第二确定模块720，包括：第一确定单元，设置为根据每个图像块对应的所述第一哈希值和所述第二哈希值确定桶序号；第二确定单元，设置为根据所述桶序号确定所述每个图像块对应的滤波模型。
在上述实施例的基础上，图像处理装置，还包括：第三确定模块，设置为根据待处理图像的编码噪声强度确定每个图像块的滤波模型。
在上述实施例的基础上，第一滤波模块730是设置为通过每个图像块对应的目标模型对所述每个图像块进行滤波，得到所述待处理图像的滤波图像，其中，每个图像块对应的目标模型包括根据所述待处理图像的编码噪声强度确定的所述每个图像块的滤波模型和根据所述每个图像块的图像属性确定的所述每个图像块对应的滤波模型。
在上述实施例的基础上，待处理图像的编码噪声强度的确定方式，包括：获取待处理图像对应视频编码的质量参数；根据所述质量参数确定所述待处理图像的编码噪声强度。
在上述实施例的基础上，图像处理装置，还包括：处理模块，设置为对所述待处理图像进行二值化处理，得到所述待处理图像的二值化图像。
在上述实施例的基础上，处理模块，包括：滤波单元，设置为对待处理图像进行拉普拉斯滤波，得到文字边缘图像；分割单元，设置为基于预先设定的像素阈值，对文字边缘图像进行阈值分割，得到第一图像，第一图像为去掉文字边缘图像中非文字边缘区域的图像；闭操作单元，设置为对第一图像进行形态学闭操作，得到二值化图像。
在上述实施例的基础上，图像处理装置，还包括：第二滤波模块，设置为在得到二值化图像之后，还包括：对二值化图像进行均值滤波。
在上述实施例的基础上，闭操作单元，包括：第一操作子单元，设置为对第一图像进行膨胀操作，得到膨胀图像；第二操作子单元，设置为对膨胀图像 进行腐蚀操作，以得到二值化图像。
在上述实施例的基础上，分割单元，包括：比对子单元，设置为对文字边缘图像中每个像素点对应的像素值与预先设定的像素阈值进行比对；第一调整子单元，设置为将大于像素阈值的像素点对应的像素值调整为第一像素值；第二调整子单元，设置为将小于或等于像素阈值的像素点对应的像素值调整为第二像素值；合成子单元，设置为将调整像素值后的文字边缘图像确定为第一图像。
在上述实施例的基础上，图像处理装置，还包括：第一权重确定模块，设置为将第一像素值和二值图像的像素值的差值，与第一像素值之间的比值作为第一权重；第二权重确定模块，设置为将二值图像的像素值与第一像素值之间的比值作为第二权重；去噪模块，设置为对滤波图像的像素值和第一权重的乘积值，与待处理图像的像素值和第二权重的乘积值进行相加，以得到去噪图像。
在上述实施例的基础上，对滤波模型进行训练的过程包括：获取第一预设尺寸的高清图像和携带编码噪声的低清图像，高清图像和低清图像的分辨率相同；分别从高清图像和低清图像中抽取多个第二预设尺寸的图像块，以分别组成第一矩阵和第二矩阵，第二预设尺寸小于第一预设尺寸；基于最小二乘法的正规方程，并根据第一矩阵和第二矩阵得到滤波模型。
在上述实施例的基础上，滤波模型的尺寸和图像块的尺寸相同。
在上述实施例的基础上，图像处理装置，还包括：第四确定模块，设置为根据待处理图像的编码噪声强度确定所述待处理图像对应的多个滤波模型；第二确定模块720，是设置为根据待处理图像中每个图像块的图像属性从所述待处理图像对应的多个滤波模型中确定所述每个图像块对应的滤波模型。
本实施例提供的图像处理装置可适用于本申请任意实施例提供的图像处理方法。
图9是本申请实施例提供的一种移动终端视频处理装置的结构示意图。如图9所示，该装置包括：解码模块810、抽取模块820、处理模块830和播放模块840。
解码模块810，设置为解码从视频流中拉取的待处理视频；抽取模块820，设置为抽取待处理视频中的视频帧；处理模块830，设置为采用上述任意实施例中所述的图像处理方法处理作为待处理图像的视频帧，得到对应的滤波图像；播放模块840，设置为根据滤波图像，进行视频播放。
本实施例提供的技术方案，通过对视频中的当前视频帧进行去噪处理，提高了视频的去噪效率，实现了视频的实时去噪，以及提高了视频观看时的清晰 度，从而提升了用户的观看体验。
本实施例提供的移动终端视频处理装置可适用于本申请任意实施例提供的移动终端视频处理方法。
图10为本申请实施例提供的一种设备的结构示意图，如图10所示，该设备包括处理器910、存储装置920、输入装置930和输出装置940。该设备中的处理器910可以是一个或多个，图10中以一个处理器910为例，设备中的处理器910、存储器920、输入装置930和输出装置940可以通过总线或其他方式连接，图10中以通过总线连接为例。
该设备中的存储器920作为一种计算机可读存储介质，可用于存储一个或多个程序，程序可以是软件程序、计算机可执行程序以及模块，如本申请实施例所提供的图像处理方法对应的程序指令/模块(例如，图8所示的图像处理装置中的模块，包括：第一确定模块710、第二确定模块720和第一滤波模块730)。处理器910通过运行存储在存储器920中的软件程序、指令以及模块，从而执行设备的多种功能应用以及数据处理，即实现上述方法实施例中图像处理方法。
存储器920可包括存储程序区和存储数据区，存储程序区可存储操作系统、至少一个功能所需的应用程序；存储数据区可存储根据设备的使用所创建的数据等。此外，存储器920可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他非易失性固态存储器件。在一些实例中，存储器920可进一步包括相对于处理器910远程设置的存储器，这些远程存储器可以通过网络连接至设备。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
输入装置930可设置为接收用户输入的数字或字符信息，以产生与终端设备的用户设置以及功能控制有关的键信号输入。输出装置940可包括显示屏等显示设备。
并且，当上述设备所包括一个或者多个程序被一个或者多个处理器910执行时，程序进行如下操作：确定待处理图像中多个图像块的每个图像块的图像属性；根据每个图像像块的图像属性确定所述每个图像块对应的滤波模型；通过每个图像块对应的滤波模型对所述每个图像块进行滤波，得到待处理图像的滤波图像。
本实施例提供的一种设备可设置为执行本申请任意实施例提供的图像处理方法。
一实施例中，本实施例提供的一种设备可设置为执行本申请任意实施例提供的移动终端视频处理方法。
本申请实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时可实现本申请任意实施例中的图像处理方法或者移动终端视频处理方法。该图像处理方法可以包括：确定待处理图像中多个图像块的每个图像块的图像属性；根据每个图像块的图像属性确定所述每个图像块对应的滤波模型；通过每个图像块对应的滤波模型对所述每个图像块进行滤波，得到待处理图像的滤波图像
该移动终端视频处理方法包括：解码从视频流中拉取的待处理视频；抽取待处理视频中的视频帧；对采用上述任一实施例所述的图像处理方法处理作为待处理图像的视频帧，得到视频帧对应的滤波图像；根据视频帧的滤波图像，进行视频播放。
本申请实施例所提供的一种包含计算机可执行指令的存储介质，其计算机可执行指令不限于如上所述的方法操作，还可以执行本申请任意实施例所提供的图像处理方法或者移动终端视频处理方法中的相关操作。
通过以上关于实施方式的描述，所属领域的技术人员可以了解到，本申请可借助软件及必需的通用硬件来实现，也可以通过硬件实现，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案可以以软件产品的形式体现出来，该计算机软件产品可以存储在计算机可读存储介质中，如计算机的软盘、只读存储器(Read-Only Memory,ROM)、随机存取存储器(Random Access Memory,RAM)、闪存(FLASH)、硬盘或光盘等，包括多个指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请多个实施例所述的图像处理方法或移动终端视频处理方法。

Claims (20)

  1. 一种图像处理方法，包括：
    确定待处理图像中多个图像块的每个图像块的图像属性；
    根据每个图像块的图像属性确定所述每个图像块对应的滤波模型；
    通过每个图像块对应的所述滤波模型对所述每个图像块进行滤波，得到所述待处理图像的滤波图像。
  2. 根据权利要求1所述的方法，其中，所述图像属性包括：第一哈希值和第二哈希值；
    所述第一哈希值用于表示每个图像块所对应图像梯度的侧重方向；所述第二哈希值用于表示每个图像块在第一方向所对应图像梯度和所述每个图像块在第二方向所对应图像梯度之间的接近度。
  3. 根据权利要求2所述的方法，其中，所述确定待处理图像中多个图像块的每个图像块的图像属性，包括：
    根据每个图像块在第一方向的图像梯度和所述每个图像块在第二方向的图像梯度，计算得到所述每个图像块在第一方向所对应图像梯度的平方值、所述每个图像块在第二方向所对应图像梯度的平方值，以及所述每个图像块在第一方向所对应图像梯度和所述每个图像块在第二方向所对应图像梯度的乘积值；
    通过积分图算法对所述第一方向所对应图像梯度的平方值、所述第二方向所对应图像梯度的平方值，以及所述第一方向所对应图像梯度和所述第二方向所对应图像梯度的乘积值进行计算，以得到所述每个图像块对应的第一哈希值和第二哈希值。
  4. 根据权利要求2所述的方法，其中，所述根据每个图像块的图像属性确定所述每个图像块对应的滤波模型，包括：
    根据每个图像块对应的所述第一哈希值和所述第二哈希值确定桶序号；
    根据所述桶序号确定所述每个图像块对应的滤波模型。
  5. 根据权利要求1-4任一所述的方法，还包括：根据待处理图像的编码噪声强度确定每个图像块的滤波模型；
    所述通过每个图像块对应的所述滤波模型对所述每个图像块进行滤波，得到所述待处理图像的滤波图像，包括：
    通过每个图像块对应的目标模型对所述每个图像块进行滤波，得到所述待处理图像的滤波图像，其中，每个图像块对应的目标模型包括根据所述待处理图像的编码噪声强度确定的所述每个图像块的滤波模型和根据所述每个图像块 的图像属性确定的所述每个图像块对应的滤波模型。
  6. 根据权利要求5所述的方法，其中，所述待处理图像的编码噪声强度的确定方式，包括：
    获取待处理图像对应视频编码的质量参数；
    根据所述质量参数确定所述待处理图像的编码噪声强度。
  7. 根据权利要求1-4任一所述的方法，还包括：
    对所述待处理图像进行二值化处理，得到所述待处理图像的二值化图像。
  8. 根据权利要求7所述的方法，其中，所述对所述待处理图像进行二值化处理，得到所述待处理图像的二值化图像，包括：
    对所述待处理图像进行拉普拉斯滤波，得到文字边缘图像；
    基于预先设定的像素阈值，对所述文字边缘图像进行阈值分割，得到第一图像，所述第一图像为去掉所述文字边缘图像中非文字边缘区域的图像；
    对所述第一图像进行形态学闭操作，得到所述二值化图像。
  9. 根据权利要求8所述的方法，在所述得到所述二值化图像之后，还包括：对所述二值化图像进行均值滤波。
  10. 根据权利要求8所述的方法，其中，所述对所述第一图像进行形态学闭操作，得到所述二值化图像，包括：
    对所述第一图像进行膨胀操作，得到膨胀图像；
    对所述膨胀图像进行腐蚀操作，得到所述二值化图像。
  11. 根据权利要求8所述的方法，其中，所述基于预先设定的像素阈值，对所述文字边缘图像进行阈值分割，得到第一图像，包括：
    将所述文字边缘图像中每个像素点对应的像素值与预先设定的像素阈值进行比对；
    将大于所述像素阈值的像素点对应的像素值调整为第一像素值；
    将小于或等于所述像素阈值的像素点对应的像素值调整为第二像素值；
    将调整像素值后的文字边缘图像确定为第一图像。
  12. 根据权利要求11所述的方法，还包括：
    将所述第一像素值和所述二值化图像的像素值的差值，与所述第一像素值之间的比值作为第一权重；
    将所述二值化图像的像素值与所述第一像素值之间的比值作为第二权重；
    将所述滤波图像的像素值和所述第一权重的乘积值，与所述待处理图像的像素值和所述第二权重的乘积值进行相加，得到所述待处理图像的去噪图像。
  13. 根据权利要求1至4任一项所述的方法，其中，对所述滤波模型进行训练的过程，包括：
    获取第一预设尺寸的高清图像和携带编码噪声的低清图像，所述高清图像和所述低清图像的分辨率相同；
    分别从所述高清图像和低清图像中抽取多个第二预设尺寸的图像块，以分别组成第一矩阵和第二矩阵，所述第二预设尺寸小于所述第一预设尺寸；
    按照最小二乘法的正规方程，并根据所述第一矩阵和第二矩阵得到滤波模型。
  14. 根据权利要求1至4任一项所述的方法，其中，所述滤波模型的尺寸和所述图像块的尺寸相同。
  15. 根据权利要求1至4任一项所述的方法，在根据每个图像块的图像属性确定所述每个图像块对应的滤波模型之前，还包括：根据待处理图像的编码噪声强度确定所述待处理图像对应的多个滤波模型；
    所述根据每个图像块的图像属性确定所述每个图像块对应的滤波模型包括：根据待处理图像中每个图像块的图像属性从所述待处理图像对应的多个滤波模型中确定所述每个图像块对应的滤波模型。
  16. 一种移动终端视频处理方法，包括：
    解码从视频流中拉取的待处理视频；
    抽取所述待处理视频中的视频帧；
    采用权利要求1至15任一项所述的方法处理作为待处理图像的所述视频帧，得到所述视频帧对应的滤波图像；
    根据所述滤波图像，进行视频播放。
  17. 一种图像处理装置，包括：
    第一确定模块，设置为确定待处理图像中多个图像块的每个图像块的图像属性；
    第二确定模块，设置为根据每个图像块的图像属性确定所述每个图像块对应的滤波模型；
    第一滤波模块，设置为通过每个图像块对应的所述滤波模型对所述每个图像块进行滤波，得到所述待处理图像的滤波图像。
  18. 一种移动终端视频处理装置，包括：
    解码模块，设置为解码从视频流中拉取的待处理视频；
    抽取模块，设置为抽取所述待处理视频中的视频帧；
    处理模块，设置为采用权利要求1至15任一项所述的方法处理作为待处理图像的所述视频帧，得到所述视频帧对应的滤波图像；
    播放模块，设置为根据所述视频帧的滤波图像，进行视频播放。
  19. 一种设备，包括：
    至少一个处理器；
    存储器，设置为存储至少一个程序；
    当所述至少一个程序被所述至少一个处理器执行，所述至少一个处理器实现如权利要求1-15中任一所述的图像处理方法，或者实现如权利要求16所述的移动终端视频处理方法。
  20. 一种计算机可读存储介质，存储有计算机程序，所述计算机程序被处理器执行时实现如权利要求1-15中任一所述的图像处理方法，或者实现如权利要求16所述的移动终端视频处理方法。
PCT/CN2020/101243 2019-07-23 2020-07-10 图像处理和移动终端视频处理方法、装置、设备和介质 WO2021012965A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/629,132 US20220239908A1 (en) 2019-07-23 2020-07-10 Method for processing images, method for processing videos, computer device and medium
EP20844069.3A EP4006824A4 (en) 2019-07-23 2020-07-10 IMAGE PROCESSING METHOD AND APPARATUS, MOBILE TERMINAL VIDEO PROCESSING METHOD AND APPARATUS, DEVICE AND MEDIA

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN201910667696.6 2019-07-23
CN201910667696 2019-07-23
CN202010348909.1A CN111445424B (zh) 2019-07-23 2020-04-28 图像处理和移动终端视频处理方法、装置、设备和介质
CN202010348909.1 2020-04-28

Publications (1)

Publication Number Publication Date
WO2021012965A1 true WO2021012965A1 (zh) 2021-01-28

Family

ID=71653546

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/101243 WO2021012965A1 (zh) 2019-07-23 2020-07-10 图像处理和移动终端视频处理方法、装置、设备和介质

Country Status (4)

Country Link
US (1) US20220239908A1 (zh)
EP (1) EP4006824A4 (zh)
CN (1) CN111445424B (zh)
WO (1) WO2021012965A1 (zh)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021181681A1 (ja) * 2020-03-13 2021-09-16 日本電信電話株式会社 数理モデル導出装置、数理モデル導出方法及びプログラム
CN112233049B (zh) * 2020-12-14 2021-03-02 成都中轨轨道设备有限公司 一种用于提升图像清晰度的图像融合方法
CN113038135B (zh) * 2021-05-19 2022-02-18 杭州微帧信息科技有限公司 基于块效应检测的crf自适应方法
CN113808020A (zh) * 2021-09-18 2021-12-17 北京字节跳动网络技术有限公司 图像处理方法及设备
WO2023070582A1 (en) * 2021-10-29 2023-05-04 Huawei Technologies Co.,Ltd. A device and method for noise-adaptive high dynamic range image processing

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101043581A (zh) * 2006-03-21 2007-09-26 中国科学院计算技术研究所 一种多模板混合滤波的图像去噪方法
US20160219231A1 (en) * 2015-01-27 2016-07-28 Semiconductor Components Industries, Llc Dark current gradient estimation using optically black pixels
CN108550119A (zh) * 2018-03-27 2018-09-18 福州大学 一种结合边缘信息的图像去噪方法
CN108765343A (zh) * 2018-05-29 2018-11-06 Oppo（重庆）智能科技有限公司 图像处理的方法、装置、终端及计算机可读存储介质
CN108765322A (zh) * 2018-05-16 2018-11-06 上饶师范学院 图像去噪方法及装置

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4599457B1 (ja) * 2009-09-18 2010-12-15 株式会社東芝 画像処理装置、表示装置及び画像処理方法
CN104103064A (zh) * 2013-04-10 2014-10-15 无锡金帆钻凿设备股份有限公司 基于梯度相似度的无参考噪声图像质量评价方法
US20150039637A1 (en) * 2013-07-31 2015-02-05 The Nielsen Company (Us), Llc Systems Apparatus and Methods for Determining Computer Apparatus Usage Via Processed Visual Indicia
US10335026B2 (en) * 2017-01-11 2019-07-02 Canon Kabushiki Kaisha Image processing apparatus that generates a tomographic image using an estimation value of pixel values, and related optical coherence tomography apparatus, image processing method, and computer-readable storage medium
CN107527321B (zh) * 2017-08-22 2020-04-17 维沃移动通信有限公司 一种图像重建方法、终端及计算机可读存储介质
US10521878B1 (en) * 2017-09-27 2019-12-31 United Services Automobile Association (Usaa) Machine-learning for enhanced machine reading of non-ideal capture conditions
NL2019905B1 (en) * 2017-11-14 2019-05-20 Mat Systems and methods for segmenting images

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101043581A (zh) * 2006-03-21 2007-09-26 中国科学院计算技术研究所 一种多模板混合滤波的图像去噪方法
US20160219231A1 (en) * 2015-01-27 2016-07-28 Semiconductor Components Industries, Llc Dark current gradient estimation using optically black pixels
CN108550119A (zh) * 2018-03-27 2018-09-18 福州大学 一种结合边缘信息的图像去噪方法
CN108765322A (zh) * 2018-05-16 2018-11-06 上饶师范学院 图像去噪方法及装置
CN108765343A (zh) * 2018-05-29 2018-11-06 Oppo（重庆）智能科技有限公司 图像处理的方法、装置、终端及计算机可读存储介质

Also Published As

Publication number Publication date
EP4006824A4 (en) 2022-11-30
CN111445424B (zh) 2023-07-18
CN111445424A (zh) 2020-07-24
US20220239908A1 (en) 2022-07-28
EP4006824A1 (en) 2022-06-01

Similar Documents

Publication Publication Date Title
WO2021012965A1 (zh) 图像处理和移动终端视频处理方法、装置、设备和介质
CN109743473A (zh) 视频图像3d降噪方法、计算机装置及计算机可读存储介质
WO2018082185A1 (zh) 图像处理方法和装置
CN110796615B (zh) 一种图像去噪方法、装置以及存储介质
US8594449B2 (en) MPEG noise reduction
US8244054B2 (en) Method, apparatus and integrated circuit capable of reducing image ringing noise
US8594423B1 (en) Automatic background identification in video images
KR102038789B1 (ko) 포커스 검출
WO2020253103A1 (zh) 视频图像处理方法、装置、设备及存储介质
CN110852334B (zh) 用于自适应像素滤波的系统和方法
WO2022143421A1 (zh) 噪声强度估计方法、装置及电子设备
CN113068034B (zh) 视频编码方法及装置、编码器、设备、存储介质
He et al. Spatio-temporal saliency-based motion vector refinement for frame rate up-conversion
CN109255752A (zh) 图像自适应压缩方法、装置、终端及存储介质
Kim et al. Constrained adversarial loss for generative adversarial network‐based faithful image restoration
Liu et al. Video stabilization algorithm based on Pearson correlation coefficient
CN112929668B (zh) 一种视频编码的方法、装置、设备和存储介质
US20130141531A1 (en) Computer program product, computer readable medium, compression method and apparatus of depth map in 3d video
CN116363018B (zh) 一种自适应的nlm降噪方法
US8831354B1 (en) System and method for edge-adaptive and recursive non-linear filtering of ringing effect
CN110418195B (zh) 一种基于最佳裁剪的实时视频缩略方法
CN105893999B (zh) 一种感兴趣区域提取方法及装置
KR101829742B1 (ko) 이중 선형 필터 및 퍼지-기반 가중 평균 필터에 기반한 디인터레이싱 장치 및 방법
CN116760983B (zh) 用于视频编码的环路滤波方法及装置
US11836893B2 (en) Online AI super resolution engine and operations

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20844069

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020844069

Country of ref document: EP

Effective date: 20220223