CN113949869A - Method for estimating motion vector of pixel block, video processing apparatus, video processing device, and medium - Google Patents
Method for estimating motion vector of pixel block, video processing apparatus, video processing device, and medium Download PDFInfo
- Publication number
- CN113949869A CN113949869A CN202010687720.5A CN202010687720A CN113949869A CN 113949869 A CN113949869 A CN 113949869A CN 202010687720 A CN202010687720 A CN 202010687720A CN 113949869 A CN113949869 A CN 113949869A
- Authority
- CN
- China
- Prior art keywords
- motion vector
- candidate motion
- pixel block
- candidate
- block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 239000013598 vector Substances 0.000 title claims abstract description 356
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000005070 sampling Methods 0.000 claims abstract description 37
- 230000002123 temporal effect Effects 0.000 claims description 17
- 238000010845 search algorithm Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 11
- 230000000694 effects Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 9
- 238000004422 calculation algorithm Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/513—Processing of motion vectors
- H04N19/521—Processing of motion vectors for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/136—Incoming video signal characteristics or properties
- H04N19/137—Motion inside a coding unit, e.g. average field, frame or block difference
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/136—Incoming video signal characteristics or properties
- H04N19/137—Motion inside a coding unit, e.g. average field, frame or block difference
- H04N19/139—Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/513—Processing of motion vectors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/53—Multi-resolution motion estimation; Hierarchical motion estimation
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Television Systems (AREA)
Abstract
The invention discloses a method for estimating a motion vector of a pixel block, a video processing device, an electronic device and a storage medium, wherein the method for estimating the motion vector of the pixel block comprises the following steps: obtaining a plurality of candidate motion vectors of a current pixel block in a current video frame, wherein the plurality of candidate motion vectors at least comprise down-sampling candidate motion vectors; calculating the reliability of the difference value between the other candidate motion vectors except the down-sampling candidate motion vector in the plurality of candidate motion vectors and the down-sampling candidate motion vector; and determining the candidate motion vector with the highest reliability of the difference value between the other candidate motion vectors and the down-sampling candidate motion vector as the motion vector of the current pixel block. The method, the video processing device and the electronic equipment can obtain more accurate motion vectors from the candidate motion vectors, improve the reliability of motion estimation, reduce video jitter or unsmooth and improve the video playing effect.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method for estimating a motion vector of a pixel block, and a video processing apparatus, an electronic device, and a non-transitory computer storage medium.
Background
The frame rate of playing video may be higher than that of the video source, so that a processing step of increasing the frame rate of the video source may be performed, for example, a 24Hz video is processed and then a 60Hz video is output, and increasing the frame rate of the video may reduce jitter and provide a smoother and smoother visual effect.
If the video frame rate is increased by simply repeating the original frames, the video will still be jittery. Therefore, the frame rate is increased by the methods of motion estimation and motion compensation. The quality of frame rate conversion mainly depends on the reliability of the motion estimation algorithm, and the information of the frame to be inserted is calculated through the motion vector obtained by the motion estimation algorithm.
In practical applications, each frame of picture is generally divided into several non-overlapping and equal-sized blocks for motion estimation and motion compensation. Estimating the motion vector of a block generates a plurality of candidate motion vectors, and the matching error of two blocks pointed by each candidate motion vector can judge the reliability of the motion vector to a certain extent. However, when there are a plurality of candidate motion vectors with the smallest block matching error, if the finally selected motion vector is not appropriate, the video still has the phenomenon of jittering or unsmooth flow, and the video playing quality is reduced.
Disclosure of Invention
The present invention is directed to solving at least one of the problems of the prior art. Therefore, an object of the present invention is to provide a method for estimating a motion vector of a pixel block, which can obtain a more accurate motion vector from candidate motion vectors, improve the reliability of motion estimation, reduce video jitter or video dysfluencies, and improve the video playing effect.
It is a second object of the invention to propose a non-transitory computer storage medium.
A third object of the present invention is to provide a video processing apparatus.
A fourth object of the present invention is to provide an electronic device.
In order to achieve the above object, a method for estimating a motion vector of a pixel block according to an embodiment of the first aspect of the present invention includes: obtaining a plurality of candidate motion vectors of a current pixel block in a current video frame, wherein the plurality of candidate motion vectors at least comprise down-sampling candidate motion vectors; calculating the reliability of the difference value between the other candidate motion vectors except the down-sampling candidate motion vector and the down-sampling candidate motion vector in a plurality of candidate motion vectors; and determining the candidate motion vector with the highest reliability of the difference value between the other candidate motion vectors and the downsampled candidate motion vector as the motion vector of the current pixel block.
According to the method for estimating the motion vector of the pixel block, the candidate motion vector with higher estimation speed and higher reliability and the candidate motion vector with higher reliability of the difference value with the motion vector candidate under the down sampling are more reliable as the candidate motion vector is closer to the motion vector candidate under the down sampling, and the candidate motion vector with the highest reliability of the difference value with the motion vector candidate under the down sampling in other candidate motion vectors is determined to be the final motion vector of the current pixel block, so that the reliability of the obtained motion vector can be ensured, the precision of the motion vector can be ensured, the reliability of motion estimation is improved, the video jitter and unsmooth phenomena can be reduced, and the video quality is improved.
In some embodiments, obtaining the plurality of candidate motion vectors for the current block of pixels in the current video frame comprises: obtaining a downsampled frame image of the current video frame; dividing the downsampled frame image into a plurality of candidate blocks of the same size; and calculating the downsampling candidate motion vector of the current pixel block by adopting a hierarchical search algorithm.
In some embodiments, obtaining a plurality of candidate motion vectors for a current block of pixels in a current video frame further comprises: taking a block region of the current video frame, in which a motion vector has been estimated, as a spatial recursive search region of the current pixel block; and recursing the motion vector of each candidate block in the spatial recursive search area to the current pixel block of the current video frame by adopting a spatial recursion method to be used as the spatial recursive candidate motion vector of the current pixel block.
In some embodiments, obtaining a plurality of candidate motion vectors for a current block of pixels in a current video frame further comprises: taking a block area scanned after a time-adjacent pixel block corresponding to the current pixel block in a previous video frame of the current video frame as a time recursive search area of the current pixel block; and recursing the motion vector of each candidate block in the time recursive search area to the current pixel block of the current video frame by adopting a time recursion method to be used as the time recursive candidate motion vector of the current pixel block.
In some embodiments, obtaining a plurality of candidate motion vectors for a current block of pixels in a current video frame further comprises: acquiring a recursive motion vector of a pixel block at a preset neighborhood internal position around the current pixel block; adding a random motion offset to the recursive motion vector to obtain a random candidate motion vector for the current block of pixels.
In some embodiments, calculating the confidence level between the other candidate motion vectors of the plurality of candidate motion vectors except the downsampled candidate motion vector and the downsampled candidate motion vector comprises:
calculating a first difference absolute value of the spatial recursive candidate motion vector and the downsampled candidate motion vector, and obtaining a first reliability of the spatial recursive candidate motion vector and the downsampled candidate motion vector according to the first difference absolute value;
calculating a second difference absolute value between the temporal recursion candidate motion vector and the downsampling candidate motion vector, and obtaining a second reliability of the temporal recursion candidate motion vector and the downsampling candidate motion vector according to the second difference absolute value;
and calculating a third difference absolute value of the random candidate motion vector and the downsampled candidate motion vector, and obtaining a third reliability of the random candidate motion vector and the downsampled candidate motion vector according to the third difference absolute value.
In some embodiments, determining the candidate motion vector with the highest confidence level with the downsampled candidate motion vector among the other candidate motion vectors as the motion vector of the current pixel block comprises: obtaining a highest confidence level of the first confidence level, the second confidence level, and the third confidence level; and determining the candidate motion vector corresponding to the highest credibility as the motion vector of the current pixel block.
In order to achieve the above object, a second embodiment of the present invention proposes a non-transitory computer storage medium having stored thereon a computer program which, when executed by a processor, implements the method for estimating a motion vector of a pixel block.
In order to achieve the above object, a third embodiment of the present invention provides a video processing apparatus, including: a processor; a memory communicatively coupled to the processor; wherein said memory has stored therein a computer program executable by said processor, said computer program when executed by said processor implementing said method of estimating a motion vector of a block of pixels.
In order to achieve the above object, an electronic device according to a fourth aspect of the present invention includes an image display apparatus and the video processing apparatus, and the video processing apparatus is connected to the image display apparatus.
The electronic device of the embodiment of the present invention, using the video processing apparatus of the above embodiment, may use the following sampling candidate motion vector as the reference motion vector, which may improve the estimation speed, and the candidate motion vector with higher reliability and higher reliability of the difference value with the down-sampling candidate motion vector, and the candidate motion vector with the highest reliability with the down-sampling candidate motion vector is closer to the down-sampling candidate motion vector, that is, more reliable, and determines the candidate motion vector with the highest reliability of the difference value with the down-sampling candidate motion vector among the other candidate motion vectors as the final motion vector of the current pixel block, thereby not only ensuring the reliability of the obtained motion vector, but also ensuring the precision of the motion vector, improving the reliability of the motion estimation, and further reducing the video jitter and improving the video quality.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a flow diagram of a method of estimating a motion vector for a block of pixels according to one embodiment of the invention;
fig. 2 (a) and (b) are schematic diagrams of a search block of a down-sampled frame image and an original image according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a difference versus confidence curve for a downsampled candidate motion vector and other candidate motion vectors, according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a search range for estimating a motion vector for a current block of pixels using a recursive search algorithm in accordance with one embodiment of the present invention;
FIG. 5 is a block diagram of a video processing device according to one embodiment of the present invention;
FIG. 6 is a block diagram of an electronic device according to one embodiment of the invention.
Reference numerals:
100: an electronic device;
10: a video processing device; 20: an image display device;
11: a processor; 12: a memory.
Detailed Description
Embodiments of the present invention will be described in detail below, the embodiments described with reference to the drawings being illustrative, and the embodiments of the present invention will be described in detail below.
In order to reduce jitter and provide a smoother and smoother visual effect, electronic devices such as televisions, computers, etc. increase the frame rate of a video source when a video stream is processed, for example, a method of motion estimation and motion compensation in the public world increases the frame rate. The basic idea of motion estimation is to divide each frame of an image sequence into a plurality of non-overlapping macroblocks, consider the displacement of all pixels in the macroblocks to be the same, and then find out the block most similar to the current block, i.e. the matching block, from each macroblock to a reference frame within a given specific search range according to a certain matching criterion, where the relative displacement between the matching block and the current block is the motion vector.
When a certain pixel block is subjected to motion estimation, a plurality of motion vectors which are possibly correct for the current pixel block, namely candidate motion vectors, are estimated, and the most suitable motion vector is selected from the candidate motion vectors to be used as the motion vector of the current pixel block. The embodiment of the invention provides a method for estimating a pixel motion vector, and the estimated candidate motion vectors are used as a standard for judging the reliability, besides the block matching error, a standard for judging the reliability of the candidate motion vectors is added, so that the accuracy of block motion estimation is improved.
A method of estimating a motion vector of a pixel block according to an embodiment of the present invention is described below with reference to fig. 1 to 4.
Fig. 1 is a flowchart of a method of estimating a motion vector of a pixel block according to an embodiment of the present invention, and as shown in fig. 1, the method of estimating a motion vector of a pixel block of an embodiment of the present invention includes at least steps S1-S3, each of which is as follows.
In step S1, a plurality of candidate motion vectors of the current pixel block in the current video frame are obtained.
For example, since one moving object covers a plurality of pixel blocks, motion vectors of neighboring pixel blocks in a spatial domain have a strong correlation, and motion vectors of corresponding pixel blocks of neighboring frames in a temporal domain also have a strong correlation. In an embodiment, a three-dimensional recursive search algorithm, such as spatial recursive and temporal recursive methods, may be employed to estimate the motion vector of the current pixel block within the search range as the candidate motion vector.
In other embodiments, the motion vector obtained by the temporal-spatial domain recursion may be further added with an offset to obtain a motion vector of the current pixel block, which is also used as the candidate motion vector. Alternatively, other search methods may be used to obtain the candidate motion vector for the current pixel block.
In an embodiment, a hierarchical search algorithm is employed to obtain a down-sampled candidate motion vector for a current block of pixels as a candidate motion vector.
In step S2, the confidence level between the downsampled candidate motion vector and the other candidate motion vector except the downsampled candidate motion vector among the plurality of candidate motion vectors is calculated.
In step S3, the candidate motion vector with the highest confidence level with the downsampled candidate motion vector among the other candidate motion vectors is determined as the motion vector of the current pixel block.
In the embodiment, a hierarchical search algorithm is adopted to obtain a downsampling candidate motion vector, namely, a current frame image is downsampled to obtain a low-resolution image, the motion vector of each pixel block is estimated on the low-resolution image, namely, the downsampling frame image of the current video frame is obtained, the downsampling frame image is divided into a plurality of candidate blocks with the same size, and the block matching error of each candidate block in a search area in the current pixel block and the downsampling frame image is calculated; determining a target candidate block with the minimum matching error with the current pixel block; and estimating the motion vectors of the current pixel block and the target candidate block to serve as the down-sampling candidate motion vector.
The down-sampled frame image, i.e. the low-resolution image, is divided into a plurality of pixel blocks with the same size, compared with the high-resolution image, the number of data blocks is obviously reduced, the speed of block matching can be reduced, and the processing speed is improved.
As shown in fig. 2, in which pixel blocks at oblique lines in (a) of fig. 2 represent search blocks of a down-sampled frame image and pixel blocks at oblique lines in (b) of fig. 2 represent search blocks of an original image, it can be seen from fig. 2 that the search blocks of a next-sampled frame image have a wider range with respect to the original image, and the motion vectors obtained by motion estimation in a flat area have relatively good consistency but are less accurate than the motion vectors obtained by search of the original image.
In the motion estimation method, it is considered that the consistency of the motion vector of the down-sampled frame image is relatively high, and the reliability of the down-sampled candidate motion vector obtained by the hierarchical search algorithm is relatively high compared with other candidate motion vectors. Although the pixel blocks of the layered search are the same in size as the pixel blocks of the full-resolution image, the image information contained in the two pixel blocks is different, the block image information of the down-sampling frame image is richer, and the motion vector obtained by motion estimation can accurately reflect the motion trend of the current pixel block. However, since the pixel block obtained by the hierarchical search algorithm is a pixel block subjected to down-sampling, the accuracy of the motion vector obtained by the hierarchical search algorithm is not high. Therefore, in the embodiment of the present invention, the downsampling candidate motion vector obtained by the hierarchical search algorithm is used as the reference motion vector of the current pixel block, the confidence of the difference between the motion vector obtained by the hierarchical search algorithm and other candidate motion vectors is calculated, and the reliability of the candidate motion vector is determined.
In an embodiment, the reliability of the hierarchical search algorithm to obtain the downsampled candidate motion vector and the other candidate motion vectors may be determined by the following formula:
confi＝α×abs(cmvi-hmv), i ═ 1,2, Λ m equation (1)
Wherein confiRepresenting the ith candidate motion vector cmviHmv denotes the down-sampled candidate motion vector and alpha denotes the confidence coefficient.
Fig. 3 is a schematic diagram of a relationship curve between the difference value and the reliability of the down-sampled candidate motion vector and the other candidate motion vectors according to an embodiment of the present invention, and as shown in fig. 3, it can be seen that the larger the difference value between the other candidate motion vectors and the down-sampled candidate motion vector is, the lower the reliability of the candidate motion vector is, and the probability of being determined as the final motion vector of the current pixel block is reduced. Therefore, in the method for estimating a motion vector of a pixel block according to the embodiment of the present invention, the confidence level of the difference between other motion vectors and the down-sampled candidate motion vector and the block matching error of the candidate motion vector jointly determine the motion vector of the current block.
The lower sampling candidate motion vector is used as a reference motion vector, the estimation speed can be improved, the reliability is higher, and the candidate motion vector with higher reliability of the difference value of the lower sampling candidate motion vector is more reliable as the candidate motion vector is closer to the lower sampling candidate motion vector, and the candidate motion vector with the highest reliability of the difference value of the lower sampling candidate motion vector in other candidate motion vectors is determined to be the final motion vector of the current pixel block, so that the reliability of obtaining the motion vector can be ensured, the precision of the motion vector can be ensured, the reliability of motion estimation is improved, the video jitter can be reduced, and the video quality is improved.
In the embodiment, when motion estimation is performed on a certain pixel block, a large number of motion vectors which are possibly correct for the current pixel block, that is, candidate motion vectors, are estimated, the down-sampled candidate motion vectors are used as reference motion vectors, and the most suitable motion vector is selected from other candidate motion vectors as the motion vector of the current pixel block according to the above method, so that the reliability of the motion vector of the current pixel block is ensured, the precision of the motion vector can be ensured, and the precision of the motion estimation is improved.
In an embodiment, the candidate motion vectors of the current pixel block may comprise the above-mentioned down-sampled candidate motion vector and other candidate motion vectors, e.g. spatial recursive candidate motion vectors obtained using a spatial recursive algorithm, temporal recursive candidate motion vectors obtained using a temporal recursive algorithm.
In the motion vector estimation process, a motion vector of each pixel block is estimated sequentially from top to bottom and from left to right on the current video frame image.
In some embodiments, the motion vector used by the current block of pixels for spatial recursion may be expressed as the following formula:
MVspatial＝{MVr+i,c+j|(i<0or(i＝0and j<0),(i,j)∈Ωs) -formula (2);
wherein omegasRepresenting a specified spatial domain search range around the current pixel block, (r, c) representing the position of the current pixel block, (r + i, c + j) representing the position of the block within the specified search range, MVspatialRepresenting a spatial recursive candidate motion vector for the current block of pixels.
FIG. 4 is a diagram illustrating the search range for estimating a motion vector of a current pixel block by using a recursive search algorithm according to an embodiment of the present invention, as shown in FIG. 4, the spatial recursive search region of the current pixel block is a block region in which the motion vector has been estimated in the current video frameNamely, the position area (the diagonal line area in the figure) of the pixel block scanned in front of the current pixel block C is used as a space recursive search area; the motion vector of each candidate block in the spatial recursive search region is recursive to the current block of pixels of the current video frame using a spatial recursive method, such as equation (2) above, as a spatial recursive candidate motion vector for the current block of pixels. For example, as shown in fig. 4, it is assumed that the motion vector of a certain pixel block S is MV1Recursion of the motion vector of the pixel block S to the current pixel block C as a spatial recursion candidate motion vector CMV of the current pixel block Cs。
In an embodiment, the motion vector used by the current block of pixels for temporal recursion may be expressed as the following formula:
MVtemporal＝{MVr+i,c+j|(i＞0or(i＝0and j＞＝0),(i,j)∈Ωt) Equation (3)
Wherein omegatIndicating a specified time domain search range around the current pixel block, (r, c) indicating the position of the current pixel block, (r + i, c + j) indicating the position of the block within the specified search range, MVtemporalRepresenting a temporally recursive candidate motion vector for a current block of pixels.
In some embodiments, a block region scanned after a time-adjacent pixel block corresponding to the current pixel block in a previous video frame of the current video frame is taken as a time recursive search region of the current pixel block, for example, a white pixel block region in fig. 4 represents a pixel block region scanned after a pixel block corresponding to the current pixel block in the previous video frame, and this region is taken as the time recursive search region of the current pixel block, so that compared with selecting all pixel blocks of the previous video frame as the time recursive search region, the speed and the calculation amount for estimating the motion vector can be reduced. Further, the motion vector of each candidate block in the temporally recursive search region is recursive to the current pixel block of the current video frame using a temporal recursive method such as equation (3) above as a temporally recursive candidate motion vector for the current pixel block. For example, in the temporal recursive search range of the current pixel block, the motion vector of one pixel block T is assumed to be MV2Recursion of the motion vector of the pixel block T to the current pixel block C as the time of the current pixel block CInter-recursive candidate motion vector CMVt。
In the embodiment of the present invention, the difference reliability calculation may be performed on the spatial recursive candidate motion vector, the temporal recursive candidate motion vector, and the down-sampling candidate motion vector of the current pixel block, and it is determined which candidate motion vector is closer to the down-sampling candidate motion vector, that is, it is more likely to be determined as the motion vector of the current pixel block.
The above spatial recursive candidate motion vector and temporal recursive candidate motion vector are used as candidate motion vectors of the current pixel block, and the motion of the neighboring blocks is relatively continuous, for example, the motion directions of the neighboring blocks in the background area are relatively consistent. However, for some changes in the image, such as texture, edges, etc., it is more desirable that the candidate motion vectors diverge more, helping to find the appropriate motion vector for the current pixel block more accurately.
Thus, in some embodiments, a pixel block at a specified position within a search range around a current pixel block is added with a random offset to the motion vector of the specified block, e.g., the recursive motion vector that obtains the pixel block at the specified position within a preset neighborhood around the current pixel block may be, for example, a temporal recursive candidate motion vector or a spatial recursive candidate motion vector; the recursive motion vector is added with a random motion offset to obtain a random candidate motion vector for the current block of pixels.
The random candidate motion vector may be obtained by the following formula:
MVrandom＝{MVr+i,c+j+Ψ|(i,j)∈Ωr) -formula (4);
wherein omegarRepresenting a specified search range, e.g. a preset neighborhood, around the current pixel block, (r, c) representing the position of the current pixel block, and (r + i, c + j) representing the position of the block within the specified search range. Ψ denotes a random motion offset, MV, added to the recursive motion vector of a surrounding blockrandomA random candidate motion vector representing the current block of pixels.
The random candidate motion vectors increase the divergence of the motion estimation for the current block of pixels, providing more possibilities for the current block of pixels to find the correct motion vector.
Further, in the embodiment, the confidence levels of the other candidate motion vectors except the down-sampled candidate motion vector among the plurality of candidate motion vectors and the down-sampled candidate motion vector may be calculated with reference to the above formula (1), for example, a first difference absolute value of the spatial recursive candidate motion vector and the down-sampled candidate motion vector is calculated, and the first confidence level of the spatial recursive candidate motion vector and the down-sampled candidate motion vector is obtained according to the first difference absolute value; calculating a second difference absolute value of the temporal recursion candidate motion vector and the downsampling candidate motion vector, and obtaining a second reliability of the temporal recursion candidate motion vector and the downsampling candidate motion vector according to the second difference absolute value; and calculating a third difference absolute value of the random candidate motion vector and the downsampled candidate motion vector, and obtaining a third reliability of the random candidate motion vector and the downsampled candidate motion vector according to the third difference absolute value.
Further, obtaining the highest credibility of the first credibility, the second credibility and the third credibility; and determining the candidate motion vector corresponding to the highest credibility as the motion vector of the current pixel block.
In the above embodiment, the down-sampling candidate motion vector is used as the reference motion vector, and the temporal recursive candidate motion vector, the spatial recursive candidate motion vector, and the random candidate motion vector are used as the selection motion vector to obtain the motion vector of the current pixel block. In the embodiment, other candidate motion vectors are obtained through other algorithms, and the difference reliability determination is performed on the candidate motion vectors and the downsampled candidate motion vectors to obtain the final motion vector of the current pixel block, which is also included in the protection scope of the present invention.
According to the method for estimating the motion vector of the pixel block, the candidate motion vector which is closest to the down-sampling candidate motion vector is obtained as the motion vector of the current pixel block by taking the down-sampling candidate motion vector as the reference motion vector, so that the reliability of the motion vector can be ensured, the precision of the motion vector can be ensured, the more appropriate motion vector of the current pixel block can be obtained, the reliability of motion estimation is improved, the video jitter can be reduced, and the video smoothness is improved.
The second aspect embodiment of the present invention also proposes a non-transitory computer storage medium, on which a computer program is stored, which when executed by a processor implements the method for estimating a motion vector of a pixel block of the above embodiments.
The third aspect of the present invention also provides a video processing apparatus, as shown in fig. 5, which is a block diagram of a video processing apparatus according to an embodiment of the present invention, the video processing apparatus 10 includes a processor 11 and a memory 12 communicatively connected to the processor 11, wherein the memory 12 stores therein a computer program executable by the processor 11, and the computer program, when executed by the processor 11, implements the method for estimating a motion vector of a pixel block according to the above embodiment.
The computer program in the memory 12 described above may be implemented in the form of software functional units and stored in a computer readable storage medium when sold or used as a stand-alone product.
The memory 12 is a computer-readable storage medium and can be used for storing software programs, computer-executable programs, such as program instructions/modules corresponding to the methods in the embodiments of the present invention. The processor 11 executes functional applications and data processing, i.e. implements the method of estimating motion vectors of pixel blocks in the above-described method embodiments, by running software programs, instructions and modules stored in the memory 12.
The memory 12 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal device, and the like. Further, the memory 12 may include a high speed random access memory and may also include a non-volatile memory.
The technical solution of the embodiment of the present invention may be embodied in the form of a software product, where the computer software product is stored in a storage medium and includes one or more instructions for enabling a computer device or an image processing device to execute all or part of the steps of the method according to the embodiment of the present invention.
The fourth aspect of the present invention also provides an electronic device, in which the electronic device may include some devices including a television, a projector, a computer, and the like, which include image processing or video processing, and image display.
As shown in fig. 6, the electronic device 100 according to an embodiment of the present invention includes an image display apparatus 20 and the video processing apparatus 10 according to the above embodiment, the video processing apparatus 10 is connected to the image display apparatus 20, the image processing apparatus 10 processes a video image to be displayed, and specifically, the method for estimating motion vectors of pixel blocks according to the above embodiment may be performed, and the processed video image is transmitted to the image display apparatus 20 in frames, and the image display apparatus 20 is configured to display a video screen image.
The video processing device 10 obtains a plurality of candidate motion vectors of a current pixel block in a current video frame, wherein the plurality of candidate motion vectors at least comprise a downsampling candidate motion vector; calculating the reliability of the difference value between the other candidate motion vectors except the down-sampling candidate motion vector in the plurality of candidate motion vectors and the down-sampling candidate motion vector; and determining the candidate motion vector with the highest reliability of the difference value between the other candidate motion vectors and the down-sampling candidate motion vector as the motion vector of the current pixel block.
The electronic device 100 of the embodiment of the present invention, using the video processing apparatus 10 of the above embodiment, may use the lower sampling candidate motion vector as the reference motion vector, which may improve the estimation speed, and the higher the reliability of the difference between the candidate motion vector and the lower sampling candidate motion vector, the closer to the lower sampling candidate motion vector, the more reliable the candidate motion vector is, and determine the candidate motion vector with the highest reliability of the difference between the candidate motion vector and the lower sampling candidate motion vector in the other candidate motion vectors as the final motion vector of the current pixel block, thereby ensuring the reliability of the obtained motion vector and the precision of the motion vector, improving the reliability of the motion estimation, and further reducing the video jitter and improving the video quality.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an illustrative embodiment," "an example," "a specific example," or "some examples" or the like mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example.
While embodiments of the invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
Claims (10)
1. A method of estimating a motion vector for a block of pixels, comprising:
obtaining a plurality of candidate motion vectors of a current pixel block in a current video frame, wherein the plurality of candidate motion vectors at least comprise down-sampling candidate motion vectors;
calculating the reliability of the difference value between the other candidate motion vectors except the down-sampling candidate motion vector and the down-sampling candidate motion vector in a plurality of candidate motion vectors;
and determining the candidate motion vector with the highest reliability of the difference value between the other candidate motion vectors and the downsampled candidate motion vector as the motion vector of the current pixel block.
2. The method of estimating a motion vector for a block of pixels as claimed in claim 1 wherein obtaining a plurality of candidate motion vectors for a current block of pixels in a current video frame comprises:
obtaining a downsampled frame image of the current video frame;
dividing the downsampled frame image into a plurality of candidate blocks of the same size;
and calculating the downsampling candidate motion vector of the current pixel block by adopting a hierarchical search algorithm.
3. The method of estimating a motion vector for a pixel block according to claim 2, wherein obtaining a plurality of candidate motion vectors for a current pixel block in a current video frame further comprises:
taking a block region of the current video frame, in which a motion vector has been estimated, as a spatial recursive search region of the current pixel block;
and recursing the motion vector of each candidate block in the spatial recursive search area to the current pixel block of the current video frame by adopting a spatial recursion method to be used as the spatial recursive candidate motion vector of the current pixel block.
4. The method of estimating a motion vector for a pixel block according to claim 3, wherein obtaining a plurality of candidate motion vectors for a current pixel block in a current video frame further comprises:
taking a block area scanned after a time-adjacent pixel block corresponding to the current pixel block in a previous video frame of the current video frame as a time recursive search area of the current pixel block;
and recursing the motion vector of each candidate block in the time recursive search area to the current pixel block of the current video frame by adopting a time recursion method to be used as the time recursive candidate motion vector of the current pixel block.
5. The method of estimating a motion vector for a pixel block according to claim 4, wherein obtaining a plurality of candidate motion vectors for a current pixel block in a current video frame further comprises:
acquiring a recursive motion vector of a pixel block at a preset neighborhood internal position around the current pixel block;
adding a random motion offset to the recursive motion vector to obtain a random candidate motion vector for the current block of pixels.
6. The method of estimating a motion vector for a pixel block according to claim 5, wherein calculating the confidence level between the motion vector candidates other than the down-sampled motion vector candidate and the down-sampled motion vector candidate comprises:
calculating a first difference absolute value of the spatial recursive candidate motion vector and the downsampled candidate motion vector, and obtaining a first reliability of the spatial recursive candidate motion vector and the downsampled candidate motion vector according to the first difference absolute value;
calculating a second difference absolute value between the temporal recursion candidate motion vector and the downsampling candidate motion vector, and obtaining a second reliability of the temporal recursion candidate motion vector and the downsampling candidate motion vector according to the second difference absolute value;
and calculating a third difference absolute value of the random candidate motion vector and the downsampled candidate motion vector, and obtaining a third reliability of the random candidate motion vector and the downsampled candidate motion vector according to the third difference absolute value.
7. The method of estimating a motion vector for a pixel block according to claim 6, wherein determining the motion vector candidate with the highest confidence level among the other motion vector candidates and the motion vector candidate with the highest confidence level among the motion vector candidates to be the motion vector for the current pixel block comprises:
obtaining a highest confidence level of the first confidence level, the second confidence level, and the third confidence level;
and determining the candidate motion vector corresponding to the highest credibility as the motion vector of the current pixel block.
8. A non-transitory computer storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the method of estimating a motion vector for a block of pixels of any one of claims 1-7.
9. A video processing apparatus, comprising:
a processor;
a memory communicatively coupled to the processor;
wherein the memory has stored therein a computer program executable by the processor, the computer program when executed by the processor implementing the method of estimating a motion vector for a block of pixels according to any one of claims 1 to 7.
10. An electronic device characterized by comprising an image display apparatus and the video processing apparatus of claim 9, the video processing apparatus being connected to the image display apparatus.
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010687720.5A CN113949869B (en) | 2020-07-16 | 2020-07-16 | Method for estimating motion vector of pixel block, video processing device, apparatus and medium |
| US17/305,751 US11653017B2 (en) | 2020-07-16 | 2021-07-14 | Method, video processing apparatus, device, and medium for estimating a motion vector of a pixel block |
| EP21186131.5A EP3941062A1 (en) | 2020-07-16 | 2021-07-16 | Method,video processing apparatus, device, and medium for estimating a motion vector of a pixel block |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010687720.5A CN113949869B (en) | 2020-07-16 | 2020-07-16 | Method for estimating motion vector of pixel block, video processing device, apparatus and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113949869A true CN113949869A (en) | 2022-01-18 |
| CN113949869B CN113949869B (en) | 2024-04-26 |
Family
ID=76958823
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010687720.5A Active CN113949869B (en) | 2020-07-16 | 2020-07-16 | Method for estimating motion vector of pixel block, video processing device, apparatus and medium |
Country Status (3)
| Country | Link |
|---|---|
| US (1) | US11653017B2 (en) |
| EP (1) | EP3941062A1 (en) |
| CN (1) | CN113949869B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1801939A (en) * | 2005-12-22 | 2006-07-12 | 天津大学 | Video transformation coding method based on spatial resolution transformation |
| CN101198064A (en) * | 2007-12-10 | 2008-06-11 | 武汉大学 | Movement vector prediction method in resolution demixing technology |
| CN101884219A (en) * | 2007-10-16 | 2010-11-10 | Lg电子株式会社 | Handle the method and apparatus of vision signal |
| EP2343900A1 (en) * | 2010-01-08 | 2011-07-13 | Research In Motion Limited | Method and device for motion vector estimation in video transcoding using full-resolution residuals |
| CN104811726A (en) * | 2015-04-24 | 2015-07-29 | 宏祐图像科技(上海)有限公司 | Method for selecting candidate motion vectors of motion estimation in frame rate conversion process |
| CN104918054A (en) * | 2013-10-03 | 2015-09-16 | 晶晨半导体(上海)有限公司 | Motion compensated de-interlacing and noise reduction |
| US20180020229A1 (en) * | 2016-07-14 | 2018-01-18 | Sharp Laboratories Of America, Inc. | Computationally efficient motion compensated frame rate conversion system |
| US20190045223A1 (en) * | 2018-09-25 | 2019-02-07 | Intel Corporation | Local motion compensated temporal noise reduction with sub-frame latency |
| US20190045211A1 (en) * | 2017-08-03 | 2019-02-07 | Samsung Electronics Co., Ltd. | Motion estimation method and apparatus for plurality of frames |
| US20190141332A1 (en) * | 2017-11-08 | 2019-05-09 | Qualcomm Incorporated | Use of synthetic frames in video coding |
| US20190164296A1 (en) * | 2017-11-27 | 2019-05-30 | Qualcomm Incorporated | Systems and methods for determining a confidence measure for a motion vector |
| US20190215518A1 (en) * | 2018-01-10 | 2019-07-11 | Qualcomm Incorporated | Histogram of gradient based optical flow |
| CN111343465A (en) * | 2018-12-18 | 2020-06-26 | 三星电子株式会社 | Electronic circuit and electronic device |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8358698B2 (en) * | 2010-01-08 | 2013-01-22 | Research In Motion Limited | Method and device for motion vector estimation in video transcoding using full-resolution residuals |
| US9621917B2 (en) * | 2014-03-10 | 2017-04-11 | Euclid Discoveries, Llc | Continuous block tracking for temporal prediction in video encoding |
| WO2020182043A1 (en) * | 2019-03-08 | 2020-09-17 | Huawei Technologies Co., Ltd. | Search region for motion vector refinement |
| TWI768324B (en) * | 2020-04-16 | 2022-06-21 | 瑞昱半導體股份有限公司 | Image processing method and image processing device |
-
2020
- 2020-07-16 CN CN202010687720.5A patent/CN113949869B/en active Active
-
2021
- 2021-07-14 US US17/305,751 patent/US11653017B2/en active Active
- 2021-07-16 EP EP21186131.5A patent/EP3941062A1/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1801939A (en) * | 2005-12-22 | 2006-07-12 | 天津大学 | Video transformation coding method based on spatial resolution transformation |
| CN101884219A (en) * | 2007-10-16 | 2010-11-10 | Lg电子株式会社 | Handle the method and apparatus of vision signal |
| CN101198064A (en) * | 2007-12-10 | 2008-06-11 | 武汉大学 | Movement vector prediction method in resolution demixing technology |
| EP2343900A1 (en) * | 2010-01-08 | 2011-07-13 | Research In Motion Limited | Method and device for motion vector estimation in video transcoding using full-resolution residuals |
| CN104918054A (en) * | 2013-10-03 | 2015-09-16 | 晶晨半导体(上海)有限公司 | Motion compensated de-interlacing and noise reduction |
| CN104811726A (en) * | 2015-04-24 | 2015-07-29 | 宏祐图像科技(上海)有限公司 | Method for selecting candidate motion vectors of motion estimation in frame rate conversion process |
| US20180020229A1 (en) * | 2016-07-14 | 2018-01-18 | Sharp Laboratories Of America, Inc. | Computationally efficient motion compensated frame rate conversion system |
| US20190045211A1 (en) * | 2017-08-03 | 2019-02-07 | Samsung Electronics Co., Ltd. | Motion estimation method and apparatus for plurality of frames |
| US20190141332A1 (en) * | 2017-11-08 | 2019-05-09 | Qualcomm Incorporated | Use of synthetic frames in video coding |
| US20190164296A1 (en) * | 2017-11-27 | 2019-05-30 | Qualcomm Incorporated | Systems and methods for determining a confidence measure for a motion vector |
| US20190215518A1 (en) * | 2018-01-10 | 2019-07-11 | Qualcomm Incorporated | Histogram of gradient based optical flow |
| US20190045223A1 (en) * | 2018-09-25 | 2019-02-07 | Intel Corporation | Local motion compensated temporal noise reduction with sub-frame latency |
| CN111343465A (en) * | 2018-12-18 | 2020-06-26 | 三星电子株式会社 | Electronic circuit and electronic device |
Non-Patent Citations (4)
| Title |
|---|
| ARUN NETRAVALI，ET AL: "A CODEC FOR HDTV", 《IEEE》 * |
| STILLER，ET AL: "ESTIMATING MOTION IN IMAGE SEQUENCES-A TUTORIAL ON MODELING AND COMPUTATION OF 2D MOTION", 《IEEE》, pages 83 - 85 * |
| 因雅晶: "帧率上转换中的运动估计算法研究", 《中国优秀硕士学位论文全文数据库（电子期刊）信息科技辑》, pages 23 - 41 * |
| 赵祖麟: "基于三维递归搜索的帧率上转换算法研究", 《中国优秀硕士学位论文全文数据库（电子期刊）信息科技辑》, pages 13 - 25 * |
Also Published As
| Publication number | Publication date |
|---|---|
| US20220021898A1 (en) | 2022-01-20 |
| EP3941062A1 (en) | 2022-01-19 |
| CN113949869B (en) | 2024-04-26 |
| US11653017B2 (en) | 2023-05-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP4997281B2 (en) | Method for determining estimated motion vector in image, computer program, and display device | |
| US8724022B2 (en) | Frame rate conversion using motion estimation and compensation | |
| JP2009527173A (en) | Method and apparatus for determining motion between video images | |
| JP2005503085A (en) | Motion estimation and / or compensation | |
| KR100727795B1 (en) | Motion estimation | |
| WO2008152951A1 (en) | Method of and apparatus for frame rate conversion | |
| EP1549047A2 (en) | Robust camera pan vector estimation using iterative center of mass | |
| Tai et al. | A multi-pass true motion estimation scheme with motion vector propagation for frame rate up-conversion applications | |
| Braspenning et al. | True-motion estimation using feature correspondences | |
| US20150110190A1 (en) | Method and apparatus for motion estimation | |
| KR20050012766A (en) | Unit for and method of estimating a motion vector | |
| WO2003102872A2 (en) | Unit for and method of estimating a motion vector | |
| WO1999040726A2 (en) | Motion or depth estimation | |
| KR20050097936A (en) | Efficient predictive image parameter estimation | |
| KR100942887B1 (en) | Motion estimation | |
| JP4796696B2 (en) | Method and apparatus for computer-aided motion compensation of digitized images and computer-readable recording medium | |
| US20080144716A1 (en) | Method For Motion Vector Determination | |
| JP2008504786A (en) | Motion compensation by video mode detection | |
| EP1606952A1 (en) | Method for motion vector determination | |
| CN113949869A (en) | Method for estimating motion vector of pixel block, video processing apparatus, video processing device, and medium | |
| KR20060103430A (en) | Motion vector fields refinement to track small fast moving objects | |
| JP5448983B2 (en) | Resolution conversion apparatus and method, scanning line interpolation apparatus and method, and video display apparatus and method | |
| JP2004533073A (en) | Feature point selection | |
| US20110109794A1 (en) | Caching structure and apparatus for use in block based video | |
| JPH08242454A (en) | Method for detecting global motion parameter |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |