CN102111613B - Image processing method and device - Google Patents

Abstract

The invention provides an image processing method and an image processing device. The method comprises the following steps of: performing motion estimation on a first frame of image and a second frame of image to obtain motion vectors of the first and second frames of images; motioning the first frame of image by a part of distance of the motion vector along a motion track indicated by the motion vector to obtain an irregular motion vector field of an interpolated frame of image between the first and second frames of images; re-sampling the irregular motion vector field to obtain a regular motion vector field of the interpolated frame of image; and performing motion compensation on the regular motion vector field to obtain the interpolated frame of image. By the scheme, computational complexity is reduced, and simultaneously the accuracy of the output motion vector field is improved.

Description

Image processing method and device

Technical field

The present invention relates to data service and handle, be meant a kind of image processing method and device especially.

Background technology

Transfer algorithm mainly just is used for the conversion between the different frame rates videoscanning form on the frame rate; Can in two frames or multi-frame video image, produce the frame of the state that mediates; As shown in Figure 1; Video frame images 11 is the A place in the position, and video frame images 13 is the B place in the position, and video frame images 12 is exactly the frame of the intermediateness of video frame images 11 and 13.

For example, in the low bit-rate video communication field, owing to be limited by limited transmission bandwidth; The frame speed of video is lower usually; Through frame rate is increased to 30fps, can eliminate the discontinuous and sense of jumping that occurs when picture shows, guarantee smooth more dynamic effect.

Transfer algorithm mainly contains two big types on the present frame speed: one type is transfer algorithm on the simple frame rate, comprising: frame iterative method and time domain linear/nonlinear interpolation; Though this kind algorithm fast operation, owing to do not consider the motion of object, phenomenon can appear twitching, fuzzy etc. in the moving region in interleave.

In order to address this problem, occurred second type based on transfer algorithm (MC-FRUC) on the frame rate of motion compensation, the movable information between considered frame carries out interpolation, and wherein, estimation and motion compensation are the keys of this type of algorithm performance quality of decision.

Estimation is the basis that obtains interframe movement information; The most frequently used method that is based on the piece coupling, this method is the five equilibrium macro block of non-overlapping copies with image division, supposes that the motion in each piece is identical; Each piece is carried out match search according to certain standard, thereby obtain motion vector.

Block matching method has that the displacement tracking ability is strong, computational complexity is low and be easy to advantages such as hardware realization, therefore, is widely used at video coding and video analysis field.

But the estimation in the MC-FRUC algorithm is greatly different with estimation in the video coding; Estimation in the MC-FRUC algorithm is an intermediate frame image of inserting out The subjective quality for interior, and the estimation in the video coding then is in order to obtain the least residual image.Therefore the MC-FRUC algorithm is more prone to obtain the real motion of object in the level and smooth motion vector field reflection image.If traditional estimation based on block matching algorithm is directly applied in the MC-FRUC algorithm; Overlapping or empty problem can appear in the image of interpolation frame; As shown in Figure 2; Video frame images 21 has been divided into video image blocks some, because the direction of motion of each video image blocks possibly be arbitrarily, the video image blocks in the video frame images 22 that estimation goes out has just occurred overlapping 221 or cavity 222; And because the defective of piece processing itself, the continuity of motion vector is relatively poor, causes that the edge of piece produces tangible blocking effect in interpolation frame.Traditional piece estimation only is fit to common translational motion, is not suitable for compound movements such as rotation, convergent-divergent and distortion.

Therefore, the inventor finds especially to produce the technology of interpolation frame in the conventional images treatment technology in realizing process of the present invention, and motion estimation complexity is high, and the motion vector of generation is difficult to accurately express real motion; Cavity and overlapping problem appear in block-based motion compensation; Blocking effect and the image blurring aesthetic quality who reduces video.

Summary of the invention

The technical problem that the present invention will solve provides the image processing method and the device of demands on a kind of suitable frame rate, when reducing computation complexity, improves the accuracy of output movement vector field.

For solving the problems of the technologies described above, embodiments of the invention provide a kind of image processing method, comprising:

First two field picture and second two field picture are carried out estimation, obtain motion vector from said first two field picture to said second two field picture;

Said first two field picture along the move part distance of said motion vector of the indicated movement locus of said motion vector, is obtained the irregular movement vector field of the interpolation frame image between said first two field picture and said second two field picture;

Said irregular movement vector field is resampled, obtain the rule motion vector field of said interpolation frame image;

Motion compensation is carried out in said rule motion vector field, obtain said interpolation frame image.

Wherein, first two field picture and second two field picture are carried out estimation, the motion vector step that obtains from said first two field picture to said second two field picture comprises:

Said first two field picture is divided at least one first image block, wherein, one second image block in corresponding said second two field picture of one first image block;

Said first image block and said second image block are carried out estimation, obtain piece motion vector from said first image block to said second image block.

Wherein, said first image block and said second image block are carried out estimation, the step that obtains the piece motion vector from said first image block to said second image block comprises:

A size information and a displacement to be selected according to said first image block and said second image block carry out estimation to said first image block and said second image block, obtain the piece motion vector from said first image block to said second image block.

Wherein, said displacement to be selected is: in the hunting zone that includes at least one second image block, and one second image block of said first image block coupling and the relative displacement between said first image block.

Wherein, second image block with said first image block coupling is meant: in said hunting zone, with the second maximum image block of the said first image block similarity.

Wherein, second image block maximum with the said first image block similarity is meant: in said hunting zone, SAD (i, j) second image block of minimum, wherein,

$\mathrm{SAD}(i,j)=\underset{m=0}{\overset{M}{\mathrm{\Σ}}}\underset{n=0}{\overset{N}{\mathrm{\Σ}}}|b_{t-1}(m,n)-b_{t+1}(m+i,n+j)|;$

Wherein, b _T-1Represent first image block, b _T+1Represent second image block, i, j represent the displacement that certain is to be selected, M presentation video block length, and the width of N presentation video piece, SAD (i, j) calculate in first image block and second image block in skew (i, j) the matching error sum of following all pixels by expression.

Wherein, Along the move part distance of said motion vector of the indicated movement locus of said motion vector, the step that obtains the irregular movement vector field of the interpolation frame image between said first two field picture and said second two field picture comprises with said first two field picture:

Said first image block along the said movement locus that motion vector is indicated, is moved to the intermediate value place of said motion vector, obtain corresponding with said first image block, the interpolated image piece between said first image block and said second image block;

The interpolated image piece that all first image blocks are corresponding forms the irregular movement vector field of said interpolation frame image.

Wherein, said irregular movement vector field is resampled, the step that obtains the rule motion vector field of said interpolation frame image is specially:

Pass through formula: ${\mathrm{\α}}_i=\mathrm{Exp}(-\frac{d_i^2}{{\stackrel{\‾}{d}}^2}),$ ${\mathrm{\β}}_i=\frac{{\mathrm{\α}}_i}{\underset{i=0}{\overset{n}{\mathrm{\Σ}}}{\mathrm{\α}}_i},$ ${\stackrel{\→}{v}}_g=\underset{i=0}{\overset{n}{\mathrm{\Σ}}}{\mathrm{\β}}_i{\stackrel{\→}{v}}_i;$ Said irregular movement vector field is resampled, obtain the motion vector on the summit of the interpolated image piece in the said rule motion vector field;

The motion vector on all interpolated image pieces and summit thereof forms the rule motion vector field of said interpolation frame image;

Wherein, d _iThe distance of the summit V of the interpolated image piece of the central point Vi that representes interpolated image piece in the said irregular movement vector field in the corresponding rule motion vector field of this interpolated image piece, d representes all d _iMean value; N is greater than 0 and less than the positive integer of the maximum quantity of interpolated image piece,

The motion vector of interpolated image piece central point Vi in the expression irregular movement vector field;

The motion vector of representing the summit V of the interpolated image piece in the said rule motion vector field.

Wherein, motion compensation is carried out in said rule motion vector field, the step that obtains said interpolation frame image comprises:

The motion vector of interpolated image piece in the said rule motion vector field according to the summit of this interpolated image piece moved; Obtain said interpolated image piece corresponding reference image block, said reference image block is the second corresponding image block of said interpolated image piece or first image block;

According to said reference image block, this interpolated image piece in the said rule motion vector field is carried out motion compensation;

After all interpolated image pieces in the rule motion vector field are carried out motion compensation, obtain said interpolation frame image.

Wherein, according to said reference image block, the step that this interpolated image piece in the said rule motion vector field is carried out motion compensation is specially:

According to formula: f (P ')=f (A ') (1-x) (1-y)+f (B ') x (1-y)+f (C ') (1-x) y+f (D ') xy motion compensation is carried out in said rule motion vector field;

Wherein, Arbitrary pixel in this interpolated image piece corresponding reference image block that interpolated image piece in the P ' expression rule motion vector field obtains after moving according to the motion vector on the summit of this interpolated image piece; F (P ') representes the coordinate of this pixel P ', the coordinate of the summit A ' of this reference image block of f (A '); F (B ') representes the coordinate of the summit B ' of this reference image block, and f (C ') representes the coordinate of the summit C ' of this reference image block; F (D ') representes the coordinate of the summit D ' of this reference image block;

X representes the abscissa of pixel P corresponding with pixel P ' in this interpolated image piece in the rule motion vector field, the ordinate of y remarked pixel P.

Embodiments of the invention also provide a kind of image processing apparatus, comprising:

First processing module is used for first two field picture and second two field picture of two continuous frames image are carried out estimation, obtains the motion vector from said first two field picture to said second two field picture;

Second processing module; Be used for said first two field picture obtaining the irregular movement vector field of the interpolation frame image between said first two field picture and said second two field picture along the move part distance of said motion vector of the indicated movement locus of said motion vector;

The 3rd processing module is used for said irregular movement vector field is resampled, and obtains the rule motion vector field of said interpolation frame image;

Manages module everywhere, is used for motion compensation is carried out in said rule motion vector field, obtains said interpolation frame image.

Wherein, said first processing module comprises:

Divide module, be used for said first two field picture is divided at least one first image block, said second two field picture is divided at least one second image block;

Processing sub is used for said first image block and said second image block are carried out estimation, obtains the piece motion vector from said first image block to said second image block.

Wherein, Said second processing module specifically is used for; With the said movement locus that motion vector is indicated in said first image block edge; Move to the intermediate value place of said motion vector, obtain corresponding with said first image block, the interpolated image piece between said first image block and said second image block; The interpolated image piece that all first image blocks are corresponding forms the irregular movement vector field of said interpolation frame image.

Wherein, said the 3rd processing module specifically is used for,

Pass through formula: ${\mathrm{\α}}_i=\mathrm{Exp}(-\frac{d_i^2}{{\stackrel{\‾}{d}}^2}),$ ${\mathrm{\β}}_i=\frac{{\mathrm{\α}}_i}{\underset{i=0}{\overset{n}{\mathrm{\Σ}}}{\mathrm{\α}}_i},$ ${\stackrel{\→}{v}}_g=\underset{i=0}{\overset{n}{\mathrm{\Σ}}}{\mathrm{\β}}_i{\stackrel{\→}{v}}_i,$ Said irregular movement vector field is resampled, obtain the motion vector on the summit of the interpolated image piece in the said rule motion vector field;

The motion vector on all interpolated image pieces and summit thereof forms the rule motion vector field of said interpolation frame image;

Wherein, d _iThe distance of the summit V of the interpolated image piece of the central point Vi that representes interpolated image piece in the said irregular movement vector field in the corresponding rule motion vector field of this interpolated image piece, d representes all d _iMean value; N is greater than 0 and less than the positive integer of the maximum quantity of interpolated image piece,

The motion vector of interpolated image piece central point Vi in the expression irregular movement vector field; The motion vector of representing the summit V of the interpolated image piece in the said rule motion vector field.

Wherein, said manage module everywhere and comprise:

Mapping block; Be used for the interpolated image piece of the said rule motion vector field motion vector according to the summit of this interpolated image piece is moved; Obtain said interpolated image piece corresponding reference image block, said reference image block is the second corresponding image block of said interpolated image piece or first image block;

The motion compensation process submodule is used for according to said reference image block, and this interpolated image piece in the said rule motion vector field is carried out motion compensation; After all interpolated image pieces in the rule motion vector field are carried out motion compensation, obtain said interpolation frame image.

The beneficial effect of technique scheme of the present invention is following:

In the such scheme, resample, obtain the rule motion vector field of said interpolation frame image, so just eliminated the overlapping or cavitation that produces in the motion estimation process through irregular movement vector field to said interpolation frame image; Again motion compensation is carried out in said rule motion vector field, obtain said interpolation frame image; Reduce computation complexity, improved the output quality of whole system operation efficient and image.

Description of drawings

Fig. 1 is for producing the design sketch of the two field picture of the state that mediates in two two field pictures in the prior art;

Fig. 2 is in the prior art, the overlapping or cavitation figure that occurs in the motion vector field of the two field picture of intermediateness;

Fig. 3 is an image processing method flow chart of the present invention;

Fig. 4 is a concrete realization flow figure of image processing method shown in Figure 3;

Fig. 5 is the estimation matching process figure based on image block;

Fig. 6 is divided into two staggered set, neighbours territory graphs of a relation each other for the image block in the two field picture;

Fig. 7 is the state diagram of the irregular movement vector field of formation after image block moves along motion vector direction;

Fig. 8 is the resampling procedure chart of the summit motion vector of image block;

Fig. 9 is that the computational process of summit motion vector in the resampling process shown in Figure 8 realizes figure;

Figure 10 is the motion compensation sketch map based on distortion of the mesh;

Figure 11 is in the movement compensation process shown in Figure 10, the pixel computational process figure in the middle interpolation frame;

Figure 12 is the structural representation of embodiments of the invention image processing apparatus.

Embodiment

For technical problem, technical scheme and advantage that the present invention will be solved is clearer, be described in detail below in conjunction with accompanying drawing and specific embodiment.

The present invention is directed in the technology of existing generation interpolation frame image; Motion estimation complexity is high; The motion vector that produces is difficult to accurately express the real motion problem; The image processing method and the device of demands on a kind of suitable frame rate are provided, when reducing computation complexity, improve the accuracy of output movement vector field.

As shown in Figure 3, the embodiments of the invention image processing method comprises:

Step 31 is carried out estimation to first two field picture and second two field picture, obtains the motion vector from said first two field picture to said second two field picture;

Step 32 along the move part distance of said motion vector of the indicated movement locus of said motion vector, obtains the irregular movement vector field of the interpolation frame image between said first two field picture and said second two field picture with said first two field picture;

Step 33 resamples to said irregular movement vector field, obtains the rule motion vector field of said interpolation frame image;

Step 34 is carried out motion compensation to said rule motion vector field, obtains said interpolation frame image.

Among this embodiment of the present invention, resample, obtain the rule motion vector field of said interpolation frame image, so just eliminated the overlapping or cavitation that produces in the motion estimation process through irregular movement vector field to said interpolation frame image; Again motion compensation is carried out in said rule motion vector field, obtain said interpolation frame image; When can reduce the estimation computation complexity like this, improve the accuracy of output movement vector field.

As shown in Figure 4; In the above embodiment of the present invention; Two width of cloth image f (t-1) and f (t+1) under known moment t-1 and t+1 find the solution the two field picture f (t) under marginal certain moment t, and f (t) is called the interpolation frame image; Prior image frame f (t-1) that being input as of entire process process treated interpolation frame and back two field picture f (t+1), the last new images f (t) that calculates that is output as;

Wherein, to the estimation of the first two field picture f (t-1) and the second two field picture f (t+1), can be based on the estimation of piece coupling;

Above-mentioned steps 31 can specifically comprise:

Step 311 is divided at least one first image block with said first two field picture, wherein, and one second image block in corresponding said second two field picture of one first image block;

Step 312 is carried out estimation to said first image block and said second image block, obtains the piece motion vector from said first image block to said second image block.

Specifically, as shown in Figure 5, above-mentioned steps 312 is specially:

A size information and a displacement to be selected according to said first image block and said second image block carry out estimation to said first image block and said second image block, obtain the piece motion vector from said first image block to said second image block;

Wherein, said displacement to be selected is: in the hunting zone that includes at least one second image block, and one second image block of said first image block coupling and the relative displacement between said first image block.

Wherein, second image block with said first image block coupling is meant: in said hunting zone, with the second maximum image block of the said first image block similarity.

Wherein, second image block maximum with the said first image block similarity is meant: in said hunting zone, SAD (i, j) second image block of minimum, wherein,

$\mathrm{SAD}(i,j)=\underset{m=0}{\overset{M}{\mathrm{\Σ}}}\underset{n=0}{\overset{N}{\mathrm{\Σ}}}|b_{t-1}(m,n)-b_{t+1}(m+i,n+j)|;$

Wherein, b _T-1Represent first image block, b _T+1Represent second image block, i, j represent the displacement that certain is to be selected, the length of M presentation video piece, and the width of N presentation video piece, SAD (i, j) expression is calculated in first image block and second image block in skew (i, j) the matching error sum of following all pixels;

That is to say; Import two two field picture f (t-1) and f (t+1); As present frame, back one frame f (t-1) is frame as a reference with f (t-1), current frame image is divided into the image block (being at least one first image block) of many non-overlapping copies; And think that the displacement of interior all pixels of first image block is all identical; Then to each first image block in the present frame, (this hunting zone can comprise at least one second image block) found out and the second the most similar image block of current first image block, i.e. match block according to certain matching criterior in the given hunting zone in the reference frame; And the relative displacement of match block and current first image block is motion vector, and Fig. 5 has provided the sketch map of a block matching motion estimation procedure; Shown in above-mentioned formula, summation absolute error (SAD, sum of absolute difference), wherein the unit of M is preferably pixel, and the unit of N is preferably pixel; b _T-1Represent current obedient image f (t-1) in first image block, b _T+1Represent reference frame image f (t+1) in second image block, (i j) calculates in this piecemeal in skew (i, j) the matching error sum of all pixels down SAD; Motion estimation process is from the hunting zone to seek to have minimum SAD (i, (i, j), this skew is motion vector in skew j).

The output result in estimation stage is a motion vector field, the motion vector information of each image block in the record current frame image, and the motion estimation process of piece coupling can use figure to explain;

For speed and the accuracy that improves estimation; This paper has introduced the iterative motion algorithm for estimating based on staggered prediction subsequently; This method is based upon on the basis of conventional block matched motion estimation, but further makes full use of the spatial domain consistency on the distribution of motion vectors through the mode of dividing staggered set;

As shown in Figure 6, above-mentioned estimation is first stage in the image processing method of the present invention, the quality of the sports ground that the quality of motion estimation algorithm has directly determined to be exported; In this method of the present invention, two basic hypothesis: 1, the size of two field picture is greater than the size of image block, and 2, the two field picture motion has inertia.Think that the motion vector of current image block has very big similitude with the neighborhood image piece; It is the consistency on the space; Therefore can utilize on the spatial domain contiguous fast motion vector that the motion vector of current image block is predicted; Avoid each possible candidate vector of search in search window, improve search speed; Object of which movement has inertia and has then reflected the consistency of motion vector on time domain, and the motion vector at co-located place can be used for the motion vector sought of initialization current image block in the former frame, accelerates convergence rate.

In this method of the present invention; At first through time domain prediction decision initial ranging vector; Then the image block in the frame to be predicted is divided into two of odd evens according to level and vertical interlaced mode at interval and handles set; Like two parts that indicated with circular and triangle among Fig. 6, two set are handled successively, for another set spatial domain prediction are provided with the mode of iteration.Wherein the processing procedure of each iteration step can be described below; The block collection of selecting triangle to represent; The video blocks that each triangle indicates can adopt the motion vectors of four circular sign pieces in four fields as search set to be selected; Calculate this four pairing SAD of motion vector, select wherein minimum initial value as expanded search.Expanded search can adopt all direction search method among a small circle, and promptly the hunting zone is that radius is 3 zone [3 ,+3] around the initial value, perhaps adopts classical search pattern, like rhombus, hexagon etc., and selects the motion vector of the minimum current the best of renewal of SAD.Finish the current iteration step, change and work as the block collection of pre-treatment set for circular expression, the triangle after utilization was upgraded just now indicates the motion vector of piece and predicts, carries out expanded search.Repeat above-mentioned iterative process repeatedly, accomplish motion estimation process.Because motion estimation process takes into full account the conforming characteristics in distribution of motion vectors spatial domain, has reduced the hunting zone through prediction, has improved search speed, and the motion vector correlation is high between image block, forms level and smooth relatively sports ground easily.All image block can parallel processing in the odd even set simultaneously, is fit to hardware and realizes.

In said method, wherein, above-mentioned steps 32 can specifically comprise:

Step 321; With the said movement locus that motion vector is indicated in said first image block edge; Move to the intermediate value place of said motion vector, obtain corresponding with said first image block, the interpolated image piece between said first image block and said second image block;

Step 322, the interpolated image piece that all first image blocks are corresponding forms the irregular movement vector field of said interpolation frame image.

Certainly, this first image block moves to the computational methods at any point place except that starting point and terminal on the said motion vector along said movement locus that motion vector is indicated, all is identical with the operation method at above-mentioned intermediate value place, repeats no more at this.

As shown in Figure 7; After the above-mentioned estimation stage finishes, obtain the motion vector of each image block, afterwards; The motion vector field that entering calculates through the front and back two field picture gets into motion vector and resamples the stage; Purpose is to set up the pairing motion vector information of interpolation frame image to be calculated, and basic thought is that first image block among the former frame image f (t-1) calculates the gained motion vector according to estimation and moves, and finally can form back one two field picture f (t); If along only a move half-distance (promptly moving to the intermediate value place of motion vector) then can form the current interpolation frame of treating of the indicated track of motion vector; As shown in Figure 7, but, can form the motion vector field that irregularity distributes because the motion vector of different images piece maybe be differing from each other.The purpose that motion vector resamples is to obtain equally distributed sports ground under the regular partitioned organization according to the motion vector calculation that these density distributes.

Like Fig. 8, shown in Figure 9, above-mentioned steps 33 is specially:

Pass through formula: ${\mathrm{\α}}_i=\mathrm{Exp}(-\frac{d_i^2}{{\stackrel{\‾}{d}}^2}),$ ${\mathrm{\β}}_i=\frac{{\mathrm{\α}}_i}{\underset{i=0}{\overset{n}{\mathrm{\Σ}}}{\mathrm{\α}}_i},$ ${\stackrel{\→}{v}}_g=\underset{i=0}{\overset{n}{\mathrm{\Σ}}}{\mathrm{\β}}_i{\stackrel{\→}{v}}_i;$ Said irregular movement vector field is resampled, obtain the motion vector on the summit of the interpolated image piece in the said rule motion vector field;

The motion vector on all interpolated image pieces and summit thereof forms the rule motion vector field of said interpolation frame image;

Wherein, d _iThe distance of the summit V of the interpolated image piece of the central point Vi that representes interpolated image piece in the said irregular movement vector field in the corresponding rule motion vector field of this interpolated image piece, d representes all d _iMean value; N is greater than 0 and less than the positive integer of the maximum quantity of interpolated image piece,

The motion vector of interpolated image piece central point Vi in the expression irregular movement vector field;

The motion vector of representing the summit V of the interpolated image piece in the said rule motion vector field.

Specifically, the problem that is primarily aimed at of the resampling of motion vector is how to reconstruct a regular motion vector field from one through the irregularity motion vector field that forms after the motion excursion.The content of the motion estimation process of previous stage is two known front and back images, however for the interpolation frame image of middle interpolation, picture material still be movable information all be unknown.At this moment, if according to the guided movement locus of motion vector image block is carried out a certain proportion of translation, then because the difference of motion vector to each other can form a non-regular motion vector field, cavity and overlapping phenomenon appear between image block.

Frame of broken lines among Fig. 8 has shown the irregularity property of image block motion vector spatial distribution after motion excursion; Method of the present invention is needed to be the motion vector that calculates each regular grid (image block shown in the solid line) summit V, draws a regular motion vector field;

In the method for the present invention, be that example describes with four image blocks, four image blocks at first will sharing same summit squint by movement locus; As shown in Figure 8, calculate motion back image block central point (V0, V1 subsequently; V2 and V3) with the distance (d0 between the grid vertex V; D1, d2 and d3), and as the average weighted weight of motion vector.

Like the right figure of Fig. 8 and shown in Figure 9,, need to consider the central point V of four adjacent image pieces for the summit V in the rule motion vector field _i, and the distance between the V is d _i

If the motion vector of each image block does

And d _i ²Be V _iAnd between the V distance square, and d is that all are apart from d _iAverage, then introduce above-mentioned weight _iFor the weight sum of four image blocks is 1, further introduce normalized weight beta _iCan guarantee this moment $\underset{i=0}{\overset{n}{\mathrm{\Σ}}}{\mathrm{\β}}_i=1.$

The motion vector on summit then

Can interpolation do ${\stackrel{\→}{v}}_g=\underset{i=0}{\overset{n}{\mathrm{\Σ}}}{\mathrm{\β}}_i{\stackrel{\→}{v}}_i.$

Wherein

represents the motion vector of piecemeal i; Be that example describes with four image blocks around the V of summit among this embodiment of the present invention, but aforementioned calculation formula according to the present invention can know that in the scheme of the present invention, around the V of summit, being not limited to is four image blocks, can be a plurality of arbitrarily; Above-mentioned computational process has guaranteed that the most subapical image block has bigger weight in Interpolation Process, keeps the accuracy of movable information, and also contribution to some extent of other image block has simultaneously also been played certain smoothing effect to sports ground.

After having obtained the rule motion vector field, the step 34 of above-mentioned method shown in Figure 3 can specifically comprise:

Step 341; The motion vector of interpolated image piece in the said rule motion vector field according to the summit of this interpolated image piece moved; Obtain said interpolated image piece corresponding reference image block, said reference image block is the second corresponding image block of said interpolated image piece or first image block;

Step 342 according to said reference image block, is carried out motion compensation to this interpolated image piece in the said rule motion vector field;

Step 343, all the interpolated image pieces in the rule motion vector field are carried out motion compensation after, obtain said interpolation frame image.

Specifically, the concrete implementation procedure of step 342 comprises:

According to formula: f (P ')=f (A ') (1-x) (1-y)+f (B ') x (1-y)+f (C ') (1-x) y+f (D ') xy motion compensation is carried out in said rule motion vector field;

Wherein, Arbitrary pixel in this interpolated image piece corresponding reference image block that interpolated image piece in the P ' expression rule motion vector field obtains after moving according to the motion vector on the summit of this interpolated image piece; F (P ') representes the coordinate of this pixel P ', the coordinate of the summit A ' of this reference image block of f (A '); F (B ') representes the coordinate of the summit B ' of this reference image block, and f (C ') representes the coordinate of the summit C ' of this reference image block; F (D ') representes the coordinate of the summit D ' of this reference image block;

X representes the abscissa of pixel P corresponding with pixel P ' in this interpolated image piece in the rule motion vector field, the ordinate of y remarked pixel P.

Specifically, like Figure 10 and shown in Figure 11,, just can utilize the picture material of the motion compensation technique calculating interpolation frame of distortion of the mesh, i.e. the pairing color of each pixel in case obtain treating the rule motion vector field of interpolation frame image.

Distortion of the mesh in this method is its rule motion vector field in current interpolation frame image, i.e. the regular network of the interpolation frame image shown in the right figure of Figure 10 is utilized the motion vector (resample obtain through motion vector) on each summit in the interpolation frame image lattice structure; Calculate the network of an irregularity in the reference frame image, shown in the left figure of Figure 10, and think per four summit (A in the current interpolation frame image; B, C, the quadrangle piecemeal that D) is surrounded be by pairing irregular quadrilateral in the reference frame image (A '; B '; C ', D ') mapping is promptly based on the motion compensation technique of distortion of the mesh.

The distortion of the mesh technology is not simple linear translation, therefore can eliminate the overlapping and cavitation that traditional block-based motion compensation occurs.The one-to-one relationship between pixel in the pixel and reference frame image is promptly set up in the interpolation frame image in the wherein establishment of mapping relations, adopts the method for bilinear interpolation in the embodiments of the invention.(C is in the image block that D) is determined for A, B by four summits at any; Suppose A, B, C, D is corresponding to the A ' in the reference frame; B ', C ' and D ', the coordinate in reference frame image are respectively f (A '), f (B '); F (C ') and f (D '), the pixel P ' in the corresponding reference frame image of each the pixel P in the interpolated image piece, then the coordinate f of P ' (P ') can obtain through four summit interpolation;

Shown in figure 11: at first, four summit A, B, C, D establishes a normalized coordinate system; Corresponding local coordinate is respectively (0,0), (1,0); (1,1) and (0,1), and the reference frame image coordinate is respectively f (A '); F (B '), f (C ') and f (D '), then wherein the local coordinate of any point P be (x, y) pairing reference frame coordinate f (P ') is:

f(P′)＝f(A′)(1-x)(1-y)+f(B′)x(1-y)+f(C′)(1-x)y+f(D′)xy

0≤x wherein, y≤1 in view of the above, sets up that each pixel because reference frame image is known, and then constructs the interpolation frame image with the mapping relations of pixel in the reference frame image in the interpolation frame image.

In sum; The present invention proposes a kind of on the frame rate of distortion of the mesh transfer algorithm; Be used for image processing process, be different from algorithm in the past, this algorithm combination the motion compensation of block-based estimation and distortion Network Based; Can make full use of the ripe advantage of block matching motion algorithm for estimating fast on the one hand; Bring into play the characteristics of the seamless distortion of grid (mesh) on the other hand, eliminated the overlapping and cavitation that traditional block-based motion compensation occurs, and then improved whole system operation efficient and output quality.

Method for estimating in the said method is divided into two independently set according to staggered layout; Single set reduces amount of calculation owing to the video blocks number reduces by half on the one hand; Spatially consistency on the other hand has benefited from moving; Can form predicted vector more accurately through the motion vector that has calculated in the set of field, reduce computation complexity.

This method can produce more level and smooth real sports ground, and all video image blocks can parallel processing in the set, is convenient to hardware and realizes raising the efficiency.

And this method can be further estimates that with other ripe block matching motions fast methods combine, and like level search, diamond search etc., further raises the efficiency.

The motion compensation of distortion of the mesh then can solve the overlapping and cavitation that the conventional block method is occurred, and eliminates blocking effect, improves the picture quality of interpolation frame.And in this method, the interpolation frame image is kept regular network, and reference frame image is adopted the deformation grid, has simplified the calculating of interpolation frame.

Shown in figure 12, embodiments of the invention also provide a kind of image processing apparatus 120, comprising:

First processing module 121 is used for first two field picture and second two field picture of two continuous frames image are carried out estimation, obtains the motion vector from said first two field picture to said second two field picture;

Second processing module 122; Be used for said first two field picture obtaining the irregular movement vector field of the interpolation frame image between said first two field picture and said second two field picture along the move part distance of said motion vector of the indicated movement locus of said motion vector;

The 3rd processing module 123 is used for said irregular movement vector field is resampled, and obtains the rule motion vector field of said interpolation frame image;

Manages module 124 everywhere, is used for motion compensation is carried out in said rule motion vector field, obtains said interpolation frame image.

Wherein, said first processing module 121 comprises:

Divide module, be used for said first two field picture is divided at least one first image block, said second two field picture is divided at least one second image block;

Processing sub is used for said first image block and said second image block are carried out estimation, obtains the piece motion vector from said first image block to said second image block.

Specifically; Processing sub specifically is used for: according to the size information and a displacement to be selected of said first image block and said second image block; Said first image block and said second image block are carried out estimation, obtain piece motion vector from said first image block to said second image block;

Wherein, said displacement to be selected is: in the hunting zone that includes at least one second image block, and one second image block of said first image block coupling and the relative displacement between said first image block.

Wherein, one second image block with said first image block coupling is meant: in said hunting zone, with the second maximum image block of the said first image block similarity.

Wherein, second image block maximum with the said first image block similarity is meant: in said hunting zone, SAD (i, j) second image block of minimum, wherein,

$\mathrm{SAD}(i,j)=\underset{m=0}{\overset{M}{\mathrm{\Σ}}}\underset{n=0}{\overset{N}{\mathrm{\Σ}}}|b_{t-1}(m,n)-b_{t+1}(m+i,n+j)|;$

Wherein, b _T-1Represent first image block, b _T+1Represent second image block; I, j represent the displacement that certain is to be selected, and M representes the length of first image block and second image block; N representes the width of first image block and second image block; SAD (i, j) expression is calculated in first image block and second image block in skew (i, j) the matching error sum of following all pixels;

In addition; Said second processing module 122 specifically is used for; With the said movement locus that motion vector is indicated in said first image block edge; Move to the intermediate value place of said motion vector, obtain corresponding with said first image block, the interpolated image piece between said first image block and said second image block; The interpolated image piece that all first image blocks are corresponding forms the irregular movement vector field of said interpolation frame image.

In addition, said the 3rd processing module 123 specifically is used for,

Pass through formula: ${\mathrm{\α}}_i=\mathrm{Exp}(-\frac{d_i^2}{{\stackrel{\‾}{d}}^2}),$ ${\mathrm{\β}}_i=\frac{{\mathrm{\α}}_i}{\underset{i=0}{\overset{n}{\mathrm{\Σ}}}{\mathrm{\α}}_i},$ ${\stackrel{\→}{v}}_g=\underset{i=0}{\overset{n}{\mathrm{\Σ}}}{\mathrm{\β}}_i{\stackrel{\→}{v}}_i,$ Said irregular movement vector field is resampled, obtain the motion vector on the summit of the interpolated image piece in the said rule motion vector field;

The motion vector on all interpolated image pieces and summit thereof forms the rule motion vector field of said interpolation frame image;

Wherein, d _iThe distance of the summit V of the interpolated image piece of the central point Vi that representes interpolated image piece in the said irregular movement vector field in the corresponding rule motion vector field of this interpolated image piece, d representes all d _iMean value; N is greater than 0 and less than the positive integer of the maximum quantity of interpolated image piece,

The motion vector of interpolated image piece central point Vi in the expression irregular movement vector field;

The motion vector of representing the summit V of the interpolated image piece in the said rule motion vector field.

In addition, above-mentioned said manage module 124 everywhere and comprise:

Mapping block; Be used for the interpolated image piece of the said rule motion vector field motion vector according to the summit of this interpolated image piece is moved; Obtain said interpolated image piece corresponding reference image block, said reference image block is the second corresponding image block of said interpolated image piece or first image block;

The motion compensation process submodule is used for according to said reference image block, and this interpolated image piece in the said rule motion vector field is carried out motion compensation; After all interpolated image pieces in the rule motion vector field are carried out motion compensation, obtain said interpolation frame image;

Specifically: the motion compensation process submodule specifically is used for:

According to formula: f (P ')=f (A ') (1-x) (1-y)+f (B ') x (1-y)+f (C ') (1-x) y+f (D ') xy motion compensation is carried out in said rule motion vector field;

Wherein, Arbitrary pixel in this interpolated image piece corresponding reference image block that interpolated image piece in the P ' expression rule motion vector field obtains after moving according to the motion vector on the summit of this interpolated image piece; F (P ') representes the coordinate of this pixel P ', the coordinate of the summit A ' of this reference image block of f (A '); F (B ') representes the coordinate of the summit B ' of this reference image block, and f (C ') representes the coordinate of the summit C ' of this reference image block; F (D ') representes the coordinate of the summit D ' of this reference image block;

X representes the abscissa of pixel P corresponding with pixel P ' in this interpolated image piece in the rule motion vector field, the ordinate of y remarked pixel P.

Need to prove: the concrete implementation of all characteristics among above-mentioned Fig. 3-Figure 11 all is applicable to the corresponding module among this device embodiment, also can reach identical technique effect, repeats no more at this.

Said method of the present invention and device go for especially to video monitoring service, through said method, in the incoherent process of video, inserting some interpolation frame images in any business that needs the interpolation frame image, make image smoothness more; Under the bad situation of network quality, when wanting to transmit continuous video, all can use said method of the present invention and device.

The above is a preferred implementation of the present invention; Should be pointed out that for those skilled in the art, under the prerequisite that does not break away from principle according to the invention; Can also make some improvement and retouching, these improvement and retouching also should be regarded as protection scope of the present invention.

Claims (13)

1. an image processing method is characterized in that, comprising:

First two field picture and second two field picture are carried out estimation, obtain motion vector from said first two field picture to said second two field picture;

Said first two field picture along the move part distance of said motion vector of the indicated movement locus of said motion vector, is obtained the irregular movement vector field of the interpolation frame image between said first two field picture and said second two field picture;

Said irregular movement vector field is resampled, obtain the rule motion vector field of said interpolation frame image;

Motion compensation is carried out in said rule motion vector field, obtain said interpolation frame image;

Wherein, said irregular movement vector field is resampled, the step that obtains the rule motion vector field of said interpolation frame image is specially:

Pass through formula: ${\mathrm{\α}}_i=\mathrm{Exp}(-\frac{d_i^2}{{\stackrel{\‾}{d}}^2}),$ ${\mathrm{\β}}_i=\frac{{\mathrm{\α}}_i}{\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{\mathrm{\α}}_i},$ ${\stackrel{\→}{v}}_g=\underset{i=0}{\overset{n}{\mathrm{\Σ}}}{\mathrm{\β}}_i{\stackrel{\→}{v}}_i,$ Said irregular movement vector field is resampled, obtain the motion vector on the summit of the interpolated image piece in the said rule motion vector field;

All the interpolated image pieces in the said rule motion vector field and the motion vector on summit thereof form the rule motion vector field of said interpolation frame image;

Wherein, d _iThe distance of the summit V of the interpolated image piece of the central point Vi that representes interpolated image piece in the said irregular movement vector field in the corresponding rule motion vector field of this interpolated image piece,

Represent all d _iMean value; N is greater than 0 and less than the positive integer of the maximum quantity of interpolated image piece,

The motion vector of interpolated image piece central point Vi in the expression irregular movement vector field;

The motion vector of representing the summit V of the interpolated image piece in the said rule motion vector field.

2. image processing method according to claim 1 is characterized in that, first two field picture and second two field picture are carried out estimation, and the motion vector step that obtains from said first two field picture to said second two field picture comprises:

Said first two field picture is divided at least one first image block, wherein, one second image block in corresponding said second two field picture of one first image block;

Said first image block and said second image block are carried out estimation, obtain piece motion vector from said first image block to said second image block.

3. image processing method according to claim 2 is characterized in that, said first image block and said second image block are carried out estimation, and the step that obtains the piece motion vector from said first image block to said second image block comprises:

A size information and a displacement to be selected according to said first image block and said second image block carry out estimation to said first image block and said second image block, obtain the piece motion vector from said first image block to said second image block.

4. image processing method according to claim 3; It is characterized in that; Said displacement to be selected is: in the hunting zone that includes at least one second image block, and one second image block of said first image block coupling and the relative displacement between said first image block.

5. image processing method according to claim 4 is characterized in that, second image block that matees with said first image block is meant: in said hunting zone, with the second maximum image block of the said first image block similarity.

6. image processing method according to claim 5 is characterized in that, second image block maximum with the said first image block similarity is meant: in said hunting zone, SAD (i, j) second image block of minimum, wherein,

$\mathrm{SAD}(i,j)=\underset{m=0}{\overset{M}{\mathrm{\Σ}}}\underset{n=0}{\overset{N}{\mathrm{\Σ}}}|b_{t-1}(m,n)-b_{t+1}(m+i,n+j)|;$

Wherein, b _T-1Represent first image block, b _T+1Represent second image block, i, j represent the displacement that certain is to be selected, M presentation video block length, and the width of N presentation video piece, SAD (i, j) calculate in first image block and second image block in skew (i, j) the matching error sum of following all pixels by expression.

7. image processing method according to claim 2; It is characterized in that; Along the move part distance of said motion vector of the indicated movement locus of said motion vector, the step that obtains the irregular movement vector field of the interpolation frame image between said first two field picture and said second two field picture comprises with said first two field picture:

Said first image block along the said movement locus that motion vector is indicated, is moved to the intermediate value place of said motion vector, obtain corresponding with said first image block, the interpolated image piece between said first image block and said second image block;

The interpolated image piece that all first image blocks are corresponding forms the irregular movement vector field of said interpolation frame image.

8. image processing method according to claim 7 is characterized in that, motion compensation is carried out in said rule motion vector field, and the step that obtains said interpolation frame image comprises:

The motion vector of interpolated image piece in the said rule motion vector field according to the summit of this interpolated image piece moved; Obtain said interpolated image piece corresponding reference image block, said reference image block is the second corresponding image block of said interpolated image piece or first image block;

According to said reference image block, this interpolated image piece in the said rule motion vector field is carried out motion compensation;

After all interpolated image pieces in the rule motion vector field are carried out motion compensation, obtain said interpolation frame image.

9. image processing method according to claim 8 is characterized in that, according to said reference image block, the step that this interpolated image piece in the said rule motion vector field is carried out motion compensation is specially:

According to formula: f (P ')=f (A ') (1-x) (1-y)+f (B ') x (1-y)+f (C ') (1-x) y+f (D ') xy this interpolated image piece in the said rule motion vector field is carried out motion compensation;

Wherein, Arbitrary pixel in this interpolated image piece corresponding reference image block that interpolated image piece in the P ' expression rule motion vector field obtains after moving according to the motion vector on the summit of this interpolated image piece; F (P ') representes the coordinate of this pixel P ', and f (A ') representes the coordinate of the summit A ' of this reference image block; F (B ') representes the coordinate of the summit B ' of this reference image block, and f (C ') representes the coordinate of the summit C ' of this reference image block; F (D ') representes the coordinate of the summit D ' of this reference image block;

X representes the abscissa of pixel P corresponding with pixel P ' in this interpolated image piece in the rule motion vector field, the ordinate of y remarked pixel P.

10. an image processing apparatus is characterized in that, comprising:

First processing module is used for first two field picture and second two field picture of two continuous frames image are carried out estimation, obtains the motion vector from said first two field picture to said second two field picture;

Second processing module; Be used for said first two field picture obtaining the irregular movement vector field of the interpolation frame image between said first two field picture and said second two field picture along the move part distance of said motion vector of the indicated movement locus of said motion vector;

The 3rd processing module is used for said irregular movement vector field is resampled, and obtains the rule motion vector field of said interpolation frame image;

Manages module everywhere, is used for motion compensation is carried out in said rule motion vector field, obtains said interpolation frame image;

Wherein, said the 3rd processing module specifically is used for through formula:

${\mathrm{\α}}_i=\mathrm{Exp}(-\frac{d_i^2}{{\stackrel{\‾}{d}}^2}),$ ${\mathrm{\β}}_i=\frac{{\mathrm{\α}}_i}{\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{\mathrm{\α}}_i},$ ${\stackrel{\→}{v}}_g=\underset{i=0}{\overset{n}{\mathrm{\Σ}}}{\mathrm{\β}}_i{\stackrel{\→}{v}}_i,$ To said irregular movement vector field

Resample, obtain the motion vector on the summit of the interpolated image piece in the said rule motion vector field;

All the interpolated image pieces in the said rule motion vector field and the motion vector on summit thereof form the rule motion vector field of said interpolation frame image;

Wherein, d _iThe distance of the summit V of the interpolated image piece of the central point Vi that representes interpolated image piece in the said irregular movement vector field in the corresponding rule motion vector field of this interpolated image piece,

Represent all d _iMean value; N is greater than 0 and less than the positive integer of the maximum quantity of interpolated image piece,

The motion vector of interpolated image piece central point Vi in the expression irregular movement vector field;

The motion vector of representing the summit V of the interpolated image piece in the said rule motion vector field.

11. image processing apparatus according to claim 10 is characterized in that, said first processing module comprises:

Divide module, be used for said first two field picture is divided at least one first image block, said second two field picture is divided at least one second image block;

Processing sub is used for said first image block and said second image block are carried out estimation, obtains the piece motion vector from said first image block to said second image block.

12. image processing apparatus according to claim 11; It is characterized in that; Said second processing module specifically is used for, and said first image block along the said movement locus that motion vector is indicated, is moved to the intermediate value place of said motion vector; Obtain corresponding with said first image block, the interpolated image piece between said first image block and said second image block; The interpolated image piece that all first image blocks are corresponding forms the irregular movement vector field of said interpolation frame image.

13. image processing apparatus according to claim 12 is characterized in that, said manages module everywhere comprises:

Mapping block; Be used for the interpolated image piece of the said rule motion vector field motion vector according to the summit of this interpolated image piece is moved; Obtain said interpolated image piece corresponding reference image block, said reference image block is the second corresponding image block of said interpolated image piece or first image block;

The motion compensation process submodule is used for according to said reference image block, and this interpolated image piece in the said rule motion vector field is carried out motion compensation; After all interpolated image pieces in the rule motion vector field are carried out motion compensation, obtain said interpolation frame image.

Images