CN101854465A - Image processing method and device based on optical flow algorithm - Google Patents

Image processing method and device based on optical flow algorithm Download PDF

Info

Publication number
CN101854465A
CN101854465A CN 201010107256 CN201010107256A CN101854465A CN 101854465 A CN101854465 A CN 101854465A CN 201010107256 CN201010107256 CN 201010107256 CN 201010107256 A CN201010107256 A CN 201010107256A CN 101854465 A CN101854465 A CN 101854465A
Authority
CN
China
Prior art keywords
pixel
frame image
parameter
global motion
position coordinates
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN 201010107256
Other languages
Chinese (zh)
Other versions
CN101854465B (en
Inventor
马程
王超
蔡巍伟
贾永华
胡扬忠
邬伟琪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Hikvision Digital Technology Co Ltd
Original Assignee
Hangzhou Hikvision Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Hikvision Software Co Ltd filed Critical Hangzhou Hikvision Software Co Ltd
Priority to CN 201010107256 priority Critical patent/CN101854465B/en
Publication of CN101854465A publication Critical patent/CN101854465A/en
Application granted granted Critical
Publication of CN101854465B publication Critical patent/CN101854465B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Image Analysis (AREA)
  • Studio Devices (AREA)

Abstract

The embodiment of the invention discloses image processing method and device based on an optical flow algorithm. The method comprises the following steps of: acquiring a current frame image in a video sequence to be processed, and then acquiring the position of a group of pixel points of the current frame image in the next frame image according to the optical flow algorithm; sampling the group of pixel points according to a global parameter model of two adjacent frame images established in advance, and screening out the pixel point of which the displacement distance is within the preset range; acquiring a global motion parameter from the current frame image to the next frame image with a least square method according to the position coordinates of the screened pixel point in the current frame image and the next frame image; and compensating the next frame image according to the smoothing result on the global motion parameter. In the invention, frame images are processed in a way of combining the optical flow algorithm with global motion model sampling, therefore, the calculation precision of the global motion parameter is ensured, the calculated amount is reduced, and meanwhile, the requirement for real-time stable correction of video is met.

Description

Image processing method and device based on optical flow algorithm
Technical field
The application relates to the Computer Image Processing field, relates in particular to a kind of image processing method and device based on optical flow algorithm.
Background technology
Along with the popularization and application of digital camera, people wish can be steady and audible by the video image of shot by camera, is convenient to watch and analyze, and wishes that also captured image can compress better simultaneously, is convenient to save memory space and reduces data transmission rate.
The video stabilization technology is an important video enhancement techniques of digital image video process field, can stablize and compresses processing video image.When using the video stabilization technology, at first to calculate the global motion parameter between adjacent two two field pictures in the video image, extract the true global motion parameter of video camera then, use suitable video transformation compensating technique picture at last, make video smooth and stablize.The global motion parameter of calculating between adjacent two two field pictures is the core of whole video stabilization technique.Ask for the global motion parameter in the prior art and adopt BMA usually, video image is divided into a lot of macro blocks, then that current macro is corresponding with former frame image macro block mates, obtain the local motion vector of current macro, derive the global motion parameter according to local motion vector at last.
The inventor finds in to the research of prior art and practice process, carries out video stabilization when handling by BMA, need a plurality of macro blocks of dividing be calculated, so the amount of calculation of this algorithm is bigger, is difficult to be applied to real-time video stabilization and handles; And,, therefore limited the precision of BMA because the shake between video image has rotation usually and amplifies the character of dwindling.
Summary of the invention
The purpose of the embodiment of the present application provides a kind of image processing method and device based on optical flow algorithm, existing carry out video stabilization by BMA and handles and cause the big and not high problem of precision of amount of calculation to solve.
For solving the problems of the technologies described above, the embodiment of the present application provides following technical scheme:
A kind of image processing method based on optical flow algorithm comprises:
After obtaining the current frame image in the pending video sequence, obtain one group of pixel position in the next frame image in the described current frame image according to optical flow algorithm;
Global parameter model according to adjacent two two field pictures of setting up is in advance sampled to described one group of pixel, filters out the pixel of displacement in preset range;
According to the position coordinates of the described pixel that filters out in current frame image and next frame image, obtain global motion parameter from described current frame image to the next frame image by least square method;
According to described global motion parameter is carried out level and smooth result, described next frame image is compensated.
Describedly obtain according to optical flow algorithm that one group of pixel position in the next frame image comprises in the described current frame image:
Described current frame image is divided into several macro blocks, and obtains a pixel in each macro block;
Follow the tracks of and calculate the local motion vector of described each pixel in the next frame image by optical flow algorithm;
Calculate the position coordinates of described pixel in the next frame image according to the position coordinates and the local motion vector of described pixel in the next frame image of each pixel in current frame image.
Described global parameter model according to adjacent two two field pictures of setting up is in advance sampled to described one group of pixel and is filtered out the pixel of displacement in preset range and comprise:
The position coordinates of two pixels in the described current frame image of picked at random and the described global parameter model of the position coordinates substitution of described two pixels in the next frame image are obtained the global motion parameter, and described global motion parameter comprises image stretch parameter, image rotation parameter, image transverse translation parameter and image longitudinal translation parameter;
Global motion equation with the position coordinates input of the residual pixel point except that two pixels of described picked at random in one group of pixel in the described current frame image is determined according to described global motion parameter obtains the position coordinates of described residual pixel point in the next frame image;
Add up the pixel of distance in described preset range between described position coordinates that obtains according to the global motion equation and the described position coordinates that obtains according to optical flow algorithm.
Described method also comprises: set in advance described one group of number of times that pixel is sampled;
Described filtering out before the pixel of displacement in preset range also comprises:
According to described global parameter model described one group of pixel is carried out the sampling consistent with described frequency in sampling;
The number of the pixel of displacement in preset range that record sampling is each time obtained;
Pixel with the number of described pixel when maximum is as the described pixel that filters out.
Described described global motion parameter is smoothly comprised:
The curve movement of determining the global motion parameter according to the global motion parameter and the frame number of all two field pictures in the described video sequence;
By gaussian filtering the curve movement of described global motion parameter is carried out motion smoothing;
Asking for each global motion parameter after level and smooth on the curve movement behind the described motion smoothing according to frame number.
The described next frame image is compensated comprises:
Described global motion parameter after level and smooth and the described global motion parameter of obtaining by least square method asked be compensated parameter after differing from;
By described compensating parameter described next frame image is compensated.
A kind of image processing apparatus based on optical flow algorithm comprises:
The two field picture acquiring unit is used for obtaining the current frame image of pending video sequence;
The optical flow algorithm unit is used for obtaining one group of pixel position in the next frame image of described current frame image according to optical flow algorithm;
Sampling screening unit is used for according to the global parameter model of adjacent two two field pictures of setting up in advance described one group of pixel being sampled, and filters out the pixel of displacement in preset range;
The global parameter acquiring unit is used for according to the position coordinates of the described pixel that filters out at current frame image and next frame image, obtains global motion parameter from described current frame image to the next frame image by least square method;
The parameter smoothing unit is used for described global motion parameter is carried out smoothly;
Image compensation unit is used for according to the level and smooth result of described parameter smoothing unit described next frame image being compensated.
Described optical flow algorithm unit comprises:
The macroblock partitions subelement is used for described current frame image is divided into several macro blocks, and obtains a pixel in each macro block;
The Local Vector computation subunit is used for following the tracks of and calculate the local motion vector of described each pixel at the next frame image by optical flow algorithm;
The position coordinates computation subunit is used for calculating the position coordinates of described pixel in the next frame image according to each pixel at the position coordinates and the local motion vector of described pixel in the next frame image of current frame image.
Described sampling screening unit comprises:
Global parameter is obtained subelement, be used for the position coordinates and the described global parameter model of the position coordinates substitution of described two pixels in the next frame image of two pixels of the described current frame image of picked at random are obtained the global motion parameter, described global motion parameter comprises image stretch parameter, image rotation parameter, image transverse translation parameter and image longitudinal translation parameter;
Position coordinates obtains subelement, be used for the global motion equation that the position coordinates input of the residual pixel point except that two pixels of described picked at random in one group of pixel of described current frame image is definite according to described global motion parameter, obtain the position coordinates of described residual pixel point in the next frame image;
Pixel statistics subelement is used to add up the pixel of distance in described preset range between described position coordinates that obtains according to the global motion equation and the described position coordinates that obtains according to optical flow algorithm.
Also comprise:
Default unit is used to set in advance to described one group of number of times that pixel is sampled;
Described sampling screening unit also comprises:
The frequency in sampling control sub unit is used to control described global parameter acquiring unit, position coordinates acquiring unit and pixel statistic unit and carries out the sampling consistent with described frequency in sampling in proper order;
Sampling number record subelement is used to write down the number of the pixel of displacement in preset range that sampling each time obtained;
Sampling results is confirmed subelement, is used for the number of described pixel pixel when maximum as the described pixel that filters out.
Described parameter smoothing unit comprises:
Curve movement is determined subelement, is used for the curve movement of determining the global motion parameter according to the global motion parameter and the frame number of described all two field pictures of video sequence;
The motion smoothing subelement is used for by gaussian filtering the curve movement of described global motion parameter being carried out motion smoothing;
Smoothing parameter is asked for subelement, is used for asking for each global motion parameter after level and smooth on the curve movement behind the described motion smoothing according to frame number.
Described image compensation unit comprises:
Compensating parameter obtains subelement, is used for described global motion parameter after level and smooth and the described global motion parameter of obtaining by least square method asked being compensated parameter after differing from;
Two field picture compensation subelement is used for by described compensating parameter described next frame image being compensated.
As seen, after obtaining the current frame image in the pending video sequence in the embodiment of the present application, obtain one group of pixel position in the next frame image in the current frame image according to optical flow algorithm, global parameter model according to adjacent two two field pictures of setting up is in advance sampled to one group of pixel, filter out the pixel of displacement in preset range, according to the position coordinates of the pixel that filters out in current frame image and next frame image, obtain global motion parameter by least square method from current frame image to the next frame image, according to the global motion parameter is carried out level and smooth result, the next frame image is compensated.In the embodiment of the present application owing to two field picture is handled by optical flow algorithm and the global motion model mode that combines of sampling, therefore when guaranteeing global motion calculation of parameter precision, reduced amount of calculation, improve antijamming capability, satisfied real-time video stabilization alignment requirements.
Description of drawings
In order to be illustrated more clearly in the embodiment of the present application or technical scheme of the prior art, to do to introduce simply to the accompanying drawing of required use in embodiment or the description of the Prior Art below, apparently, the accompanying drawing that describes below only is some embodiment that put down in writing among the application, for those of ordinary skills, under the prerequisite of not paying creative work, can also obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is the first embodiment flow chart of the application based on the image processing method of optical flow algorithm;
Fig. 2 is the second embodiment flow chart of the application based on the image processing method of optical flow algorithm;
Fig. 3 is the first embodiment block diagram of the application based on the image processing apparatus of optical flow algorithm;
Fig. 4 A is the second embodiment block diagram of the application based on the image processing apparatus of optical flow algorithm;
Fig. 4 B is the embodiment block diagram of optical flow algorithm unit among the application's device second embodiment;
Fig. 4 C is the embodiment block diagram of sampling screening unit among the application's device second embodiment;
Fig. 4 D is the embodiment block diagram of parameter smoothing unit among the application's device second embodiment;
Fig. 4 E is the embodiment block diagram of image compensation unit among the application's device second embodiment.
Embodiment
The embodiment of the present application provides a kind of image processing method and device based on optical flow algorithm.
In order to make those skilled in the art person understand technical scheme in the embodiment of the present application better, and the above-mentioned purpose of the embodiment of the present application, feature and advantage can be become apparent more, below in conjunction with accompanying drawing technical scheme in the embodiment of the present application is described in further detail.
Referring to Fig. 1, be the first embodiment flow chart of the application based on the image processing method of optical flow algorithm:
Step 101: obtain the current frame image in the pending video sequence.
Step 102: obtain one group of pixel position in the next frame image in the current frame image according to optical flow algorithm.
Concrete, current frame image is divided into several macro blocks, and obtain a pixel in each macro block, follow the tracks of and calculate the local motion vector of each pixel in the next frame image by optical flow algorithm, calculate the position coordinates of this pixel in the next frame image according to the position coordinates and the local motion vector of this pixel in the next frame image of each pixel in current frame image.
Step 103: the global parameter model according to adjacent two two field pictures of setting up is in advance sampled to described one group of pixel, filters out the pixel of displacement in preset range.
Concrete, the global parameter model that the position coordinates of two pixels in the current frame image of picked at random and two the position coordinates substitutions of pixel in the next frame image are set up in advance obtains to comprise the image parameter of stretching, the image rotation parameter, image transverse translation parameter and image longitudinal translation parameter are in interior global motion parameter, the global motion equation that the position coordinates input of the residual pixel point except that two pixels of picked at random in one group of pixel in the current frame image is definite according to the global motion parameter, obtain the position coordinates of residual pixel point in the next frame image, the pixel of distance in preset range between position coordinates that statistics obtains according to the global motion equation and the position coordinates that obtains according to optical flow algorithm.
Step 104:, obtain global motion parameter from current frame image to the next frame image by least square method according to the position coordinates of the pixel that filters out in current frame image and next frame image.
Step 105: according to the global motion parameter is carried out level and smooth result, the next frame image is compensated, finish current flow process.
Concrete, the curve movement of determining the global motion parameter according to the global motion parameter and the frame number of all two field pictures in the video sequence, by gaussian filtering the curve movement of global motion parameter is carried out motion smoothing, asking for each global motion parameter after level and smooth on the curve movement behind the motion smoothing according to frame number, be compensated parameter after global motion parameter after level and smooth and the global motion parameter of obtaining by least square method asked difference, the next frame image compensated by this compensating parameter.
Referring to Fig. 2, be the second embodiment flow process of the application based on the image processing method of optical flow algorithm, this embodiment shows in detail the process of image being handled by optical flow algorithm:
Step 201: obtain the current frame image in the pending video sequence.
The embodiment of the present application is mainly used in to be handled the video sequence that produces shake, before processing, at first need to obtain pending video sequence, each video sequence is made up of some two field pictures, is exactly results set after each two field picture is handled respectively to the processing of video sequence.
Step 202: current frame image is divided into several macro blocks, and obtains a pixel in each macro block.
With the example that is treated to of a current frame image, this current frame image can be divided into m*n macro block, from each macro block, get central pixel point, get m*n pixel altogether.
Step 203: follow the tracks of and calculate the local motion vector of each pixel in the next frame image by optical flow algorithm.
Optical flow algorithm in the embodiment of the present application mainly refers to Lucas-Kanade optical flow algorithm (being called for short the L-K optical flow algorithm), this algorithm is a kind of light stream algorithm for estimating of two frame differences, its principle is owing to the information change amount of same pixel in adjacent two two field pictures is very little, therefore can think their conservations, conservation according to pixel obtains the two field picture constraint equation, suppose I (x, y) be that position coordinates is (x in the current frame image, y) pixel, the constraint equation to this pixel I in then adjacent two two field pictures is:
I(x，y，t)＝I(x+σx，y+σy，t+σt)
Therefore can use above-mentioned constraint equation respectively to m*n the pixel that is obtained, use Taylor series expansion just can obtain the local motion vector of each pixel I, promptly can obtain m*n local motion vector altogether this equation.
Step 204: calculate the position coordinates of this pixel in the next frame image according to the position coordinates and the local motion vector of this pixel in the next frame image of each pixel in current frame image.
In step 203, calculate m*n the m*n of pixel in next frame image local motion vector in the current frame image, just can obtain the position coordinates of each pixel in the next frame image with the position coordinates of each pixel in current frame image in m*n the pixel with corresponding local motion vector addition respectively.
Step 205: the position coordinates and the position coordinates substitution global parameter model of two pixels in the next frame image of two pixels in the current frame image of picked at random are obtained the global motion parameter.
Because translation, rotation and flexible may appear in two field picture in motion process, therefore the global parameter model of setting up in advance is as follows:
x i + 1 y i + 1 = s cos α - sin α sin α cos α x i y i + d x d y
In the above-mentioned model, s is the stretching motion parameter of two field picture, and α is the parameter that rotatablely moves of two field picture, d xBe horizontal (being directions X in the coordinate system) the translational motion parameter of two field picture, d yBe vertical (being Y direction in the coordinate system) translation parameters of two field picture; (x i, y i) be the position coordinates of pixel in the i two field picture, (x I+1, y I+1) be the position coordinates of pixel in the i+1 two field picture.
Because four global motion parameters in the above-mentioned global parameter model are unknown quantity to be asked, can try to achieve by four equations for four unknown quantitys, therefore adopt RANSAC (random sampling consistency) method from the m*n of aforementioned current frame image and next frame image to extracting any two pairs of pixels the pixel, the above-mentioned global parameter model of its coordinate figure substitution can be obtained four equations, can be according to these four equations in the hope of four kinematic parameter s, α, d xAnd d y
Step 206: the global motion equation with the position coordinates input of the residual pixel point except that two pixels of picked at random in one group of pixel in the current frame image is determined according to the global motion parameter obtains the position coordinates of residual pixel point in the next frame image.
Four kinematic parameter substitution global parameter models of trying to achieve in the step 205 are obtained the global motion equation, the position coordinates of the residual pixel point except that two pixels of picked at random in m*n the pixel of current frame image is updated in the global motion equation, can be in the hope of the position coordinates of each pixel in the next frame image.
Step 207: the pixel of distance in preset range between position coordinates that statistics obtains according to the global motion equation and the described position coordinates that obtains according to optical flow algorithm.
Suppose that preset range is 2 distances between the pixel, then for each position coordinates of pixel in the next frame image that obtains according to the global motion Equation for Calculating, difference between the position coordinates of this pixel in the next frame image that obtains with it and according to optical flow algorithm (i.e. the distance of determining according to two position coordinateses) with should default distance compare the pixel of statistics in this predeterminable range scope.
Step 208: the number that writes down this pixel of displacement in preset range of being obtained of sampling.
After the pixel of displacement in preset range that obtains in the sampling process each time added up, write down the number of all qualified pixels.
Step 209: judge whether to have carried out the sampling consistent, if then execution in step 210 with preset times; Otherwise, return step 205.
Step 210: according to the number of pixel of each sampling of record, the pixel with the number of pixel when maximum is as the pixel that filters out.
Wherein, frequency in sampling K can rule of thumb be worth by following formula and draws:
K = log ( 1 - z ) log ( 1 - w l )
Wherein, z is the correct probability of arithmetic result, w is the ratio that the pixel that filters out accounts in whole pixel, thereby can think w 〉=0.5, l gets l=2 for the number of each point of getting in set, z=0.99, can obtain K=17, promptly repeat 17 random samplings, then the correct probability of outcome of the pixel that is filtered out is greater than 0.99.
Step 211:, obtain global motion parameter from current frame image to the next frame image by least square method according to the position coordinates of the pixel that filters out in current frame image and next frame image.
Step 212: the curve movement of determining the global motion parameter according to the global motion parameter and the frame number of all two field pictures in the video sequence.
Suppose total N two field picture in the video sequence, then can organize the global motion parameter in the hope of N-1, for each global motion parameter, with the global motion parameter d for all two field pictures in the video sequence xBe example, then can be in coordinate system according to the d under every two field picture xValue and corresponding frame number are determined d in this video sequence xCurve movement; In like manner, can obtain the curve movement of other three global motion parameters.
Step 213: by gaussian filtering the curve movement of global motion parameter is carried out motion smoothing, and asking for each global motion parameter after level and smooth on the curve movement behind the motion smoothing according to frame number.
For by the smooth motion curve behind the gaussian filtering, can obtain the global motion parameter of this frame correspondence according to each frame in the video sequence.
Step 214: the global motion parameter after inciting somebody to action smoothly is compensated parameter after asking difference with the global motion parameter of obtaining by least square method.
Step 215: by compensating parameter the next frame image is compensated, finish current flow process.
Global motion parameter after supposing smoothly is T iThe global motion parameter of obtaining by least square method is A i, the compensating parameter Δ i=T i-A i, with this compensating parameter corresponding two field picture is compensated, can obtain anti-shake two field picture.
In conjunction with above-mentioned the application's image processing method embodiment, the shake video sequence that is 352*288 with one section resolution is the detailed process that example is described the application's image processing below.
The i two field picture is divided into 11*9 macro block, gets the central pixel point of each macro block, thereby obtain 99 pixels, calculate the side-play amount of these pixels in the i+1 frame, can obtain 99 local motion vectors with the L-K optical flow method;
The global motion model of being set up that comprises four global motion parameters is as follows:
x i + 1 y i + 1 = s cos α - sin α sin α cos α x i y i + d x d y
2 pairs of pixels in picked at random i two field picture and the i+1 two field picture, generation as above-mentioned global motion model are tried to achieve 4 global motion parameters, the coordinate that the above-mentioned global motion model of four global motion parameters substitution is obtained between i frame pixel and the i+1 frame pixel concerns, the coordinate of all the other 97 pixels of i frame is changed with this model, tried to achieve 97 corresponding in i+1 frame pixel coordinates.Meet this model if the distance between 97 pixel coordinates obtaining originally in corresponding 97 pixel coordinates and the i+1 frame in the i+1 frame after the conversion less than 2 pixel values, then is set at, write down the number n of all qualified pixels.Said process is repeated 20 times, the set of pixel when record n is maximum, the set of the pixel based on this n when maximum is determined 4 global motion parameters in the global motion model with least square method.
After obtaining the global motion parameter of successive frame, use Gaussian function level and smooth s respectively, α, d x, d yThe curve movement of these 4 global motion parameters, the Gaussian kernel of high speed function is:
G ( k ) = 1 2 πσ e - k 2 / σ 2
In the following formula, σ is a standard deviation, can get σ=2 here, and k is the difference between present frame sequence number and the target frame sequence number, and for example present frame is the 5th frame, and target frame is the 10th frame, k=10-5=5 so.
The i two field picture is A to the global motion parameter of i+1 two field picture I+1, smoothly the global motion parameter after is T I+1, the motion compensation vector Δ I+1=T I+1-A I+1, pass through Δ I+1The i+1 two field picture is carried out motion compensation can obtain two field picture after stable.
Corresponding based on the embodiment of the image processing method of optical flow algorithm with the application, the application also provides the embodiment based on the image processing apparatus of optical flow algorithm.
Referring to Fig. 3, be the first embodiment block diagram of the application based on the image processing apparatus of optical flow algorithm.
This device comprises: two field picture acquiring unit 420, optical flow algorithm unit 430, sampling screening unit 440, global parameter acquiring unit 450, parameter smoothing unit 460 and image compensation unit 470.
Wherein, two field picture acquiring unit 420 is used for obtaining the current frame image of pending video sequence;
Optical flow algorithm unit 430 is used for obtaining one group of pixel position in the next frame image of current frame image according to optical flow algorithm;
Sampling screening unit 440 is used for according to the global parameter model of adjacent two two field pictures of setting up in advance this group of pixel being sampled, and filters out the pixel of displacement in preset range;
Global parameter acquiring unit 450 is used for according to the pixel that the filters out position coordinates at current frame image and next frame image, obtains global motion parameter from current frame image to the next frame image by least square method;
Parameter smoothing unit 460 is used for the global motion parameter is carried out smoothly;
Image compensation unit 470 is used for according to the level and smooth result of parameter smoothing unit 460 the next frame image being compensated.
Referring to Fig. 4 A, be the second embodiment block diagram of the application based on the image processing apparatus of optical flow algorithm.
This device comprises: default unit 410, two field picture acquiring unit 420, optical flow algorithm unit 430, sampling screening unit 440, global parameter acquiring unit 450, parameter smoothing unit 460 and image compensation unit 470.
Wherein, default unit 410 is used to set in advance the number of times that one group of pixel is sampled;
Two field picture acquiring unit 420 is used for obtaining the current frame image of pending video sequence;
Optical flow algorithm unit 430 is used for obtaining one group of pixel position in the next frame image of current frame image according to optical flow algorithm;
Sampling screening unit 440 is used for according to the global parameter model of adjacent two two field pictures of setting up in advance one group of pixel being sampled, and filters out the pixel of displacement in preset range;
Global parameter acquiring unit 450 is used for according to the pixel that the filters out position coordinates at current frame image and next frame image, obtains global motion parameter from current frame image to the next frame image by least square method;
Parameter smoothing unit 460 is used for the global motion parameter is carried out smoothly;
Image compensation unit 470 is used for according to the level and smooth result of parameter smoothing unit 460 the next frame image being compensated.
Concrete, referring to Fig. 4 B, the embodiment block diagram for optical flow algorithm unit 430 comprises:
Macroblock partitions subelement 431 is used for current frame image is divided into several macro blocks, and obtains a pixel in each macro block;
Local Vector computation subunit 432 is used for following the tracks of and calculate the local motion vector of each pixel at the next frame image by optical flow algorithm;
Position coordinates computation subunit 433 is used for calculating the position coordinates of this pixel in the next frame image according to each pixel at the position coordinates and the local motion vector of this pixel in the next frame image of current frame image.
Concrete, referring to Fig. 4 C, the embodiment block diagram for sampling screening unit 440 comprises:
Frequency in sampling control sub unit 441 is used to control that global parameter is obtained subelement 442, position coordinates obtains subelement 443 and pixel statistics subelement 444 orders are carried out the sampling consistent with frequency in sampling;
Global parameter is obtained subelement 442, be used for the position coordinates and the position coordinates substitution global parameter model of two pixels in the next frame image of two pixels of the current frame image of picked at random are obtained the global motion parameter, the global motion parameter comprises image stretch parameter, image rotation parameter, image transverse translation parameter and image longitudinal translation parameter;
Position coordinates obtains subelement 443, be used for the global motion equation that the position coordinates input of the residual pixel point except that two pixels of picked at random in one group of pixel of current frame image is definite according to the global motion parameter, obtain the position coordinates of residual pixel point in the next frame image;
Pixel statistics subelement 444, be used to add up the position coordinates that obtains according to the global motion equation and the position coordinates that obtains according to optical flow algorithm between the pixel of distance in preset range;
Sampling number record subelement 445 is used to write down the number of the pixel of displacement in preset range that sampling each time obtained;
Sampling results is confirmed subelement 446, is used for the number of pixel pixel when maximum as the pixel that filters out.
Concrete, referring to Fig. 4 D, the embodiment block diagram for parameter smoothing unit 450 comprises:
Curve movement is determined subelement 451, is used for the curve movement of determining the global motion parameter according to the global motion parameter and the frame number of all two field pictures of video sequence;
Motion smoothing subelement 452 is used for by gaussian filtering the curve movement of global motion parameter being carried out motion smoothing;
Smoothing parameter is asked for subelement 453, is used for asking for each global motion parameter after level and smooth on the curve movement behind the motion smoothing according to frame number.
Concrete, referring to Fig. 4 E, the embodiment block diagram for image compensation unit 460 comprises:
Compensating parameter obtains subelement 461, is compensated parameter after being used for global motion parameter after level and smooth and the global motion parameter of obtaining by least square method asked difference;
Two field picture compensation subelement 462 is used for by compensating parameter the next frame image being compensated.
As seen through the above description of the embodiments, after obtaining the current frame image in the pending video sequence in the embodiment of the present application, obtain one group of pixel position in the next frame image in the current frame image according to optical flow algorithm, global parameter model according to adjacent two two field pictures of setting up is in advance sampled to one group of pixel, filter out the pixel of displacement in preset range, according to the position coordinates of the pixel that filters out in current frame image and next frame image, obtain global motion parameter by least square method from current frame image to the next frame image, according to the global motion parameter is carried out level and smooth result, the next frame image is compensated.In the embodiment of the present application owing to two field picture is handled by optical flow algorithm and the global motion model mode that combines of sampling, therefore when guaranteeing global motion calculation of parameter precision, reduced amount of calculation, improve antijamming capability, satisfied real-time video stabilization alignment requirements.
As seen through the above description of the embodiments, those skilled in the art can be well understood to the application and can realize by the mode that software adds essential general hardware platform.Based on such understanding, the part that the application's technical scheme contributes to prior art in essence in other words can embody with the form of software product, this computer software product can be stored in the storage medium, as ROM/RAM, magnetic disc, CD etc., comprise that some instructions are with so that a computer equipment (can be a personal computer, server, the perhaps network equipment etc.) carry out the described method of some part of each embodiment of the application or embodiment.
Each embodiment in this specification all adopts the mode of going forward one by one to describe, and identical similar part is mutually referring to getting final product between each embodiment, and each embodiment stresses all is difference with other embodiment.Especially, for system embodiment, because it is substantially similar in appearance to method embodiment, so description is fairly simple, relevant part gets final product referring to the part explanation of method embodiment.
The application can be used in numerous general or special purpose computingasystem environment or the configuration.For example: personal computer, server computer, handheld device or portable set, plate equipment, multicomputer system, the system based on microprocessor, set top box, programmable consumer-elcetronics devices, network PC, minicom, mainframe computer, comprise distributed computing environment (DCE) of above any system or equipment or the like.
The application can describe in the general context of the computer executable instructions of being carried out by computer, for example program module.Usually, program module comprises the routine carrying out particular task or realize particular abstract, program, object, assembly, data structure or the like.Also can in distributed computing environment (DCE), put into practice the application, in these distributed computing environment (DCE), by by communication network connected teleprocessing equipment execute the task.In distributed computing environment (DCE), program module can be arranged in the local and remote computer-readable storage medium that comprises memory device.
Though described the application by embodiment, those of ordinary skills know, the application has many distortion and variation and the spirit that do not break away from the application, wish that appended claim comprises these distortion and variation and the spirit that do not break away from the application.

Claims (12)

1. the image processing method based on optical flow algorithm is characterized in that, comprising:
After obtaining the current frame image in the pending video sequence, obtain one group of pixel position in the next frame image in the described current frame image according to optical flow algorithm;
Global parameter model according to adjacent two two field pictures of setting up is in advance sampled to described one group of pixel, filters out the pixel of displacement in preset range;
According to the position coordinates of the described pixel that filters out in current frame image and next frame image, obtain global motion parameter from described current frame image to the next frame image by least square method;
According to described global motion parameter is carried out level and smooth result, described next frame image is compensated.
2. method according to claim 1 is characterized in that, describedly obtains according to optical flow algorithm that one group of pixel position in the next frame image comprises in the described current frame image:
Described current frame image is divided into several macro blocks, and obtains a pixel in each macro block;
Follow the tracks of and calculate the local motion vector of described each pixel in the next frame image by optical flow algorithm;
Calculate the position coordinates of described pixel in the next frame image according to the position coordinates and the local motion vector of described pixel in the next frame image of each pixel in current frame image.
3. method according to claim 1 is characterized in that, described global parameter model according to adjacent two two field pictures of setting up is in advance sampled to described one group of pixel and filtered out the pixel of displacement in preset range and comprise:
The position coordinates of two pixels in the described current frame image of picked at random and the described global parameter model of the position coordinates substitution of described two pixels in the next frame image are obtained the global motion parameter, and described global motion parameter comprises image stretch parameter, image rotation parameter, image transverse translation parameter and image longitudinal translation parameter;
Global motion equation with the position coordinates input of the residual pixel point except that two pixels of described picked at random in one group of pixel in the described current frame image is determined according to described global motion parameter obtains the position coordinates of described residual pixel point in the next frame image;
Add up the pixel of distance in described preset range between described position coordinates that obtains according to the global motion equation and the described position coordinates that obtains according to optical flow algorithm.
4. method according to claim 3 is characterized in that, described method also comprises: set in advance described one group of number of times that pixel is sampled;
Described filtering out before the pixel of displacement in preset range also comprises:
According to described global parameter model described one group of pixel is carried out the sampling consistent with described frequency in sampling;
The number of the pixel of displacement in preset range that record sampling is each time obtained;
Pixel with the number of described pixel when maximum is as the described pixel that filters out.
5. method according to claim 1 is characterized in that, described described global motion parameter is smoothly comprised:
The curve movement of determining the global motion parameter according to the global motion parameter and the frame number of all two field pictures in the described video sequence;
By gaussian filtering the curve movement of described global motion parameter is carried out motion smoothing;
Asking for each global motion parameter after level and smooth on the curve movement behind the described motion smoothing according to frame number.
6. method according to claim 1 is characterized in that, the described next frame image is compensated comprises:
Described global motion parameter after level and smooth and the described global motion parameter of obtaining by least square method asked be compensated parameter after differing from;
By described compensating parameter described next frame image is compensated.
7. the image processing apparatus based on optical flow algorithm is characterized in that, comprising:
The two field picture acquiring unit is used for obtaining the current frame image of pending video sequence;
The optical flow algorithm unit is used for obtaining one group of pixel position in the next frame image of described current frame image according to optical flow algorithm;
Sampling screening unit is used for according to the global parameter model of adjacent two two field pictures of setting up in advance described one group of pixel being sampled, and filters out the pixel of displacement in preset range;
The global parameter acquiring unit is used for according to the position coordinates of the described pixel that filters out at current frame image and next frame image, obtains global motion parameter from described current frame image to the next frame image by least square method;
The parameter smoothing unit is used for described global motion parameter is carried out smoothly;
Image compensation unit is used for according to the level and smooth result of described parameter smoothing unit described next frame image being compensated.
8. device according to claim 7 is characterized in that, described optical flow algorithm unit comprises:
The macroblock partitions subelement is used for described current frame image is divided into several macro blocks, and obtains a pixel in each macro block;
The Local Vector computation subunit is used for following the tracks of and calculate the local motion vector of described each pixel at the next frame image by optical flow algorithm;
The position coordinates computation subunit is used for calculating the position coordinates of described pixel in the next frame image according to each pixel at the position coordinates and the local motion vector of described pixel in the next frame image of current frame image.
9. device according to claim 7 is characterized in that, described sampling screening unit comprises:
Global parameter is obtained subelement, be used for the position coordinates and the described global parameter model of the position coordinates substitution of described two pixels in the next frame image of two pixels of the described current frame image of picked at random are obtained the global motion parameter, described global motion parameter comprises image stretch parameter, image rotation parameter, image transverse translation parameter and image longitudinal translation parameter;
Position coordinates obtains subelement, be used for the global motion equation that the position coordinates input of the residual pixel point except that two pixels of described picked at random in one group of pixel of described current frame image is definite according to described global motion parameter, obtain the position coordinates of described residual pixel point in the next frame image;
Pixel statistics subelement is used to add up the pixel of distance in described preset range between described position coordinates that obtains according to the global motion equation and the described position coordinates that obtains according to optical flow algorithm.
10. device according to claim 9 is characterized in that, also comprises:
Default unit is used to set in advance to described one group of number of times that pixel is sampled;
Described sampling screening unit also comprises:
The frequency in sampling control sub unit is used to control that described global parameter is obtained subelement, position coordinates acquiring unit and pixel statistics subelement is carried out the sampling consistent with described frequency in sampling in proper order;
Sampling number record subelement is used to write down the number of the pixel of displacement in preset range that sampling each time obtained;
Sampling results is confirmed subelement, is used for the number of described pixel pixel when maximum as the described pixel that filters out.
11. device according to claim 7 is characterized in that, described parameter smoothing unit comprises:
Curve movement is determined subelement, is used for the curve movement of determining the global motion parameter according to the global motion parameter and the frame number of described all two field pictures of video sequence;
The motion smoothing subelement is used for by gaussian filtering the curve movement of described global motion parameter being carried out motion smoothing;
Smoothing parameter is asked for subelement, is used for asking for each global motion parameter after level and smooth on the curve movement behind the described motion smoothing according to frame number.
12. device according to claim 7 is characterized in that, described image compensation unit comprises:
Compensating parameter obtains subelement, is used for described global motion parameter after level and smooth and the described global motion parameter of obtaining by least square method asked being compensated parameter after differing from;
Two field picture compensation subelement is used for by described compensating parameter described next frame image being compensated.
CN 201010107256 2010-02-01 2010-02-01 Image processing method and device based on optical flow algorithm Active CN101854465B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN 201010107256 CN101854465B (en) 2010-02-01 2010-02-01 Image processing method and device based on optical flow algorithm

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN 201010107256 CN101854465B (en) 2010-02-01 2010-02-01 Image processing method and device based on optical flow algorithm

Publications (2)

Publication Number Publication Date
CN101854465A true CN101854465A (en) 2010-10-06
CN101854465B CN101854465B (en) 2012-06-27

Family

ID=42805710

Family Applications (1)

Application Number Title Priority Date Filing Date
CN 201010107256 Active CN101854465B (en) 2010-02-01 2010-02-01 Image processing method and device based on optical flow algorithm

Country Status (1)

Country Link
CN (1) CN101854465B (en)

Cited By (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102436487A (en) * 2011-11-03 2012-05-02 北京电子科技学院 Optical flow method based on video retrieval system
CN102629969A (en) * 2012-03-30 2012-08-08 邓晓波 Smear eliminating method during shooting of plane objects
CN102685438A (en) * 2012-05-08 2012-09-19 清华大学 Up-conversion method of video frame rate based on time-domain evolution
CN103810692A (en) * 2012-11-08 2014-05-21 杭州海康威视数字技术股份有限公司 Method for video tracking of video monitoring equipment and video monitoring equipment
CN103824254A (en) * 2014-03-12 2014-05-28 广州博冠信息科技有限公司 Image processing method and device based on view point
CN105323420A (en) * 2014-07-29 2016-02-10 腾讯科技（深圳）有限公司 Video image processing method and apparatus
CN105791705A (en) * 2016-05-26 2016-07-20 厦门美图之家科技有限公司 Video anti-shake method and system suitable for movable time-lapse photography and shooting terminal
CN103824254B (en) * 2014-03-12 2016-11-30 广州博冠信息科技有限公司 A kind of image processing method based on viewpoint and device
CN106251317A (en) * 2016-09-13 2016-12-21 野拾（北京）电子商务有限公司 Space photography stabilization processing method and processing device
CN106469456A (en) * 2015-08-11 2017-03-01 株式会社理光 Method for estimating and movement estimation apparatus
CN108540720A (en) * 2018-04-14 2018-09-14 长沙全度影像科技有限公司 A kind of system and method for choosing optimum image automatically for the bullet time
CN109255797A (en) * 2017-07-14 2019-01-22 富士通株式会社 Image processing apparatus and method, electronic equipment
CN109302545A (en) * 2018-11-15 2019-02-01 深圳市炜博科技有限公司 Video image stabilization method, device and computer readable storage medium
CN109640641A (en) * 2016-09-05 2019-04-16 渔觅创新私人有限公司 Feeding system and bait-throwing method
CN110827194A (en) * 2019-10-25 2020-02-21 北京迈格威科技有限公司 Image processing method, device and computer storage medium
CN113034552A (en) * 2019-12-25 2021-06-25 Tcl集团股份有限公司 Optical flow correction method and computer equipment
CN113034383A (en) * 2021-02-24 2021-06-25 大连海事大学 Method for obtaining video image based on improved grid motion statistics
CN113160277A (en) * 2021-01-29 2021-07-23 北京小米松果电子有限公司 Image processing method and device, electronic equipment and storage medium
CN113436302A (en) * 2021-06-08 2021-09-24 合肥综合性国家科学中心人工智能研究院(安徽省人工智能实验室) Face animation synthesis method and system
CN113660443A (en) * 2020-05-12 2021-11-16 武汉Tcl集团工业研究院有限公司 Video frame insertion method, terminal and storage medium
CN114322839A (en) * 2020-09-28 2022-04-12 上海微电子装备（集团）股份有限公司 Method for detecting light spot profile

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108280445B (en) * 2018-02-26 2021-11-16 江苏裕兰信息科技有限公司 Method for detecting moving objects and raised obstacles around vehicle

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101453557B (en) * 2008-12-30 2010-09-15 浙江大学 Quick global motion estimation method based on motion vector cancellation and differential principle

Cited By (36)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102436487B (en) * 2011-11-03 2014-03-05 北京电子科技学院 Optical flow method based on video retrieval system
CN102436487A (en) * 2011-11-03 2012-05-02 北京电子科技学院 Optical flow method based on video retrieval system
CN102629969A (en) * 2012-03-30 2012-08-08 邓晓波 Smear eliminating method during shooting of plane objects
CN102685438A (en) * 2012-05-08 2012-09-19 清华大学 Up-conversion method of video frame rate based on time-domain evolution
CN102685438B (en) * 2012-05-08 2015-07-29 清华大学 A kind of up-conversion method of video frame rate based on time-domain evolution
CN103810692A (en) * 2012-11-08 2014-05-21 杭州海康威视数字技术股份有限公司 Method for video tracking of video monitoring equipment and video monitoring equipment
CN103810692B (en) * 2012-11-08 2016-12-21 杭州海康威视数字技术股份有限公司 Video monitoring equipment carries out method and this video monitoring equipment of video tracking
CN103824254B (en) * 2014-03-12 2016-11-30 广州博冠信息科技有限公司 A kind of image processing method based on viewpoint and device
CN103824254A (en) * 2014-03-12 2014-05-28 广州博冠信息科技有限公司 Image processing method and device based on view point
CN105323420B (en) * 2014-07-29 2019-02-22 腾讯科技（深圳）有限公司 Method of video image processing and device
CN105323420A (en) * 2014-07-29 2016-02-10 腾讯科技（深圳）有限公司 Video image processing method and apparatus
CN106469456A (en) * 2015-08-11 2017-03-01 株式会社理光 Method for estimating and movement estimation apparatus
CN106469456B (en) * 2015-08-11 2019-04-19 株式会社理光 Method for estimating and movement estimation apparatus
CN105791705B (en) * 2016-05-26 2019-06-11 厦门美图之家科技有限公司 Video anti-fluttering method, system and camera terminal suitable for mobile time-lapse photography
CN105791705A (en) * 2016-05-26 2016-07-20 厦门美图之家科技有限公司 Video anti-shake method and system suitable for movable time-lapse photography and shooting terminal
CN109640641A (en) * 2016-09-05 2019-04-16 渔觅创新私人有限公司 Feeding system and bait-throwing method
CN106251317B (en) * 2016-09-13 2018-12-18 野拾（北京）电子商务有限公司 Space photography stabilization processing method and processing device
CN106251317A (en) * 2016-09-13 2016-12-21 野拾（北京）电子商务有限公司 Space photography stabilization processing method and processing device
JP7014005B2 (en) 2017-07-14 2022-02-01 富士通株式会社 Image processing equipment and methods, electronic devices
CN109255797A (en) * 2017-07-14 2019-01-22 富士通株式会社 Image processing apparatus and method, electronic equipment
JP2019021297A (en) * 2017-07-14 2019-02-07 富士通株式会社 Image processing device and method, and electronic apparatus
CN108540720A (en) * 2018-04-14 2018-09-14 长沙全度影像科技有限公司 A kind of system and method for choosing optimum image automatically for the bullet time
CN108540720B (en) * 2018-04-14 2020-07-03 长沙全度影像科技有限公司 System and method for automatically selecting optimal image for bullet time
CN109302545A (en) * 2018-11-15 2019-02-01 深圳市炜博科技有限公司 Video image stabilization method, device and computer readable storage medium
CN109302545B (en) * 2018-11-15 2021-06-29 深圳万兴软件有限公司 Video image stabilization method and device and computer readable storage medium
CN110827194A (en) * 2019-10-25 2020-02-21 北京迈格威科技有限公司 Image processing method, device and computer storage medium
CN113034552A (en) * 2019-12-25 2021-06-25 Tcl集团股份有限公司 Optical flow correction method and computer equipment
CN113034552B (en) * 2019-12-25 2024-02-20 Tcl科技集团股份有限公司 Optical flow correction method and computer equipment
CN113660443A (en) * 2020-05-12 2021-11-16 武汉Tcl集团工业研究院有限公司 Video frame insertion method, terminal and storage medium
CN114322839A (en) * 2020-09-28 2022-04-12 上海微电子装备（集团）股份有限公司 Method for detecting light spot profile
CN114322839B (en) * 2020-09-28 2023-05-05 上海微电子装备（集团）股份有限公司 Method for detecting light spot contour
CN113160277A (en) * 2021-01-29 2021-07-23 北京小米松果电子有限公司 Image processing method and device, electronic equipment and storage medium
CN113034383A (en) * 2021-02-24 2021-06-25 大连海事大学 Method for obtaining video image based on improved grid motion statistics
CN113034383B (en) * 2021-02-24 2024-06-28 大连海事大学 Method for acquiring video image based on improved grid motion statistics
CN113436302A (en) * 2021-06-08 2021-09-24 合肥综合性国家科学中心人工智能研究院(安徽省人工智能实验室) Face animation synthesis method and system
CN113436302B (en) * 2021-06-08 2024-02-13 合肥综合性国家科学中心人工智能研究院(安徽省人工智能实验室) Face animation synthesis method and system

Also Published As

Publication number Publication date
CN101854465B (en) 2012-06-27

Similar Documents

Publication Publication Date Title
CN101854465B (en) Image processing method and device based on optical flow algorithm
Li et al. PDR-Net: Perception-inspired single image dehazing network with refinement
CN110058685B (en) Virtual object display method and device, electronic equipment and computer-readable storage medium
CN105981075B (en) Utilize the efficient facial landmark tracking in wire shaped homing method
US11265531B2 (en) Systems and methods of creating a three-dimensional virtual image
WO2021048607A1 (en) Motion deblurring using neural network architectures
CN111208783B (en) Action simulation method, device, terminal and computer storage medium
CN107749987B (en) Digital video image stabilization method based on block motion estimation
CN105791705A (en) Video anti-shake method and system suitable for movable time-lapse photography and shooting terminal
CN107610069B (en) DVS visual video denoising method based on shared K-SVD dictionary
CN111160298A (en) Robot and pose estimation method and device thereof
CN112819876B (en) Monocular vision depth estimation method based on deep learning
CN112639878A (en) Unsupervised depth prediction neural network
CN115294275A (en) Method and device for reconstructing three-dimensional model and computer readable storage medium
CN105100546A (en) Movement estimation method and device
CN115690382A (en) Training method of deep learning model, and method and device for generating panorama
CN111476835A (en) Unsupervised depth prediction method, system and device for consistency of multi-view images
DE112020005730T5 (en) Joint rolling shutter correction and image sharpening
CN111191553A (en) Face tracking method and device and electronic equipment
CN114584785A (en) Real-time image stabilizing method and device for video image
CN112241001B (en) Radar human body action recognition method, radar human body action recognition device, electronic equipment and storage medium
CN110120009B (en) Background blurring implementation method based on salient object detection and depth estimation algorithm
Jang et al. Optimal sampling for shape from focus by using gaussian process regression
CN116468745A (en) Image processing method, related device and computer readable storage medium
CN115937374A (en) Digital human body modeling method, device, equipment and medium

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant
ASS Succession or assignment of patent right

Owner name: HANGZHOU HIKVISION DIGITAL TECHNOLOGY CO., LTD.

Free format text: FORMER OWNER: HANGZHOU HAIKANG WEISHI SOFTWARE CO., LTD.

Effective date: 20130107

C41 Transfer of patent application or patent right or utility model
COR Change of bibliographic data

Free format text: CORRECT: ADDRESS; FROM: 310012 HANGZHOU, ZHEJIANG PROVINCE TO: 310051 HANGZHOU, ZHEJIANG PROVINCE

TR01 Transfer of patent right

Effective date of registration: 20130107

Address after: Hangzhou City, Zhejiang province 310051 Binjiang District East Road Haikang Science Park No. 700, No. 1

Patentee after: Hangzhou Hikvision Digital Technology Co., Ltd.

Address before: Ma Cheng Road Hangzhou City, Zhejiang province 310012 No. 36

Patentee before: Hangzhou Haikang Weishi Software Co., Ltd.