CN104506775A - Image collection jitter removing method and device based on stereoscopic visual matching - Google Patents

Abstract

The invention is applicable to the field of intelligent terminal and image processing, and provides an image collection jitter removing method based on stereoscopic visual matching; the method comprises the following steps: using a FAST characteristic point to detect characteristic interest points of extracted forward and backward frames of images; using a scheme based on the ORB descriptor to match the FAST characteristic interest points of the forward and backward frames of images; using the normalization eight-point method based on the matched characteristic points to solve a transfer matrix F of the rotary motion of an image camera between the forward and backward frames; using the transfer matrix F to correct the deviation caused by the jitter, so as to obtain a corrected image. The provided method has the advantages of preventing jitter of multiple frames of images, and fast in calculating speed.

Description

Based on IMAQ jitter eliminating method and the device of stereoscopic vision coupling

Technical field

The invention belongs to intelligent terminal and image processing field, particularly relate to a kind of IMAQ jitter eliminating method based on stereoscopic vision coupling and device.

Background technology

Along with living standard constantly improves, current IMAQ, mobile phone photograph have become people and to have lived a requisite part.But due to impacts such as technique for taking and external object motions, free-hand image of taking pictures may occur shaking the phenomenons such as fuzzy, and at this moment people are often by the stable equilibrium device in the external world, as tripod, displacement detecting sensing unit etc. carry out auxiliary photographic images.By extra aid, although the problem of IMAQ stabilization effectively can be solved, also make troubles to shooting simultaneously, huge instrument must be carried with.The method of later image process, makes to take not by also solving the fuzzy problem of shake under the prerequisite of the instrument in the external world.Several can be taken when sometimes people take simultaneously, but like this at the picture do not taken in the same time due to free-hand jitter problem, often bring shake clearly to photographic images.

Along with improving constantly of scientific and technological level, handheld mobile device is provided with the function of camera shooting, it is that the light of outer scene is focused on by optical lens and forms optical imagery on the image sensor that camera obtains the process of image, and by the integration of certain hour, make optical imagery be converted into digital picture.In this process, optical imagery nature static on the image sensor must be kept, otherwise will there is fuzzy shake in acquisition integral image, affect picture quality., front industry adopts image taking shake to remove major optical stabilization and electronic flutter-proof.

By the overall relative motion vectors obtaining inter frame image, patent CN200510108036.2 and CN200510108037.7 judges whether present image exists shake, and carry out jitter compensation by this overall relative motion vectors to image.Patent CN200910188891.7 and patent CN201210013718.5 proposes to use accelerometer and displacement detecting sensing unit to carry out detection camera respectively and whether there is jitter problem, finally carry out according to detecting the shake displacement occurred the change that correction image shakes generation, but this method needs extra hardware to assist, and increases scheme Cost Design.Patent CN201010526514 is by controlling the control of shutter threshold value and carrying out filtering noise reduction process to image and improve flating Fuzzy Quality, but the jitter problem that this method is only suitable for single-frame images is eliminated.

Current most of prior art is all shake fuzzy solving for the problem of camera shooting stabilization based on single image, but user needs hand-held racket according to continuous a few two field picture or dynamic video sometimes, if at this moment by taken freehand image, often be easy to the phenomenon that shake occurs, this is also that handheld device is taken pictures one of greatest problem.

Summary of the invention

The object of the embodiment of the present invention is to provide a kind of IMAQ jitter eliminating method based on stereoscopic vision coupling and device, and it solves the problem that cannot solve multiple image shake of prior art.

The embodiment of the present invention is achieved in that on the one hand, and provide a kind of IMAQ jitter eliminating method based on stereoscopic vision coupling, described method comprises:

FAST feature point detection is adopted to extract the feature point of interest of front and back two two field pictures;

Adopt the FAST feature point of interest of two two field pictures before and after the scheme coupling based on ORB descriptor;

Characteristic point according to coupling utilizes normalization 8 methods to solve the transition matrix F in rotary moving of front and back inter frame image camera;

The skew brought after revising shake by transition matrix F obtains revised picture.

Optionally, the feature point of interest that described employing FAST feature point detection extracts front and back two two field pictures specifically comprises:

After adopting FAST feature point detection to obtain candidate feature point, with candidate feature point for the center of circle, the circumference of pre-set radius extracts 4 points every an angle of 90 degrees, obtains the difference of 4 some gray values and the candidate feature point gray value extracted; As in four differences, at least three differences are greater than gray threshold, then retain this candidate feature point, otherwise, delete this candidate feature point.

On the other hand, provide a kind of IMAQ jitter elimination device based on stereoscopic vision coupling, described device comprises:

Extraction unit, for the feature point of interest adopting FAST feature point detection to extract front and back two two field pictures;

Matching unit, for adopting the FAST feature point of interest of two two field pictures before and after the scheme coupling based on ORB descriptor;

Solve unit, for utilizing normalization 8 methods to solve the transition matrix F in rotary moving of front and back inter frame image camera according to the characteristic point of coupling;

Amending unit, for obtaining revised picture by the skew brought after transition matrix F correction shake.

Optionally, described extraction unit specifically for

After adopting FAST feature point detection to obtain candidate feature point, with candidate feature point for the center of circle, the circumference of pre-set radius extracts 4 points every an angle of 90 degrees, obtains the difference of 4 some gray values and the candidate feature point gray value extracted; As in four differences, at least three differences are greater than gray threshold, then retain this candidate feature point, otherwise, delete this candidate feature point.

In embodiments of the present invention, technical scheme provided by the invention is very high based on the algorithm operation efficiency of FAST characteristic point and ORB descriptor coupling, the memory headroom as limited in mobile phone, panel computer etc. of embedded device store and computational resource product in have good application prospect.The algorithm for stereo matching computation complexity that the present invention proposes is very low, has very large advantage in operational efficiency.The flating correction of being mated by stereoscopic vision, can solve the problem of taken freehand photo or video instability effectively, in the IMAQ etc. in future, have very large application prospect.

Accompanying drawing explanation

Fig. 1 is the flow chart of a kind of IMAQ jitter eliminating method based on stereoscopic vision coupling provided by the invention;

Fig. 2 is FAST feature point detection schematic diagram provided by the invention;

Fig. 3 is the result schematic diagram of the Image Feature Point Matching to the shooting of front and back frame provided by the invention;

The prior image frame design sketch that Fig. 4 (a) is taken freehand;

The rear two field picture design sketch that Fig. 4 (b) is taken freehand;

Fig. 4 (c) carries out revised design sketch for adopting method provided by the invention;

Fig. 5 (a) is non-correction effect Fig. 1;

Fig. 5 (b) is for adopting the revised design sketch 1 of method provided by the invention;

Fig. 6 (a) is non-correction effect Fig. 2;

Fig. 6 (b) is for adopting the revised design sketch 2 of method provided by the invention;

Fig. 7 (a) is non-correction effect Fig. 3;

Fig. 7 (b) is for adopting the revised design sketch 3 of method provided by the invention;

Fig. 8 (a) is non-correction effect Fig. 4;

Fig. 8 (b) is for adopting the revised design sketch 4 of method provided by the invention.

Embodiment

In order to make object of the present invention, technical scheme and advantage clearly understand, below in conjunction with drawings and Examples, the present invention is further elaborated.Should be appreciated that specific embodiment described herein only in order to explain the present invention, be not intended to limit the present invention.

The specific embodiment of the invention provides a kind of IMAQ jitter eliminating method based on stereoscopic vision coupling, said method is performed by mobile phone or other smart machine (such as ipad, PDA etc.), certainly in actual applications, can be performed by other smart machine, the specific embodiment of the invention does not limit to the concrete manifestation form performing the method equipment or device yet.The method as shown in Figure 1, comprising:

101, FAST feature point detection is adopted to extract the feature point of interest (i.e. characteristic point) of front and back two two field pictures;

102, the FAST feature point of interest of two two field pictures before and after the scheme coupling based on ORB descriptor is adopted;

103, normalization 8 methods are utilized to solve the transition matrix F in rotary moving of front and back inter frame image camera according to the characteristic point of coupling;

104, the skew brought after revising shake by transition matrix F obtains revised picture.

The present invention proposes the shake transformation matrix of method to inter frame image giving stereoscopic vision coupling and solves in image procossing aspect, efficiently solves the problem that flating brings.Algorithm operation efficiency based on FAST characteristic point and ORB descriptor coupling is very high, the memory headroom as limited in mobile phone, panel computer etc. of embedded device store and computational resource product in have good application prospect.The algorithm for stereo matching computation complexity that the present invention proposes is very low, has very large advantage in operational efficiency.The flating correction of being mated by stereoscopic vision, can solve the problem of taken freehand photo or video instability effectively, in the IMAQ etc. in future, have very large application prospect.

FAST feature point detection algorithm derives from the definition of corner, based on image intensity value around each characteristic point, detect the pixel value made a circle in candidate feature point week, if there is the gray value difference of abundant pixel and this candidate point enough large around candidate point in field, then think that this candidate point is a characteristic point:

$N=\underset{x\∀\left(\mathrm{circile}\left(p\right)\right)}{\mathrm{\Σ}}|I\left(x\right)-I\left(p\right)|\≥{\mathrm{\ϵ}}_d$

Wherein, I (x) is with I (p), the gray value of the circumferentially any point of pre-set radius, the gray value that I (p) is the center of circle, ε _dfor given threshold value.If N is greater than given threshold epsilon, be generally 3/4ths of circle points around, then think that p is a feature point of interest.

Optionally, the implementation method of above-mentioned 101 is specifically as follows:

After adopting FAST feature point detection to obtain candidate feature point, with candidate feature point for the center of circle, the circumference of pre-set radius extracts 4 points every an angle of 90 degrees, point 1,5,13,9 as shown in Figure 2; Obtain the difference of 4 some gray values and the candidate feature point gray value extracted; As in four differences, at least three differences are greater than gray threshold, then retain this candidate feature point, otherwise, delete this candidate feature point.Adopt radius to be 3 in Fig. 2 of the present invention, have 16 neighboring pixel points and compare, specifically as shown in Figure 2.In order to improve the efficiency compared, the pixel of the part in 16 points is usually only used to compare, i.e. FAST-N.Adopt this kind of method can accelerate the speed of FAST feature point detection, reduce the quantity of candidate feature point.

Optionally, the implementation method of above-mentioned 102 is specifically as follows:

Scheme based on ORB descriptor mates FAST feature point of interest.ORB (English full name: OrientedBRIEF) descriptor has directivity.The principal direction of its characteristic point is solved by the calculating of square, and formula is as follows:

$M_{\mathrm{ij}}=\underset{x}{\mathrm{\Σ}}\underset{y}{\mathrm{\Σ}}{x}^i{y}^{j}I(x,y)$

$c_x=\frac{M_{10}}{M_{00}},c_y=\frac{M_{01}}{M_{00}}$

$C_{\mathrm{ori}}={\tan }^{-1}\left(\frac{c_y}{c_x}\right)$

M ₁₀, M ₀₀the first moment of difference character pair point and zeroth order square, C _orifor the principal direction of characteristic point.After determining the principal direction of characteristic point, utilize the right gray scale magnitude relationship of random point in topography field to set up the Feature Descriptor of topography, not only matching speed is fast for the binary feature descriptor obtained, and memory requirement internal memory is low, therefore has fine application background in the embedded devices such as mobile phone.Figure 3 shows that the result of the present invention to the Image Feature Point Matching that front and back frame is taken.

Optionally, the implementation method of above-mentioned 103 is specifically as follows:

Suppose shown in the following formula of the change in displacement of inter frame image:

${\stackrel{\→}{M}}^{\′}=R\stackrel{\→}{M}+t$

Wherein M is the position of former frame camera, and M ' is the position of a rear frame camera, and R, t are respectively two interframe camera spin matrix and transposed matrixs, obtain m after M normalized, then:

${\stackrel{\→}{m}}^{\′}\·t\×{\stackrel{\→}{m}}^{\′}={\stackrel{\→}{m}}^{\′}\·t\×\left(R\stackrel{\→}{m}\right)$

${\stackrel{\→}{m}}^{\′}F\stackrel{\→}{m}=0$

Right according to the point matched, utilize 8 methods can solve transition matrix F.

As the front and back two field picture that Fig. 4 (a), Fig. 4 (b) are taken freehand, because two two field pictures are not being taken in the same time, free-hand meeting brings the problem of shake, as Fig. 4 (c) carries out revised design sketch for adopting method provided by the invention, the jitter problem between (c) and (b) is substantially eliminated.

Algorithm of the present invention is by a large amount of verification experimental verifications, by the object that taken freehand is different, direct uncorrected front and back two field picture is synthesized result and synthesize result by two field picture before and after the correction after algorithm for stereo matching process and contrast, as shown in drawings, can find out, mating revised result by stereoscopic vision can effectively solve flating problem.

Wherein, Fig. 5 (a) is non-correction effect figure, Fig. 5 (b) is revised design sketch;

Fig. 6 (a) is non-correction effect figure, Fig. 6 (b) is revised design sketch;

Fig. 7 (a) is non-correction effect figure, Fig. 7 (b) is revised design sketch;

Fig. 8 (a) is non-correction effect figure, Fig. 8 (b) is revised design sketch.

The present invention also provides a kind of IMAQ jitter elimination device based on stereoscopic vision coupling, and described device comprises:

Extraction unit, for the feature point of interest adopting FAST feature point detection to extract front and back two two field pictures;

Matching unit, for adopting the FAST feature point of interest of two two field pictures before and after the scheme coupling based on ORB descriptor;

Solve unit, for the transition matrix F in rotary moving utilizing normalization 8 methods to solve front and back inter frame image camera;

Amending unit, for obtaining revised picture according to the skew brought after transition matrix F correction shake.

Optionally, described extraction unit specifically for

After adopting FAST feature point detection to obtain candidate feature point, with candidate feature point for the center of circle, the circumference of pre-set radius extracts 4 points every an angle of 90 degrees, obtains the difference of 4 some gray values and the candidate feature point gray value extracted; As in four differences, at least three differences are greater than gray threshold, then retain this candidate feature point, otherwise, delete this candidate feature point.

Said apparatus can be mobile terminal, the smart machine such as panel computer or PDA.

It should be noted that in above-described embodiment, included unit is carry out dividing according to function logic, but is not limited to above-mentioned division, as long as can realize corresponding function; In addition, the concrete title of each functional unit, also just for the ease of mutual differentiation, is not limited to protection scope of the present invention.

In addition, one of ordinary skill in the art will appreciate that all or part of step realized in the various embodiments described above method is that the hardware that can carry out instruction relevant by program has come, corresponding program can be stored in a computer read/write memory medium, described storage medium, as ROM/RAM, disk or CD etc.

The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, all any amendments done within the spirit and principles in the present invention, equivalent replacement and improvement etc., all should be included within protection scope of the present invention.

Claims (4)

1., based on an IMAQ jitter eliminating method for stereoscopic vision coupling, it is characterized in that, described method comprises:

FAST feature point detection is adopted to extract the feature point of interest of front and back two two field pictures;

Adopt the FAST feature point of interest of two two field pictures before and after the scheme coupling based on ORB descriptor;

Characteristic point according to coupling utilizes normalization 8 methods to solve the transition matrix F in rotary moving of front and back inter frame image camera;

The skew brought after revising shake by transition matrix F obtains revised picture.

2. method according to claim 1, is characterized in that, the feature point of interest that described employing FAST feature point detection extracts front and back two two field pictures specifically comprises:

After adopting FAST feature point detection to obtain candidate feature point, with candidate feature point for the center of circle, the circumference of pre-set radius extracts 4 points every an angle of 90 degrees, obtains the difference of 4 some gray values and the candidate feature point gray value extracted; As in four differences, at least three differences are greater than gray threshold, then retain this candidate feature point, otherwise, delete this candidate feature point.

3., based on an IMAQ jitter elimination device for stereoscopic vision coupling, it is characterized in that, described device comprises:

Extraction unit, for the feature point of interest adopting FAST feature point detection to extract front and back two two field pictures;

Matching unit, for adopting the FAST feature point of interest of two two field pictures before and after the scheme coupling based on ORB descriptor;

Solve unit, for utilizing normalization 8 methods to solve the transition matrix F in rotary moving of front and back inter frame image camera according to the characteristic point of coupling;

Amending unit, for obtaining revised picture by the skew brought after transition matrix F correction shake.

4. device according to claim 3, is characterized in that, described extraction unit specifically for

After adopting FAST feature point detection to obtain candidate feature point, with candidate feature point for the center of circle, the circumference of pre-set radius extracts 4 points every an angle of 90 degrees, obtains the difference of 4 some gray values and the candidate feature point gray value extracted; As in four differences, at least three differences are greater than gray threshold, then retain this candidate feature point, otherwise, delete this candidate feature point.