CN102243537A - Method and device for detecting displacement of motion image as well as optical mouse - Google Patents

Abstract

The invention discloses a method and a device for detecting the displacement of a motion image, and the method and device provided by the invention are used for analyzing an image sequence with a temporal relation so as to determine the motion trail of a photoinduced component relative to an object in the process of acquiring the image sequence, and can be used for an optical mouse. The method comprises the following steps: receiving the image sequence by an image signal processing unit according to a time sequence; and based on the image sequence, sequentially carrying out data processing on each current image, carrying out two-dimensional optical-flow computed processing on the images according to each pixel on a difference image corresponding to the current image, and calculating and outputting a displacement vector corresponding to the current image by using a global matrix operational method. Through using an optical mouse disclosed by the invention, the accuracy of navigation positioning and the stability of detection are effectively improved.

Description

Moving image displacement detecting method and device, optical mouse

It is the electric numerical data treatment technology of analytic target that technical field the present invention relates to the moving image, relates in particular to the data processing method and the device of the moving displacement detection that is used for the optical mouse field.

Background technology is as the main flow of current mouse circle, and the light emitting diode localization method that optical mouse adopted has also become present main flow localization method.

Fig. 1 illustrated the inner structure of optical mouse to comprise light emitting diode 1, from the light of this light emitting diode 1 by 2 guiding of one group of optical mirror and irradiation toward the surface of touching with the optical mouse bottom connection; By the light part of this return reflection surface through one group of optical lens 2 ' being transferred to photoinduction device 3 (such as but not limited to little imager of exportable data image signal) carries out imaging.Like this, when optical mouse moved, its motion track just can be disclosed by the set of diagrams picture that high speed " continuous shooting " is noted.Utilize the included image dissector of optical mouse 4 (can be based on the special chip of digital signal processor DSP) to come this group image of analyzing and processing, can draw described motion track.In actual applications, described photoinduction device 3 and image dissector 4 often are integrated in the chips.

Fig. 2 has illustrated the electricity work principle of optical mouse, and signal is the electric structure of optical mouse in the frame of broken lines, has the interface that an interface and that connects button connects host computer.This electricity structure comprises imageing sensor (sensor), is used for gathering and is electrical picture signal with the light image conversion of signals; This electrical picture signal is sent to analog to digital converter to convert data image signal to, is sent to image signal processing unit again and calculates generation mouse displacement vector; Micro-control unit fetches controlling packet and draws together imageing sensor, analog to digital converter or image signal processing unit at each interior circuit module by being electrically connected, and receives the status signal from each circuit module; Pack together by this micro-control unit from the displacement vector data of image signal processing unit with from the key information of mousebutton and to send to host computer.Described little imager has comprised imageing sensor and analog to digital converter at least.Image signal processing unit can adopt a special-purpose chips, perhaps in micro-control unit is integrated in a chips.

Fig. 3 has illustrated described mouse displacement vector.Suppose that a, b are corresponding two two field pictures of time successively, the same unique point on the triangle representative image, then image signal processing unit is by extracting described unique point and analyzing its change in location and judge mouse moving direction and displacement.Figure c has illustrated that characteristic point position changes the Δ X that is calculated, and Δ Y uniquely corresponding to the mouse displacement vector, thereby finishes cursor positioning.Existing localization method adopts the motion vector that obtains mouse based on the interpolation algorithm of relevant matches mostly, as shown in Figure 4, can (a) obtain current displacement vector (as figure c) in the position of unique point optimum matching by in next frame (as figure b), seeking with present frame as figure, wherein scheming d and illustrated the template of a 3*3 pixel, can be that optimum matching is sought in the unit in image-region with this template.This method has the advantage of stable performance, but because the limitation of interpolation algorithm, can't effectively obtain the precision of sub-pix size.

In order to obtain higher kinematic accuracy, prior art further is used for the mouse location with the method for zoning light stream, to obtain up to the precision more than 1/4 pixel.This method is set up overdetermination binary optical flow equation to the pixel on the image earlier, finds the solution by this equation and obtains single pixel motion vector, and simulate a plurality of pixel motion vector averages, the i.e. motion vector of mouse with least square method.In order to simplify the description to this method, it is that example is illustrated that Fig. 5 is simplified to one-dimensional signal to the picture signal from the photoinduction device.If curve A is a t signal constantly, curve B is a t+ Δ t signal constantly; Suppose constantly at t, the signal amplitude value of x position be f (x, t); At t+ Δ t constantly, the signal amplitude value of x+ Δ x position is f (x+ Δ x, t+ Δ t).If signal has moved Δ x distance at t in the time, under the situation of not considering other interference, can know so:

f(x+Δx，t+Δt)＝f(x，t) (1)

The following formula left side is got as Taylor expansion:

$f(x+\mathrm{\Δx},t+\mathrm{\Δt})=f(x,t)+\mathrm{\Δx}\frac{\∂f}{\∂x}+\mathrm{\Δt}\frac{\∂f}{\∂t}+o(\mathrm{\Δx},\mathrm{\Δt})$

Simultaneous (1) formula, and ignore o (Δ x, Δ t):

$\mathrm{\Δx}\frac{\∂f}{\∂x}+\mathrm{\Δt}\frac{\∂f}{\∂t}=0---\left(2\right)$

That is:

I _xu _x+I _t＝0 (3)

In the formula,

Be the slope of signal in the x position, For signal in the x position rate over time.

Be the movement velocity of signal on the x direction.Formula (3) is the optical flow equation of motion in one dimension signal.

In like manner, the optical flow equation of 2D signal is:

I _xu _x+I _yu _y+I _t＝0 (4)

This moment, an equation was not enough to find the solution two unknown number u _x, u _yTherefore to seek a plurality of about u _x, u _yEquation find the solution:

So

${(u_x,u_y)}^T={\left({\stackrel{\→}{I}}^T\stackrel{\→}{I}\right)}^{-1}{\stackrel{\→}{I}}^T{I}_t={\left(\begin{array}{cc}\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{I}_{\mathrm{xi}}^2& \underset{i=1}{\overset{n}{\mathrm{\Σ}}}{I}_{\mathrm{xi}}{I}_{\mathrm{yi}}\\ \underset{i=1}{\overset{n}{\mathrm{\Σ}}}{I}_{\mathrm{xi}}{I}_{\mathrm{yi}}& \underset{i=1}{\overset{n}{\mathrm{\Σ}}}{I}_{\mathrm{yi}}^2\end{array}\right)}^{-1}\left(\begin{array}{c}-\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{I}_{\mathrm{xi}}{I}_{\mathrm{ti}}\\ -\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{I}_{\mathrm{yi}}{I}_{\mathrm{ti}}\end{array}\right)={\left(\begin{array}{cc}a& b\\ b& c\end{array}\right)}^{-1}\left(\begin{array}{c}e\\ f\end{array}\right)---\left(6\right)$

For instructing foundation, in described image signal processing unit, the signal processing circuit of existing optical flow method displacement calculating has treatment scheme as shown in Figure 6 with formula 6.After data image signal is sent to this signal processing circuit, earlier through one remove make an uproar, the differentiate processing module generates space derivative I _x, I _y(two components of pixel gradient value) and time-derivative I _t, and further calculate the long-pending of acquisition derivative thus

I _xI _y, I _xI _t, I _yI _tCharacteristic area is searched module and follow-up parameter calculating module thereof and is chosen specific region on the image according to described data image signal, with this regional m*m=n pixel is that template is searched characteristic area, and based on this characteristic area be used to make an uproar from removing, the corresponding data of differentiate processing module comes parameter a, b, c, e and f in the accounting equation 6, promptly

With

Each parameter add up and; The displacement vector computing module calculates and exports displacement vector (u according to these parameters again _x, u _y).In the above-mentioned processing, generally choose the bigger zone of Grad as characteristic area.

The shortcoming of above-mentioned prior art is: when calculating described displacement vector easily by background disturb influence, thereby influence the bearing accuracy of mouse.Because the influence of aspects such as optical lens (for example inhomogeneous) and the encapsulation of optical sensor later stage with impurity or reflectivity, image from photoinduction device 3 in the structure shown in Figure 1 will have inevitable background noise, and also there is fixing noise in mouse circuit itself in addition.As shown in Figure 7, Fig. 7 a is two frame image signals before and after undesired signal is represented respectively with two curves 1 and 2 can ignore the time; 7b represents that background disturbs and can not ignore that promptly disturb (thick line 3 with bending is represented) fainter situation of useful signal with respect to background, adopt optical flow method to carry out calculation of parameter and will bring very computation error this moment.

The summary of the invention the technical problem to be solved in the present invention is at above-mentioned the deficiencies in the prior art part, and proposes a kind of method and device that is used for the moving image displacement detecting, effectively reduces the influence that the background in the motion images acquired disturbs.

For solving the problems of the technologies described above, of the present inventionly be contemplated that substantially: consider that background noise has the characteristics slowly that change, for the front and back two field picture that the high speed continuous shooting is obtained, can suppose that its background noise is approximate consistent, therefore handle if come the alternative image signal to carry out two-dimentional optical flow computation with the inter-frame difference picture signal, and in calculating process, adopt overall matrix operation method, will greatly get rid of the influence that background noise disturbs undoubtedly, and simplify calculating process.

As the technical scheme that realizes the present invention's design be, a kind of moving image displacement detecting method be provided, be used to analyze have time relationship image sequence to determine the gathering movement locus of this image sequence time sensing device with respect to object, comprise step:

A. image signal processing unit receives described image sequence in chronological order; Especially, described image sequence comprises three frames or the above image of three frames; Also comprise step

B. based on described image sequence, successively each present image is carried out data processing, foundation is calculated with each pixel on the corresponding difference image of this current image, by following formula calculating and the output displacement vector (u corresponding with present image _x, u _y)

D wherein _x, D _yBe the space derivative of differential pixel, representing two components of differential pixel Grad; D _tTime-derivative for difference image; N is the pixel count in this image.

Specifically, in the such scheme, described step B is specially and comprises step:

1. produce the difference image sequence according to described image sequence, promptly frame image data behind the frame in this image sequence is deducted after the current frame image data difference image data as present frame, setting this back frame image data again is the current frame image data ... the rest may be inferred constitutes described difference image sequence up to obtaining two frames or the difference image more than two frames;

2. utilize frame difference image behind current difference image and the one to calculate the described space derivative D of each pixel in the current difference image _x, D _yWith time-derivative D _t

3. by multiplying each other computing to obtain each middle coefficient in the described formula in twos

D _xD _y, D _xD _t, D _yD _t

4. in current difference image scope, calculate each middle coefficient in the described formula add up and; And and then calculate described displacement vector (u _x, u _y).

As the technical scheme that realizes the present invention's design still, provide a kind of moving image displacement detector, be used for optical mouse; Comprise a data-interface and a data delivery outlet that is used for receiving data image signal; Especially, also comprise difference processing module and differentiate processing module that polyphone connects, is after unit carries out the derivative operation of the calculus of differences of interframe data and differentiate processing module from the data image signal of described data-interface with the frame through the difference processing module, the space derivative D of output and each pixel of the corresponding difference image of current frame image _x, D _yWith time-derivative D _tBe sent to the global operations module, calculate each middle coefficient of all pixels in this image preset range respectively

D _xD _y, D _xD _t, D _yD _tAdd up and; This global operations module is exported these and is added up and be sent to the displacement vector computing unit, with by formula

Come displacement calculating vector (u _x, u _y), wherein n is the quantity of the pixel in the described image preset range; This displacement vector computing unit output displacement vector (u _x, u _y) toward described data output.

As the technical scheme that realizes the present invention's design still, a kind of optical mouse is provided, comprise the light emitting diode that irradiates light is provided, be used for responding to irradiates light that is reflected by object plane and the imageing sensor that light signal is converted to analog picture signal, convert described analog picture signal the analog to digital converter of data image signal to, and an image signal processing unit that receives described data image signal; Comprise that also one is used for being electrically connected the micro-control unit of host computer and mousebutton, this micro-control unit fetches control described imageing sensor, analog to digital converter or image signal processing unit by being electrically connected, and receives the status signal from described imageing sensor, analog to digital converter or image signal processing unit; The data output of described image signal processing unit connects described micro-control unit; Especially, this image signal processing unit is described a kind of moving image displacement detector.

Adopt the optical mouse of above-mentioned each technical scheme, the influence that disturbed by background reduces greatly, thereby when effectively having improved the degree of accuracy of mouse navigator fix, has increased the stability that detects; Help improving the finished product rate.

Description of drawings Fig. 1 is the light principle and the inner structure synoptic diagram of existing mouse performance;

Fig. 2 is the electric principle and the electrical block diagram of existing mouse performance;

Fig. 3 has illustrated the calculating principle of existing mouse displacement vector;

Fig. 4 is the algorithm synoptic diagram that has now based on the relevant matches location feature zone of Fig. 3;

Fig. 5 is to be the generation synoptic diagram of example based on the optical flow equation of Fig. 3 with the one-dimensional signal;

Fig. 6 has illustrated the signal processing flow block diagram in the conventional images signal processing unit;

Fig. 7 has illustrated background to disturb the influence to signal Processing, wherein 7a has illustrated background to disturb optical sensor shot image signals when ignoring, described picture signal when 7b has illustrated to exist background to disturb, 7c has illustrated the inventive method to eliminate background interference back difference image signal;

Fig. 8 has illustrated the inventive method flow process;

Fig. 9 is for adopting the image signal processing unit internal signal treatment circuit block diagram of the present invention of Fig. 8 method;

Figure 10 has illustrated the image acquisition among Fig. 8 to import and difference differentiate process;

Figure 11 is the example procedure figure that image difference and differentiate are handled among Figure 10;

Figure 12 is for being used for selecting the template instances figure of image pixel with differentiate among Figure 10 or Figure 11;

Figure 13 is one of embodiment that realizes Fig. 8 method;

Figure 14 is two of the embodiment that realizes Fig. 8 method.

Below the embodiment, the most preferred embodiment shown in is further set forth the present invention in conjunction with the accompanying drawings.

Moving image displacement detecting method of the present invention is used for analyzing image sequence with time relationship to determine to gather the movement locus of this image sequence process photoinduction device with respect to object.The approach that obtains described image sequence includes but not limited to: image taking sensor is an example, the analog electrical signal that will be in the taken reflection input light intensity change information of this imageing sensor in the relative motion generates digital signal and sends into DSP (digital signal processor) through analog to digital converter.In this DSP, described digital signal is converted into the view data of a frame frame earlier, promptly described image sequence.The inventive method comprises step:

A. image signal processing unit receives described image sequence in chronological order; Described image sequence comprises three frames or the above image of three frames;

B. based on described image sequence, successively each present image is carried out data processing, foundation is calculated with each pixel on the corresponding difference image of this current image.For example three two field pictures or the external phase of continuous adjacent are carried out interframe difference in twos every the data of three two field pictures of k (k is a natural number) frame, promptly the gray-scale value of three two field pictures subtracts each other in twos in order and obtains differentiated two frame image datas, set up model based on this two frame differences view data with optical flow method again and come the displacement calculating vector, can eliminate existing background noise interference in the original image signal like this.

Because optical flow method must have only the three frame original images of employing just can obtain a displacement vector based on two two field pictures; For comprising the above effectively image sequence of image of k ' (k '＞3) frame, we set the effective image of each frame successively is present image, then can obtain k '-2 displacement vector at most, so the inventive method can also comprise step:

C. simulate described movement locus according to each displacement vector among the step B and the time interval between correspondence image thereof.

In the foregoing description, described interframe difference in twos refers to each pixel of image and all carries out corresponding interframe data difference (when described view data is transmitted in the mode of video stream signal, refer to the data difference between the video stream signal of adjacent or the pairing time difference of k frame of being separated by, wherein k is a natural number; Easy for narrating, this place no longer does difference to the video stream signal mode and describes): i.e. D (x, y, t)=and P (x, y, t)-P (x, y, t-k), P (x wherein, y t) represents x in the t moment frame image data, the gray-scale value of the pixel of y position, D (x, y t) represents x in the t moment frame difference view data, the gray-scale value of the pixel of y position.Wherein the k value depends on the movement velocity of relative motion in the frame frequency of imaging and the imaging process.Frame frequency is big and under the situation that movement velocity is slow, k can suitably get greatly; Otherwise frame frequency is little and under the situation that movement velocity is fast, k is the smaller the better.

In above-mentioned steps B, use the detailed process of optical flow method as follows:

It still is example with the one-dimensional signal.If t picture signal constantly is shown in curve 1 among Fig. 7 (a), t+ Δ t time chart image signal is shown in curve 2; Shown in Fig. 7 (b), influenced by background noise, the signal that imageing sensor obtains is along the track fluctuation of curve 3.Do not consider other noise if only consider background noise to disturb, actual detected to picture signal generally can be expressed as:

P(x，t)＝I(x，t)A(x)+B(x) (7)

Wherein, (x t) represents actual signal to I, and A (x) represents the property the taken advantage of interference coefficient of background, and B (x) represents the additivity interference coefficient of background, establishes both not t variations in time of A (x) and B (x).

With the I in the formula (3) _xChange P into _xAnd both sides are to time t differentiate, and are located at signal at the uniform velocity (being reflected in the two dimensional image signal promptly is that mouse is at the uniform velocity mobile) under the mobile condition, the u in the formula (3) _xNot variation in time or approximate constant can get the difference optical flow equation:

P _xtu _x+P _tt＝0 (8)

(8) formula is made normalized, and I _Xt=P (x, t) _Xt, I _Tt=P (x, t) _Tt, simultaneous formula (7):

{[I _i+1(x)-I _i(x)] _xA(x)+[I _i+1(x)-I _i(x)]A(x) _x}u _x+[I _i+1(x)-I _i(x)] _iA(x)＝0

Usually, background noise is very little at the rate of change in space, [I _I+1(x)-I _i(x)] _xA (x) [I _I+1(x)-I _i(x)] A (x) _x, but so the following formula abbreviation is:

[I _i+1(x)-I _i(x)] _xu _x+[I _i+1(x)-I _i(x)] _t＝0

Make D again _i(x)=I _I+1(x)-I _i(x), the substitution following formula gets:

D _xu _x+D _t＝0 (9)

Then formula (9) for after the normalization with differential signal D _i(x) be the difference optical flow equation of signal source.This formula shows that D is exactly the difference signal of two frames before and after the signal I.Shown in Fig. 7 (c),, eliminated the background interference basically through differentiated signal.

For two-dimentional difference optical flow equation:

D _xu _x+D _yu _y+D _t＝0 (10)

Because ask for two variable u _x, u _ySo, a plurality of pixels are set up overdetermination difference optical flow equation group:

Here suppose the displacement vector (u of each pixel _x, u _y) approximately equal (all equaling mouse moving displacement vector), it is as follows to get the difference optical flow equation after the arrangement:

${(u_x,u_y)}^T={\left({\stackrel{\&RightArrow;}{D}}^T\stackrel{\&RightArrow;}{D}\right)}^{-1}{\stackrel{\&RightArrow;}{D}}^T{D}_t={\left(\begin{array}{cc}\underset{i=1}{\overset{\mathrm{<figure-callout\; id="2"\; label="n\; D\; xi"\; filenames="HSA00000124798300011.png,HSA00000124798300032.png"\; state="\{\{state\}\}">n</figure-callout>}}{\mathrm{\&Sigma;}}}{D}_{\mathrm{xi}}^{}{}_2^{}& \underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{D}_{\mathrm{xi}}{D}_{\mathrm{yi}}\\ \underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{D}_{\mathrm{xi}}{D}_{\mathrm{yi}}& \underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{D}_{\mathrm{yi}}^2\end{array}\right)}^{-1}\left(\begin{array}{c}-\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{D}_{\mathrm{xi}}{D}_{\mathrm{ti}}\\ -\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{D}_{\mathrm{yi}}{D}_{\mathrm{ti}}\end{array}\right)={\left(\begin{array}{cc}A& B\\ B& C\end{array}\right)}^{-1}\left(\begin{array}{c}E\\ F\end{array}\right)---\left(12\right)$

In this equation, use D _x, D _y, D _tReplaced the I in traditional optical flow equation _x, I _y, I _t, promptly to image differentiate (asking slope) time, use difference image to replace original image.Therefore, displacement vector (u that can by formula calculate (12) and output is corresponding with present image among the step B of the present invention _x, u _y), D wherein _x, D _yBe the space derivative of differential pixel, representing two components of differential pixel Grad; D _tBe the time number; N is the pixel count in this image.

Fig. 8 has illustrated to use the detailed process of the inventive method:

Obtained image sequence before this: specifically can be as shown in figure 10, imageing sensor is gathered light intensity signal that has reflected true picture and then the analog voltage signal that converts one dimension in the line scanning mode, and this analog voltage signal is sent into DSP after converting data image signal to through analog to digital converter; Suppose that the single-frame images pixel is m * m, for example getting natural number m is 16, then separated by (T=) per 256 clock period at the quilt of data image signal described in the DSP, converts the view data of a frame frame to.Original image F2 as shown, F3, every two field picture has comprised 256 pixels among the F4......, and each pixel has a gray-scale value.We can use P _{K, i, j}(i=1,2......m, j=1 2......m) represent that the k two field picture is in coordinate (i, j) the pixel corresponding gray scale value of position (its physical meaning is the sample signal strength of this pixel correspondence, down together).

As shown in Figure 8, then carry out difference processing, produce the difference image sequence according to described image sequence, promptly frame image data behind the frame in this image sequence is deducted after the current frame image data difference image data as present frame, setting this back frame image data again is the current frame image data ... the rest may be inferred constitutes described difference image sequence up to obtaining two frames or the difference image more than two frames.Described back two field picture or be the next frame image of present image in the described image sequence perhaps is following some (〉=2) two field picture of present image in the described image sequence.That is to say, establish with F _kThe image of representing the k frame, the difference image of this two field picture correspondence is F _K+i-F _kThe result, the next frame difference image of this frame difference image then is F _K+i+m-F _K+iThe result, wherein k, i, m are natural number.As described i=m=1, then representing successively, two frame difference images are to carry out interframe difference acquisition in twos by three adjacent successively in image sequence two field pictures; When i=m ≠ 1, then representing two frame difference images successively is that three two field pictures by being separated by successively in the image sequence frame pitch of etc.ing carry out interframe difference acquisition in twos; As i ≠ m, then represent successively two frame difference images be by in the image sequence successively not to wait frame pitch three two field pictures adjacent or that be separated by to carry out interframe difference acquisition in twos.

Figure 10 with three two field pictures adopt consecutive frame carry out interframe in twos difference be example, i.e. original image F3-original image F2=difference image DF5, original image F4-original image F3=difference image DF6.Figure 11 has illustrated a difference embodiment among Figure 10: establish P _{K-1, i, j}, P _{K, i, j}, P _{K+1, i, j}(i=1,2......16; J=1 2......16) represents k-1 respectively, k, and the k+1 two field picture is in that (then difference image k frame data are D at the gray-scale value of the pixel of (i, j) position for i, the j) gray-scale value of the pixel of position _{K, i, j}=P _{K, i, j}-P _{K-1, i, j}Difference image k+1 frame data are D at the gray-scale value of the pixel of i, j position _{K+1, j, j}=P _{K+1, i, j}-P _{K, i, j}The rest may be inferred can try to achieve the gray-scale value D of other each pixel among this embodiment _{K, i+1, j}, D _{K, i-1, j}, D _{K+1, i+1, j}, D _{K+i, i-1, j}, D _{K, i+1, j}, D _{K+1, i+1, j}, D _{K, i-1, j}, D _{K+1, i-1, j}, D _{K+1, i, j}, D _{K+1, i, j}Two frame difference image D have been produced with three two field pictures like this _{K, i, j}, D _{K+1, i, j}In the practical application, can also in more than the image sequence of three frames, set the image call condition, the image adjacent successively or that be separated by that satisfies predetermined condition is chosen to be three two field pictures of desiring to carry out calculus of differences, and the rest may be inferred up to accessed time of this image sequence.Therefore, may have some images in the described image sequence is excluded outside described effective image because of failing to satisfy this predetermined condition.

Then among Fig. 8 the difference image sequence is carried out differentiate and handle, utilize frame difference image behind current difference image and the one to calculate the described space derivative D of each pixel in the current difference image _x, D _yWith time-derivative D _tFigure 10 is with difference image DF5, and DF6 is an example.On this basis, can computing obtain each middle coefficient in the formula (12) by multiplying each other in twos

D _xD _y, D _xD _t, D _yD _tTo carry out the global operations among Fig. 8, promptly in current difference image scope, calculate each middle coefficient in the described formula add up and

Described multiplying each other in twos can realize with multiplier, for example

All the other are no longer given unnecessary details at this.

More than, can only handle the central area of each two field picture, correspondingly, described n is the pixel count in this central area, and the pixel of promptly establishing a two field picture is M*M, and wherein M is a natural number, and then n is less than M*M.As M=16 among the above-mentioned embodiment, calculate herein add up and the time n can but be not limited to be taken as 10 * 10=100.Usually, processing region is the maximal correlation zone of differentiating of image-region to the maximum, and for example to the embodiment of same M=16, the pixel count n that this maximal correlation zone comprises is 14 * 14.

Calculate based on formula (12) and output displacement vector (u as Fig. 8 at last _x, u _y).

Specific implementation process about differentiate in the foregoing description can adopt prior art, and just the object of differentiate changes as difference image data.Illustrate with an embodiment below: because the D that will obtain _x, D _y, D _tBe respectively D _{K, i, j}At x, the partial derivative on the y direction, the calculating that normalizes on the pixel is the poor of neighbor gray-scale value; The derivative of time t normalizes to calculating on the pixel, and to be same pixel at the gray-scale value of adjacent two frames poor, specifically is calculated as follows:

D _x＝[D _k，i，j+1+D _{k+1，i，j+1}]-[D _k，i，j-1+D _{k+1，i，j-1}]

D _y＝[D _k，i+1，j+D _{k+1，i+1，j}]-[D _k，i-1，j+D _{k+1，i-1，j}]

D _t＝4×[D _k+1，i，j-D _k，i，j]

Wherein coefficient can be represented with template shown in Figure 12: wherein 1 expression coefficient is that 1 ,-1 expression coefficient is that-1, two squares are represented adjacent two frames.According to this template with D _{K, i, j}Move select on two two field pictures corresponding each pixel data in the hope of this D _{K, i, j}Corresponding space derivative.

Calculate D _x, D _y, D _tThe step (as Figure 13) that differentiate is handled after adopting above-mentioned first difference processing, can also adopt the second way, comprise step:

1. two field picture calculates the space derivative I that obtains each pixel in the present image after utilizing present image and one according to described image sequence _x, I _yWith time-derivative I _t

2. the space derivative I of each pixel in the back two field picture _x, I _yWith time-derivative I _tDeduct the space derivative and the time-derivative of respective pixel in the current frame image, produce described space derivative D with each pixel of the corresponding difference image of present image _x, D _yWith time-derivative D _t

Being that the back difference processing is handled in first differentiate, is example with as shown in figure 14 three two field pictures of getting specifically: do differentiate earlier on original image and calculate two couples of derivative value I _X1, I _Y1, I _T1And I _X2, I _Y2, I _T2, again these two pairs of derivative value are done difference and obtain D _x, D _y, D _t

Existing optic flow technique has to seek in image and determine the bigger characteristic area of gradient, mainly consider to detect in the video area to have a plurality of moving objects, and the big young pathbreaker of characteristic area is subject to the size of estimated moving target.The inventive method is during then based on mouse movement, the displacement vector basically identical of each pixel on the image that photographs because of imageing sensor, be that moving target has only one, and signal noise ratio (snr) of image is not high, if adopt characteristic area less, and then provide full of prunes result then easily owing to noise is mistaken for the optimal computed zone.So the inventive method adopts global operations, abandon carrying out the searching process of characteristic area and the computing zone broadening is the overall situation.Generally to the entire image zone of 16 * 16 pixels, the maximum selectable zoning of the inventive method is 14 * 14.Because of guaranteeing that great majority point on the image is carried out optical flow computation, it also is feasible using zone (such as but not limited to picture centre zone 10 * 10 pixels) less than 14 * 14 to come calculating parameter A, B, C, E and F.Through the checking of matlab emulation experiment, the inventive method had both helped improving algorithm stability, also helped improving the accuracy of mouse location, had especially simplified the step of seeking characteristic area and will make that the circuit of realizing this method is simpler.

According to the inventive method, the present invention also proposes a kind of moving image displacement detector of implementing the inventive method, is used for optical mouse; Comprise a data-interface and a data delivery outlet that is used for receiving data image signal; Especially, as shown in Figure 9, this device comprises difference processing module and the differentiate processing module that polyphone connects, data image signal from described data-interface is the calculus of differences that unit carries out interframe data with the frame through the difference processing module, and after the derivative operation of differentiate processing module, the space derivative D of each pixel of output and the corresponding difference image of current frame image _x, D _yWith time-derivative D _tBe sent to the global operations module, calculate each middle coefficient of all pixels in this image preset range respectively

D _xD _y, D _xD _t, D _yD _tAdd up and; This global operations module is exported these and is added up and past displacement vector computing unit, with by formula

Come displacement calculating vector (u _x, u _y),

Wherein n is the quantity of the pixel in the described image preset range; This displacement vector computing unit output displacement vector (u _x, u _y) toward described data output.Described formula further launches, and reality is

$\left\{\begin{array}{c}{u}_x=\frac{\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{D}_{\mathrm{xi}}{D}_{\mathrm{yi}}*\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{D}_{\mathrm{yi}}{D}_{\mathrm{ti}}-\underset{i=1}{\overset{\mathrm{<figure-callout\; id="2"\; label="n\; D\; yi"\; filenames="HSA00000124798300011.png,HSA00000124798300032.png"\; state="\{\{state\}\}">n</figure-callout>}}{\mathrm{\&Sigma;}}}{D}_{\mathrm{yi}}^{}{}_2^{}*\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{D}_{\mathrm{xi}}{D}_{\mathrm{ti}}}{\underset{i=1}{\overset{\mathrm{<figure-callout\; id="2"\; label="n\; D\; xi"\; filenames="HSA00000124798300011.png,HSA00000124798300032.png"\; state="\{\{state\}\}">n</figure-callout>}}{\mathrm{\&Sigma;}}}{D}_{\mathrm{xi}}^{}{}_2^{}*\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{D}_{\mathrm{yi}}^2-\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{\left(D_{\mathrm{xi}}{D}_{\mathrm{yi}}\right)}^2}\\ u_y=\frac{\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{D}_{\mathrm{xi}}{D}_{\mathrm{yi}}*\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{D}_{\mathrm{xi}}{D}_{\mathrm{ti}}-\underset{i=1}{\overset{\mathrm{<figure-callout\; id="2"\; label="n\; D\; xi"\; filenames="HSA00000124798300011.png,HSA00000124798300032.png"\; state="\{\{state\}\}">n</figure-callout>}}{\mathrm{\&Sigma;}}}{D}_{\mathrm{xi}}^{}{}_2^{}*\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{D}_{\mathrm{yi}}{D}_{\mathrm{ti}}}{\underset{i=1}{\overset{\mathrm{<figure-callout\; id="2"\; label="n\; D\; xi"\; filenames="HSA00000124798300011.png,HSA00000124798300032.png"\; state="\{\{state\}\}">n</figure-callout>}}{\mathrm{\&Sigma;}}}{D}_{\mathrm{xi}}^{}{}_2^{}*\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{D}_{\mathrm{yi}}^2-\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{\left(D_{\mathrm{xi}}{D}_{\mathrm{yi}}\right)}^2}\end{array}\right..$

The best use field of considering apparatus of the present invention comprises optical mouse, and apparatus of the present invention can be encapsulated as a chips, can also be integrated into existing micro-control unit in this chip, and this micro-control unit has the keystroke interface that is used for receiving push button signalling; This micro-control unit receives from the data of described keystroke interface with from the displacement vector data of described displacement vector computing unit, and with these packing data outputs toward described data output.Described data output can be used to be electrically connected host computer.

When described data image signal is when providing by the video data of serial ports is streamed, apparatus of the present invention also must comprise at least two frame delay devices of polyphone mutually, connect described data-interface, are that unit separates with the frame with described data image signal; Order is away from the second frame delay device output current frame image data of described data-interface, then the output of the first frame delay device is the data of a back two field picture of this present frame, what described data-interface was exported simultaneously is the data of a back two field picture of this back two field picture, and this three frame image data is exported simultaneously toward the polyphone branch road of difference processing module and differentiate processing module.

Specifically connect with reference to Figure 13 and Figure 14.Figure 13 has illustrated one of embodiment: described difference processing module is connected between described data-interface and the differentiate processing module, the described second frame delay device and the first frame delay device provide one tunnel output to be sent to described difference processing module and to carry out inter-frame difference respectively, are sent to described differentiate processing module thereby this difference processing module produces two frame difference view data simultaneously.Specifically, the difference processing module will deduct the view data that obtains difference image D1 from the view data of the first frame delay device from the view data of data-interface, will deduct the view data that obtains difference image D2 from the view data of the second frame delay device from the view data of the first frame delay device; The differentiate processing module to difference image D1, D2 from the difference processing module ask respectively space derivative (Dx1, Dy1) and (Dx2, Dy2), to Dx1, the Dx2 summation obtains Dx, to Dy1, the Dy2 summation obtains Dy, and difference image D1, D2 are the poor time-derivative Dt that obtains.

Figure 14 has illustrated two of embodiment: described differentiate processing module is connected between described data-interface and the difference processing module, the described second frame delay device and the first frame delay device provide one tunnel output to be sent to described differentiate processing module respectively, thereby this differentiate processing module produces two frame space derivative I simultaneously _x, I _yWith time-derivative I _tBe sent to described difference processing module and carry out inter-frame difference to obtain differentiated space derivative D _x, D _yWith time-derivative D _tSpecifically, the differentiate processing module is asked two time-derivative It1 to subtracting each other in twos from the view data of data-interface, from the view data of the first frame delay device, from the view data of the second frame delay device, It2, and to from the view data of data-interface, from the view data of the first frame delay device, ask space derivative (Ix1 respectively from the view data of the second frame delay device, Iy1), (Ix2, Iy2), (Ix3, Iy3); The difference processing module is to the time-derivative It1 from the differentiate processing module, and It2 subtracts each other and obtains Dt, to space derivative (Ix1, Iy1), (Ix2, Iy2), (Ix3, Iy3) subtract each other in twos and obtain (Dx1, Dy1), (Dx2, Dy2), again to Dx1, the Dx2 summation obtains Dx, and to Dy1, the Dy2 summation obtains Dy.

Two frame delay devices among Figure 13 and Figure 14 can also be equal to alternative with the frame buffer storer.

Differentiate processing module of apparatus of the present invention and difference processing module, global operations module, displacement vector computing module can be that various calculation process are realized on the basis with a large amount of totalizers or multiplier all, how to use and connect these totalizers or multiplier, under the calculation process of its qualification, be prior art.Therefore frame delay device, difference processing module, differentiate processing module, global operations module or the displacement vector computing module of apparatus of the present invention all can be based on prior aries, realize according to its included Processing Algorithm with FPGA (FieldProgrammable Gate-Array field programmable gate array) or ASIC (Application Specific IntegrateCircuit special IC) or DSP (Digital Signal Processot digital signal processor), and use on-chip memory or plug-in storer as required.Because of these are prior art, also non-emphasis of the present invention is not so give unnecessary details at this.When in every case constituting apparatus of the present invention, will fall into protection scope of the present invention undoubtedly by the functional module of these prior aries.

Existing optical mouse, comprise the light emitting diode that irradiates light is provided, be used for responding to irradiates light that is reflected by object plane and the imageing sensor that light signal is converted to analog picture signal, convert described analog picture signal the analog to digital converter of data image signal to, and an image signal processing unit that receives described data image signal; Comprise that also one is used for being electrically connected the micro-control unit of host computer and mousebutton, this micro-control unit fetches control described imageing sensor, analog to digital converter or image signal processing unit by being electrically connected, and receives the status signal from described imageing sensor, analog to digital converter or image signal processing unit; The data output of described image signal processing unit connects described micro-control unit.If described image signal processing unit adopts moving image displacement detector of the present invention, this type of optical mouse is also within protection scope of the present invention undoubtedly.This type of optical mouse has improved the mouse accuracy of navigation greatly, has improved the finished product rate.Integrated level based on existing chip technology and encapsulation technology is more and more higher, and described imageing sensor and described analog to digital converter can be little imagers of one, and described image signal processing unit and described micro-control unit can be integrated in the chips.

Claims (13)

1. moving image displacement detecting method, be used to analyze have time relationship image sequence to determine the gathering movement locus of this image sequence time sensing device with respect to object, comprise step:

A. image signal processing unit receives described image sequence in chronological order;

It is characterized in that: described image sequence comprises three frames or the above image of three frames; Also comprise step

B. based on described image sequence, successively each present image is carried out data processing, foundation is calculated with each pixel on the corresponding difference image of this current image, by following formula calculating and the output displacement vector (u corresponding with present image _x, u _y)

D wherein _x, D _yBe the space derivative of differential pixel, representing two components of differential pixel Grad; D _tTime-derivative for difference image; N is the pixel count in this image.

2. moving image displacement detecting method according to claim 1 is characterized in that, also comprises step:

C. simulate described movement locus according to each displacement vector among the step B and the time interval between correspondence image thereof.

3. moving image displacement detecting method according to claim 1 is characterized in that, described step B is specially and comprises step:

1. produce the difference image sequence according to described image sequence, promptly frame image data behind the frame in this image sequence is deducted after the current frame image data difference image data as present frame, setting this back frame image data again is the current frame image data ... the rest may be inferred constitutes described difference image sequence up to obtaining two frames or the difference image more than two frames;

2. utilize frame difference image behind current difference image and the one to calculate the described space derivative D of each pixel in the current difference image _x, D _yWith time-derivative D _t

3. by multiplying each other computing to obtain each middle coefficient in the described formula in twos

D _xD _y, D _xD _t, D _yD _t

4. in current difference image scope, calculate each middle coefficient in the described formula add up and; And and then calculate described displacement vector (u _x, u _y).

4. moving image displacement detecting method according to claim 1 is characterized in that, described step B is specially and comprises step:

1. two field picture calculates the space derivative I that obtains each pixel in the present image after utilizing present image and one according to described image sequence _x, I _yWith time-derivative I _t

2. the space derivative I of each pixel in the back two field picture _x, I _yWith time-derivative I _tDeduct the space derivative and the time-derivative of respective pixel in the current frame image, produce described space derivative D with each pixel of the corresponding difference image of present image _x, D _yWith time-derivative D _t

3. by multiplying each other computing to obtain each middle coefficient in the described formula in twos

D _xD _y, D _xD _t, D _yD _t

4. in current difference image scope, calculate each middle coefficient in the described formula add up and; And and then calculate described displacement vector (u _x, u _y).

5. according to claim 3 or 4 described moving image displacement detecting methods, it is characterized in that:

Described back two field picture or be the next frame image of present image in the described image sequence perhaps is following some two field pictures of present image in the described image sequence.

6. moving image displacement detecting method according to claim 1 is characterized in that:

When step B carried out data processing to each present image, processing region was the maximal correlation zone of differentiating of image-region to the maximum.

7. a moving image displacement detector is used for optical mouse; Comprise a data-interface and a data delivery outlet that is used for receiving data image signal; It is characterized in that:

Also comprise difference processing module and differentiate processing module that polyphone connects, is after unit carries out the derivative operation of the calculus of differences of interframe data and differentiate processing module from the data image signal of described data-interface with the frame through the difference processing module, the space derivative D of output and each pixel of the corresponding difference image of current frame image _x, D _yWith time-derivative D _tBe sent to the global operations module, calculate each middle coefficient of all pixels in this image preset range respectively

D _xD _y, D _xD _t, D _yD _tAdd up and; This global operations module is exported these and is added up and past displacement vector computing unit, with by formula

Come displacement calculating vector (u _x, u _y), wherein n is the quantity of the pixel in the described image preset range; This displacement vector computing unit output displacement vector (u _x, u _y) toward described data output.

8. moving image displacement detector according to claim 7 is characterized in that:

Also comprising two the frame delay device or the frame buffer storeies of polyphone mutually, connect described data-interface, is that unit separates with the frame with described data image signal; Order is away from the second frame delay device or the second frame buffer storer output current frame image data of described data-interface, then the output of the first frame delay device or the first frame buffer storer is the data of a back two field picture of this present frame, what described data-interface was exported simultaneously is the data of a back two field picture of this back two field picture, and this three frame image data is exported simultaneously toward the polyphone branch road of difference processing module and differentiate processing module.

9. moving image displacement detector according to claim 8 is characterized in that:

Described difference processing module is connected between described data-interface and the differentiate processing module, described second frame delay device/frame buffer storer and first frame delay device/frame buffer storer provide one tunnel output to be sent to described difference processing module and to carry out inter-frame difference respectively, are sent to described differentiate processing module thereby this difference processing module produces two frame difference view data simultaneously.

10. moving image displacement detector according to claim 8 is characterized in that:

Described differentiate processing module is connected between described data-interface and the difference processing module, described second frame delay device/frame buffer storer and first frame delay device/frame buffer storer provide one tunnel output to be sent to described differentiate processing module respectively, thereby this differentiate processing module produces two frame space derivative I simultaneously _x, I _yWith time-derivative I _tBe sent to described difference processing module and carry out inter-frame difference to obtain differentiated space derivative D _x, D _yWith time-derivative D _t

11. moving image displacement detector according to claim 7 is characterized in that:

Also comprise a micro-control unit, this micro-control unit has the keystroke interface that is used for receiving push button signalling; This micro-control unit receives from the data of described keystroke interface with from the displacement vector data of described displacement vector computing unit, and with these packing data outputs toward described data output.

12. optical mouse, comprise the light emitting diode that irradiates light is provided, be used for responding to irradiates light that is reflected by object plane and the imageing sensor that light signal is converted to analog picture signal, convert described analog picture signal the analog to digital converter of data image signal to, and an image signal processing unit that receives described data image signal; Comprise that also one is used for being electrically connected the micro-control unit of host computer and mousebutton, this micro-control unit fetches control described imageing sensor, analog to digital converter or image signal processing unit by being electrically connected, and receives the status signal from described imageing sensor, analog to digital converter or image signal processing unit; The data output of described image signal processing unit connects described micro-control unit; It is characterized in that: this image signal processing unit is each described moving image displacement detector in the claim 7～10.

13. optical mouse according to claim 12 is characterized in that:

Described imageing sensor and described analog to digital converter are little imager of one, or described image signal processing unit and described micro-control unit are integrated in the chips.

Images