A kind of ring field image type virtual environment image connecting system and method
The present invention relates generally to a kind of map interlinking instrument, utilizes this map interlinking instrument, can construct a ring field image type virtual environment at an easy rate.
In this real world, present this and adopt the colourful world more, image is best instrument.Past in PC World, mainly presents the natural landscape of real world with the mode of drawing, in drops at leisure the scene drawing that will show is come out with lines.And this mode need spend many manpowers and time, and needs special and expensive computer equipment to show the scene of wanting to show via computing machine in real time, though interactive high shortcoming is many.Therefore, just the someone wants directly to come the constructing virtual environment with image.This mode has many benefits, and at first, needed computer equipment does not need too expensive, even personal computer just can obtain effect very true to nature.Moreover it is irrelevant to make up a virtual world manpower of spending and the scenery that will make up, no matter how complicated background is, computing machine institute's time spent is all the same.Utilize image to make up a virtual world, at first will utilize camera, it is fixed on the turning axle, then a circle is clapped in scenery ring field, just can obtain a series of image.In order to eliminate the problem of camera lens distortion (perspective distortion), before map interlinking, must be on some mathematical models (column type or ball-type) these image projection; Utilize the map interlinking device that these figure are joined together to constitute a ring field image then; Utilize this image just can construct a virtual world of real world.
Ring field image has many application in practice, for example can be applicable in film trick, electronic game, virtual shopping or the education.Make up this ring field pattern, maximum problem is exactly how a series of figure that taken by camera to be conspired to create one.A very natural method is with manual mode these figure to be connected together, and this mode is not too convenient, especially when a lot of images will connect, wastes time and energy especially.Therefore utilizing the automatic map interlinking of computing machine is a kind of relatively convenient mode.
Before these images were connected together, in order to overcome the problem that image twists because of camera lens, the image of input must project to some mathematical models, and this mathematical model can be ball-type (spherical) or column type (cylindrical).In actual applications, when being projected in data storage, ball-type can meet some problems, because in PC World, if image is near rectangle then than being easier to storage.Another problem is the bad arrangement of camera, and the position is put inappropriately a little, very big distortion and difference are just arranged between image, and this can cause the difficulty of map interlinking.For the foregoing reasons, in making up virtual world, the most frequently used to projection pattern be face of cylinder projection (cylindrical projection).
With reference to figure 2, face of cylinder projection mathematics model can be described in detail as follows: rectangle PQRS is used for presentation video plane 10, rectangle pqrs is used for representing the cylinder plane 12 (cylindrical plane) supposed, mirror axle center is O, PQRS will be projected to pqrs, the plane of delineation 10 circumscribed cylinder planes 12, and the plane of delineation 10 central points are O ', lens length is f, and the radius on cylinder plane is d.Suppose P (x, y) be on the plane of delineation 10 a bit, and P (u, v) be P (x, the y) corresponding point on cylinder plane 12, with reference to figure 2, coordinate (u v) can be expressed as follows:
With
In fact, radius d equals f, so equation (1) and (2) can be rewritten as follows:
With
Based on equation (3) and (4), can camera take the photograph image projection to some cylinders plane, utilize map interlinking device of the present invention again, just can be connected into a complete ring field image to the image of input.
Fundamental purpose of the present invention is to provide a kind of intelligent map interlinking device, utilize this map interlinking device can take the photograph by camera a series of images utmost point be connected together effectively, thereby constitute a complete ring field image.The present invention mainly is an information of utilizing image limit (edges), and a series of different images are connect.The present invention also proposes a kind of method, it utilizes the information on image limit, find out a series of possible homographic solutions (matching solutions), again by these possible separating, utilize relevant matches (correlation matching) to determine separating of last the best, utilize this optimum solution, just can be joined together adjacent different image.Because the number of this group feasible solution seldom, can find out separating at last with utmost point effective and efficient manner.
In the present invention, when two images are bonded together, at first need they are projected on some faces of cylinder, so-called distortion that Here it is (warping) computing, then from these two images that twisted, seek the information on limits (edges) individually, and the location records on these limits is got off.Utilize the position on these limits, can obtain one group of possible corresponding point and separate.Then from this group is separated, can effectively utilize very much relevant matches and find finally and separate.When find finally separate after, because the brightness of these two images is inconsistent, must do so-called mixing (blending) computing, promptly utilize on the image any brightness and the brightness of its corresponding point on another image, be weighted on average according to their distances from the border, thereby after two images become one, seem very level and smooth in the brightness of seam crossing.
Characteristics of the present invention are, when detecting the vertical edges (vertical edges) of an image, can utilize the summation of each row (column) pixels (pixels) Grad (gradient intensty value) in the image to determine, promptly each pixel of each row in the image be calculated its Grad respectively; In addition summation is average to all Grad of pixel of this row then, and this mean value is compared with a pre-set threshold again; If this mean value is bigger than threshold value, then this shows the existence of a vertical edges; Each row is all done this computing, just can find out all vertical edges.
Similarly, when seeking the horizontal sides (horizontal edges) of an image, can utilize the summation of each row (row) pixel gradient value of image to decide, promptly each pixel of each row in the image be calculated its Grad respectively; In addition summation is average to all Grad of pixel of this delegation then, and this mean value is compared with a pre-set threshold again; If this mean value is bigger than threshold value, then this delegation can be considered the existence that a horizontal sides is arranged; This computing is done in each provisional capital, just can find out all horizontal sides.
Utilize vertical edges, can infer needed homographic solution (matching solutions) possible coordinate on the x axle; Usually, utilize so-called displacement function (offset function) to produce possible the separating of this group.It is defined as follows:
Wherein: P
a(i) be illustrated in the position that a opens i bar vertical edges in the warp image (warped image);
P
b(j) be illustrated in the position that b opens j bar vertical edges in the warp image;
N
bBe illustrated in the number that b opens vertical edges in the warp image; With
K is a variable of user's definition.Suppose N
p(i, k) value is less than a predetermined threshold value (threshold) T to represent all to open its displacement function d in the warp image at a
0The limit number of vertical edges, (i, k) value is less than a predetermined threshold value T by these its d
0Vertical edges, can obtain d (i, a mean value A (k) k).If A (k) value is less than a predetermined threshold (threshold) T
1, and N
pGreater than another predetermined threshold value (threshold) T
2, the k position just can be considered the coordinate of a possible homographic solution on the x direction so.Provide different value of K, just can find out the coordinate of all possible homographic solution on the x direction.
According to a further aspect in the invention, the invention provides a kind of being used for and effectively the image of input is connected into a holonomic system of encircling field pattern, this system comprises an edge detector (edge detector), be used for detecting a and open the horizontal or vertical limit of image and open horizontal or vertical limit corresponding in the image at b, this edge detector writes down the position of these vertical edges or horizontal sides simultaneously.This system also comprises a storer, is used for storing the position data on these horizontal or vertical limits, so that further handle.This system comprises a feasible solution generator (hypothesis generator) and optimum solution selector switch (optimum hypothesisselector) simultaneously, the position that the feasible solution generator is mainly used to the horizontal or vertical limit from be stored in storer produces a series of possible homographic solutions, and the optimum solution selector switch then is used for utilizing relevant matches to select best separating from possible the separating of this group.
Specifically, this system utilizes storer to come recording image data, and this system comprises a mixing engine (blending engine), is used for the difference of brightness between smoothed image.When two images are joined together, common overlapping region (common overlapping area) must be arranged.Because the difference of photograph situation, the inevitable difference of overlapping region brightness that these two images are common, can take out some pixels from an image, simultaneously take out its corresponding pixel at another image, be weighted on average according to their distances, thereby make two images very level and smooth in the brightness of common overlapping areas from the border.In this system, edge detector (edge detector), feasible solution generator (hypothesis selector), optimum solution selector switch (optimalhypothesis selector) all can be included in the microprocessor.In addition, can utilize outside storer (as hard disk) to write down the image (warped images) that twisted.
The present invention also provides a kind of prediction will be by the method for the feasible solution of map interlinking.It utilizes the information on the horizontal or vertical limit in the image to predict possible the separating of this group basically.Because infer that accurately the number that this group is separated is very little, therefore the scope of searching is reduced widely, thereby can find out finally very soon and separate.When searching is finally separated, utilize so-called relevant matches finally to be separated.Because the relevant matches spended time is more, the invention provides a method of cutting out (pruningmethod) that is used for removing in advance unnecessary relevant matches.The principle of this method of cutting out is if the value of current relevant matches less than the value of a predefined, then need not further be mated, therefore can remove some unnecessary couplings in advance.This method of cutting out can reduce the calculating of a lot of relevant matches, has therefore improved the efficient of map interlinking greatly.Compare with traditional method, the present invention has not only had tangible improvement aspect efficient but also aspect reliability, and this mainly is because can predict possible homographic solution in advance, owing to infer that set accurate and that this group is separated is very little, can find out optimum solution soon.
The present invention may be used on many aspects, for example virtual environment, guide system, electronic game, Geographic Information System, film trick etc.
By reference accompanying drawing detailed description of the present invention, purpose of the present invention, feature and advantage will become more obvious, in the accompanying drawing:
Fig. 1 represents to utilize image to produce the systematic analysis figure of virtual environment;
Fig. 2 is the geometric graph that is used for illustrating conic projection;
Fig. 3 is the system schematic that realizes method of the present invention;
Fig. 4 A and 4B illustrate the operation of Fig. 3 system with an example;
Fig. 5 is used for the how brightness between level and smooth two images of (blending) technology of explain mixing; With
Computer organization that realizes Fig. 3 method of Fig. 6.
Fig. 3 represents whole image connecting system 20, and its cardinal principle is to utilize the information on limit (edgeinformation) to determine last optimum solution.At first, utilize edge detector 22 from image, to obtain the position of vertical edges and horizontal sides.Obtain after these positions, utilize feasible solution generator 24 to obtain a series of feasible solutions.Then, utilize optimum solution selector switch 26 that this group each in feasible solution is separated and do last affirmation, separate thereby find out at last effectively.Utilize this finally to separate then, individually adjacent image is bonded together.In order to eliminate the uncontinuity of brightness between individual image, can utilize and mix the problem that engine 28 solves this brightness uncontinuity.
In this image connecting system, edge detector only uses vertical edges and horizontal sides to predict homographic solution possible between the adjacent image in advance.Suppose g
x(p) (its clear and definite expression is g for i, the j) Grad on the x direction to represent some pixel p in the image
x(p (i, j))=| and I (p (i+1, j))-I (p (i-1, j)) |, wherein I (p) is the brightness value of pixel p in image.When the detection of vertical limit, edge detector is to their Grad of pixel accumulation at same row.There is a vertical edges in these row if this accumulated value greater than the threshold value of a predefined, is then assert, and utilizes mark to write down the existence of this vertical edges.After having detected all row, can obtain the position of all vertical edges.Available identical method is sought the position of horizontal sides.
With reference to figure 4A and Fig. 4 B, giving one example illustrates.Suppose other image I
aAnd I
bBe bonded together, can utilize rim detection from I
aIn obtain following vertical edges: (100, y), (115, y), (180, y), (200, y), (310, y), (325, y), (360, y), (390, y) and (470, y), also can be from I
bIn obtain following vertical edges: (20, y), (35, y), (100, y), (120, y), (230, y), (245, y), (280, y), (310, y) and (390, y).By these vertical edges, can only utilize their coordinates on the x direction to obtain two groups of coordinate figures, i.e. P
a=(100,115,180,200,310,325,360,390,470) and P
b=(20,35,100,120,230,245,280,310,390).Because these vertical edges correspond to each other, P
aAnd P
bSatisfy following relationship: P
a(i)=P
b(j)+d
x, wherein i is corresponding one by one with j, d
xBe P
aAnd P
bBetween deviation (offset).Clearly, d
xBe the homographic solutions (matchedsolution) of these two adjacent images on the x direction.But, may can not find or confuse under certain conditions, at I owing to there is noise
aAnd I
bSome vertical edges, this causes P
aAnd P
bBetween no longer be to concern one to one.Therefore in the present invention, propose a kind of feasible solution generator, be used for finding out possible homographic solution on the x direction from the information of these vertical edges.
Suppose N
aAnd N
bWrite down P respectively
aAnd P
bIn element number, and make displacement function
A given value k, then the feasible solution generator calculates at P
a(i is k) less than a prior predetermined threshold value T for its d in the set
0Element number.Suppose that this number is N
p, suppose also that in addition (i, mean value k) are A (k), if A (k) is less than second prior predetermined threshold value T for the d of the element that these satisfy condition
1, and N
pGreater than the 3rd threshold value T
2, then this position k is regarded as the coordinate figure of a possibility homographic solution on directions X.Different k is provided, just has one group may produce by homographic solution.Notice that this feasible solution is the possible coordinate figure on the x direction just, the set that makes this feasible solution is S
xIn this solution procedure, three pre-set threshold T
0, T
1, T
2By the experiment gained, and all the same to all images.
Also can be in a like fashion on horizontal sides.Utilize horizontal sides, can obtain the coordinate of possible homographic solution on the y direction.Suppose that these possible y coordinates constitute a S set
xBecause S
xAnd S
yThe element of this set is few, finally separates to be obtained soon by the optimum solution selector switch.
The optimum solution selector switch is mainly used to from S
xAnd S
yTwo union of sets collection { S=(x, y) | x ∈ S
x, y ∈ S
yIn determine optimum solution.The optimum solution selector switch mainly utilizes relevant matches to determine optimum solution.In the method for relevant matches, relatively Chang Yong evaluation function (measure function) has two, one is mean absolute difference MAE (mean absolute error), and another is canonical simple crosscorrelation (normalizedcross-correlation), can be with reference to the paper of L.G.Broum.Their definition is as follows respectively:
With
U wherein
iWith σ i be image I
iMean value and variable quantity.(2M+1)
2The area of expression coupling form.Though these two evaluation functions all are applicable to definite optimum solution, but when speed is not most important reference quantity, canonical simple crosscorrelation coupling is a reasonable selection, because it can tolerate the difference of brightness, but, if speed is a very important reference quantity, then the mean absolute error evaluation function is a reasonable selection.
Usually, the time that calculating mean absolute error (MAE) is spent is a lot, therefore, in the present invention, proposes a kind of tailoring technique (pruning technique), is used for quickening the calculating of MAE.At first utilize a matrix (matrix) to be recorded in the temporary transient value of accumulation MAE when calculating previous MAE, regard threshold value with this matrix.If present MAE value, then there is no need further accumulation less than its pairing threshold value (being present in the matrix) and calculates the MAE value.Adopt this tailoring technique, just can avoid many unnecessary correlation calculations, therefore obviously improved the efficient of coupling, also can calculate optimum solution soon.
In case find after the optimum solution, just can two adjacent images be joined together.Two adjacent images have the inconsistent situation of brightness, therefore, must utilize so-called hybrid technology (blendingtechnique) to eliminate the inconsistent problem of brightness.With reference to Fig. 5, suppose P
iFor in image I
aOn a bit, q
iBe p
iAt I
bOn corresponding point, l
aAnd l
bBe I
aAnd I
bIndivedual edge lines, d
aBe p
iTo l
aDistance, d
bBe q
iTo l
bDistance, if r
iBe p
iAnd q
iCorresponding point on composograph I are then utilized p
iAnd q
i, can utilize following formula to obtain r
iBrightness:
Wherein t represents index, and it is an adjustable parameter.Utilize formula (7), just can get synthetic to the end image.
Fig. 6 is a hardware configuration of the present invention, utilizes this structure to be connected together the image of input in real time.Input equipment 32 can be digital camera or scanner, can obtain a series of image by input equipment.These images can be stored in External memory equipment 32 as hard disk or directly deliver on the random access memory of microprocessor (RAM), and microprocessor comprises warp image (warping images), rim detection (edgedetection), produces possible separating functions such as (hypothesis generation).The data that temporarily obtain can be put on the RAM of microprocessor, to improve the speed of map interlinking.In case the work of map interlinking is finished, just can directly be shown to these results on the screen, these functions can be finished with general personal computer.
More than illustrated in and the explanation be a preferred embodiment of the present invention.Scope that is covered according to the present invention and spirit, the present invention can also do multiple variation.