CN102175221B - Vehicle-mounted mobile photographic surveying system based on fisheye lens - Google Patents

Abstract

The invention discloses a vehicle-mounted mobile photographic surveying system based on a fisheye lens, which is characterized in large surveying range and rich and integral image color information. The technical scheme is as follows: the system comprises a data acquisition device, a data storage device and a data processing device, wherein the data acquisition device acquires images and data, and a fisheye camera stereoscopic vision module of the data acquisition device comprises the fisheye lens, a camera and a calibration device; the data storage device stores the acquired image data; and the data processing device processes the acquired data to implement the surveying on the fisheye images.

Description

Based on fish-eye vehicle-mounted mobile Digital Photogrammetric System

Technical field

The present invention relates to a kind of vehicle-mounted mobile Digital Photogrammetric System, relate in particular to and utilize fish eye lens Quick Acquisition image data, by stereoscopic vision, proofread and correct, obtained image data is carried out to the system of measurement in real time and three-dimensional modeling.

Background technology

Quickening along with urbanization process, city size is increasing, change with each passing day, urban planning and management becomes and becomes increasingly complex, for every field such as traffic, the energy, municipal administration, infrastructure constructions, it is particularly important that the collection of information and renewal seem, particularly has the measurement with GIS information, becomes these realm informations and obtain an indispensable part.Such as, city management department gathers and all kinds of asset datas in statistics city, as the information of building in region, and the data such as billboard position, quantity, length and width height; Road traffic department statistics road asset data; The information such as the position of statistics transformer station of energy sector, quantity.These data have data volume large conventionally, add up complicated feature.If use the mode of artificial field survey statistics, workload is large, and efficiency is low, dangerous high, and easily makes mistakes.

The measuring system that existing market exists comprises laser measurement and stereoscopic vision measurement.Laser measurement, does not have the color information of image conventionally, and resolution is low, and data volume is large, and acquisition range is limited, aftertreatment complicated and time consumption.And existing stereoscopic vision measuring equipment, cameras based on common all, field range is little, in urban environment, high buildings and large mansions are in a compact mass, such as the high buildings and large mansions of taking in city, when distance is very near, can only photograph the part in building, if photograph whole building, need distance objective far, affect like this imaging precision of target object, if do the measurement based on image, can have influence on the precision of measurement, cause very large error.Therefore traditional camera Stereo Vision Measurement System is subject to very big challenge.

Along with the development of photographic goods, there is a kind of bugeye lens with super-layer visual field scope, fish eye lens can be realized field range and meet or exceed 180 degree in individual fish eye images.Be the high buildings and large mansions of taking in city equally, fish eye lens can guarantee field range and the precision of target object simultaneously, can solve well on a large scale shooting with great visual angle and the problem of measurement.

But fish eye lens, when obtaining very large field range, has also brought very large distortion to image, and in general, field range is larger, distorts also larger.Traditional stereoscopic vision measurement is that the general camera based on small distortion is measured, and cannot measure with the fisheye camera of large distortion.Concerning the camera of outfit common lens, its image-forming principle is similar to usually said pin-hole imaging.In pin-hole imaging, light is propagated along straight line (linearity).To be that image deformation is very little even can ignore its advantage, but its shortcoming also clearly, and field range (FOV) is very little.For the large scene especially unlimited scene of 180 degree, pass through the common lens imaging on sensor devices based on light rectilinear propagation, necessarily require sensor devices infinitely great, this is obviously impossible realize in practice.So direction of propagation of the light that only changes, allow it enter after camera lens no longer along rectilinear propagation, but project on sensor devices along curve (non-linear), so just can on limited sensor devices, hold unlimited scene, fish eye lens is exactly to do like this.But the curve of light is propagated several problems of also bringing simultaneously: first, because light is propagated along curve, this gives and determines that its travel path brings very large difficulty.Secondly, although be all to propagate along curve, the curvature of its travel path of light within the scope of different visual angles is different, and visual angle is larger, and curvature is larger.Showing in image, is exactly less the closer to optical center pattern distortion, larger the closer to lens edge pattern distortion.Determine that a unified propagation model just becomes more difficult therefore to the light in all angulars field of view.Again, because the distortion of lens edge is too large, the content that the several pixels on image border comprise is equivalent to even up to a hundred the contents that pixel comprises of picture centre place dozens of.Therefore on image border, the deviation of a pixel is equivalent to the deviation of tens of up to a hundred the pixels in picture centre place, and this has brought very large challenge to accuracy of model.In sum, fish eye lens is only applied to the simple occasions of information with great visual angle that need such as safety monitoring at present, is but rarely used in the occasions high to accuracy requirement such as measurement or three-dimensional modeling.

By fisheye camera being carried out to accurate mathematical modeling, obtain the accurately image parameter of fisheye camera (travel path of light), obtain further the parameter that fisheye camera stereoscopic vision is right (rotation between fisheye camera and translation), just can use the fisheye camera of large distortion to realize stereoscopic vision measurement.This imaging parameters and fisheye camera stereoscopic vision of obtaining fisheye camera is referred to as to the process of parameter the demarcation that fisheye camera stereoscopic vision is right.The existing very ripe solution of demarcation of general camera, such as the method for Zhang Zhengyou, Tsai.Yet because image-forming principle is different, these methods can not, for demarcating fish eye lens, must find new mathematical model to describe fish-eye imaging process.Recently occur that a kind of technology based on fitting of a polynomial has solved this problem preferably.

If can address the above problem, a measuring system solution based on fish-eye stereoscopic vision is provided, can solve existing measuring system field range little, cannot when measuring, obtain the problems such as comprehensive chromatic image, by data acquisition and measurement for all departments such as city management, the energy, statistics, explorations, provide a kind of brand-new and easy integrated solution, for the digital Construction in city provides new solution.

Summary of the invention

The object of the invention is to address the above problem, provide a kind of and can overcome above-mentioned deficiency based on fish-eye vehicle-mounted mobile Digital Photogrammetric System, reach that measurement range is large, image color information enriches complete feature.

Technical scheme of the present invention is: the present invention has disclosed and a kind ofly based on fish-eye vehicle-mounted mobile Digital Photogrammetric System, comprised data collector, data storage device and data processing equipment, wherein:

Data collector, for gathering image and data, it comprises fisheye camera stereoscopic vision module, for gathering fish eye images information, fisheye camera stereoscopic vision module comprises fish eye lens, camera and caliberating device, wherein:

Fish eye lens, is connected with camera, obtains flake image;

Camera, is connected with fish eye lens, and the flake image that reception fish eye lens obtains also carries out image collection;

Caliberating device, obtains fish-eye imaging parameters and fish eye lens stereoscopic vision to parameter;

Data storage device, couples data collector, the image data arriving for storage of collected;

Data processing equipment, couples data storage device, for the treatment of collected data, realizes the measurement to flake image.

According to the embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention, data collector also comprises:

Vehicle pose acquisition module, for obtaining position, the attitude of vehicle, the information of range ability;

The information of the position of the data storage device vehicle that also store car pose acquisition module obtains, attitude, range ability.

According to the embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention, data collector also comprises:

Panorama acquisition module, for gathering continuous road full-view image.

According to the embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention, fisheye camera stereoscopic vision module at least comprises two fisheye cameras.

According to the embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention, caliberating device comprises:

Flake imaging relations is set up module, sets up half unit Sphere Measurement Model, and on unit sphere model, sets up flake imaging relations;

Initialization internal reference module, couples flake imaging relations and sets up module, initialization internal reference, and wherein internal reference is the parameter of fisheye camera self, irrelevant with external environment condition;

Homography matrix computing module, couples initialization internal reference module, calculates homography matrix;

The outer moduli piece of initialization, couples homography matrix computing module, and initialization is joined outward, and it is joined is at home and abroad the parameter between fisheye camera and external environment condition;

Iteration optimization module, couples the outer moduli piece of initialization, and LM iteration minimizes re-projection error, and the interior participation after being optimized is joined outward.

According to the embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention, it is as follows that flake imaging relations is set up the processing of module:

In note space, the imaging point of certain 1 x on fish eye images is (u, v), and the incident angle that spatial point x points to the unit ball centre of sphere is

wherein θ is the angle of incident ray and unit ball Z axis positive dirction,

the projection of incident ray in unit ball XY plane and the angle of unit ball X-axis positive dirction, by the incident angle of incident ray flake imaging model to the imaging point (u, v) on fish eye images is described by following equation:

r(θ)＝k ₁θ+k ₂θ ²+k ₃θ ³+k ₄θ ⁴+k ₅θ ⁵+...k _nθ ⁿ (1)

Certain pixel on r presentation video is to the distance of figure principal point, k ₁... k _nit is fish-eye imaging parameters;

Δ _rrepresent fish-eye radial distortion, l ₁... l _n, i ₁... i ₄for radial distortion parameter;

Δ _trepresent fish-eye tangential distortion, m ₁... m _n, j ₁... j ₄for tangential distortion parameter;

X _dfor the position vector of pixel, i.e. (x _d, y _d), u _rfor vector of unit length radially,

for tangential vector of unit length;

$\left(\begin{array}{c}u\\ v\end{array}\right)=\left(\begin{array}{cc}{m}_u& 0\\ 0& m_v\end{array}\right)\left(\begin{array}{c}{x}_d\\ y_d\end{array}\right)+\left(\begin{array}{c}{u}_0\\ v_0\end{array}\right)---\left(5\right)$

(u ₀, v ₀) be the principal point coordinate of image, (m _u, m _v) be respectively the pixel count in unit distance in CCD level and vertical direction, (k ₁, k ₂, k ₃, k ₄, k ₅, l ₁, l ₂, l ₃, i ₁, i ₂, i ₃, i ₄, m ₁, m ₂, m ₃, j ₁, j ₂, j ₃, j ₄, m _u, m _v, u ₀, v ₀) be fish-eye parameter to be calibrated.

According to the embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention, the processing of initialization internal reference module is as follows:

Read lens parameters, lens parameters comprises focal distance f and maximum view angle theta _max;

Make k ₁=f, r _max=f θ _max;

Detect fish eye images border, frontier point carried out to ellipse fitting:

try to achieve u ₀, v ₀, a, b,

wherein (a, b) is oval length semiaxis, (u ₀, v ₀) be the oval center of circle;

Other parameter is set to 0.

According to the embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention, homography matrix computing module comprises:

Gridiron pattern image fetching unit, reads the cross-hatch pattern picture on scaling board;

Unit is chosen in point of crossing, couples gridiron pattern image fetching unit, chooses successively 4 point of crossing at place, gridiron pattern summit on every cross-hatch pattern picture;

Back projection unit, couples point of crossing and chooses unit, utilizes initialization internal reference, by point of crossing

back projection obtains vector of unit length to unit ball

wherein j is j width image, and i is i gridiron pattern point of crossing;

Homography matrix estimation unit, couples back projection unit, and estimate sheet is answered matrix H _j, by vector of unit length be expressed as

spatial point x on vector of unit length and scaling board ⁱbetween there is homograph H _j, by linear algorithm, estimate homograph H _j, obtain the spatial point x on scaling board ⁱat homograph H _junder corresponding point:

x wherein _p ⁱit is the volume coordinate of i point of crossing on j width cross-hatch pattern picture;

Homography matrix is optimized unit, couples homography matrix estimation unit, by LM iteration minimum error function

to optimize homography matrix H _j, wherein it is vector

with

between angle;

Point of crossing map unit, couples homography matrix and optimizes unit, and the homography matrix H after optimization is passed through in all point of crossing on scaling board _jbe mapped to and on unit ball, obtain corresponding point:

Point of crossing image coordinate acquiring unit, couples point of crossing map unit, and vector of unit length is transformed on image:

at subpoint

the image coordinate of neighbor searching point of crossing

According to the embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention, homography matrix computing module comprises:

Gridiron pattern image fetching unit, reads the cross-hatch pattern picture on scaling board;

Unit is chosen in point of crossing, couples gridiron pattern image fetching unit, chooses successively all gridiron pattern point of crossing on every cross-hatch pattern picture;

Back projection unit, couples point of crossing and chooses unit, utilizes initialization internal reference, by point of crossing

back projection obtains vector of unit length to unit ball

wherein j is j width image, and i is i gridiron pattern point of crossing;

Homography matrix estimation unit, couples back projection unit, and estimate sheet is answered matrix H _j, by vector of unit length

be expressed as

spatial point x on vector of unit length and scaling board ⁱbetween there is homograph H _j, by linear algorithm, estimate homograph H _j, obtain the spatial point x on scaling board ⁱat homograph H _junder corresponding point: x wherein _p ⁱit is the volume coordinate of i point of crossing on j width cross-hatch pattern picture;

Homography matrix is optimized unit, couples homography matrix estimation unit, by LM iteration minimum error function

to optimize homography matrix H _j, wherein

it is vector

with

between angle.

According to the embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention, the processing of the outer moduli piece of initialization is as follows:

Outer ginseng

by homography matrix H _jinitialization is as follows:

$r_j^1={\mathrm{\λ}}_j{h}_j^1,$ $r_j^2={\mathrm{\λ}}_j{h}_j^2,$ $r_j^3=r_j^1\×r_j^2,$ $t_j={\mathrm{\λ}}_j{h}_j^3$

Wherein,

r _jfor rotation parameter, T _jfor displacement parameter,

be j homography matrix H _ji column vector.

According to the embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention, the processing of iteration optimization module is as follows:

LM iteration minimizes re-projection error

internal reference after being optimized and outer ginseng, wherein

for picture point

between pixel distance, M is the point of crossing quantity on every width cross-hatch pattern picture, N is gridiron pattern amount of images.

The present invention contrasts prior art following beneficial effect: principal feature of the present invention be by fisheye camera stereoscopic vision module application in vehicle-mounted measuring system, utilize the feature that fish eye lens field range is large, realize on a large scale and measuring.By vehicle pose acquiring unit, can obtain position and the range information of gathered image, comprise the distance of longitude and latitude, attitude and vehicle operating etc., add panorama acquisition module, obtain the full-view image data with rich colors information simultaneously.Thereby it is little to have solved existing measuring system measurement range, the problem of the few or INFORMATION OF INCOMPLETE of image color, can allow the needed information of user's quick obtaining, especially for urban environment, efficiency and the precision measured have greatly been promoted, for data statistics and renewal, particularly urban construction, planning, exploration provide simpler and easy and comprehensive method.

Accompanying drawing explanation

The schematic diagram that shows the first embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention that Fig. 1 is exemplary.

The schematic diagram that shows the second embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention that Fig. 2 is exemplary.

The schematic diagram that shows the 3rd embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention that Fig. 3 is exemplary.

The schematic diagram of the embodiment that shows caliberating device of the present invention that Fig. 4 is exemplary.

The refinement schematic diagram of a kind of example that shows the homography matrix computing module in caliberating device of the present invention that Fig. 5 is exemplary.

The refinement schematic diagram of the another kind of example that shows the homography matrix computing module in caliberating device of the present invention that Fig. 6 is exemplary.

The schematic diagram that shows half unit Sphere Measurement Model of the present invention that Fig. 7 is exemplary.

Embodiment

Below in conjunction with drawings and Examples, the invention will be further described.

Before introducing each embodiment of the present invention, also to do an explanation to fish eye images (image): flake image refers to by fish eye lens and converges the image that light is transmitted on photosensitive imaging element and storage obtains, owing to obtaining by fish eye lens, claim that this class photo is flake image.

Before introducing each embodiment of the present invention, also to do an explanation to fisheye camera: camera is connected with fish eye lens, and the image energy generating obtains the field range of super wide-angle, claims that this class camera is fisheye camera.

Before introducing each embodiment of the present invention, first panorama or panoramic picture (image) are done to an explanation: if image is placed on to a spherical space, or cubic space, or cylindrical space, or cone space, or in ellipsoid space, by fixing a bit as observation point in above-mentioned space, the process of the image obtaining by this this image of observation point employing one-point perspective is panorama and plays, this image is by the spherical panorama that is called of correspondence, or cube panorama, or cylindricality panorama, or taper panorama, or ellipsoid panorama, and being referred to as this image is panorama or panoramic picture (image).

the first embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System

Fig. 1 shows the first embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention.Refer to Fig. 1.The present embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System, comprise data collector 10a, data storage device 20a, data processing unit device 30a.

Annexation between these three devices is: the output terminal connection data memory storage 20a of data collector 10a, the output terminal of the data storage device 20a 30a that connects data processing apparatus.

Data collector 10a comprises fisheye camera stereoscopic vision module 102a, by least one stereoscopic vision to forming, i.e. two fisheye cameras.Flake stereoscopic camera stereoscopic vision module 102a comprises fish eye lens 1022a, CCD camera 1024a and caliberating device 1026a.It is upper that fish eye lens 1022a is connected to CCD camera 1024a, and composition can generate the flake image of super wide-angle view.Caliberating device 1026a is for obtaining the imaging parameters of fisheye camera and fisheye camera stereoscopic vision to parameter.

Caliberating device 1026a adopts the technology based on fitting of a polynomial to set up mathematical model and describe fish-eye imaging process.

As shown in Figure 4, flake imaging relations is set up module 40, initialization internal reference module 50, homography matrix computing module 60, the outer moduli piece 70 of initialization, iteration optimization module 80 to the refinement principle of caliberating device 1026a.

Annexation between these modules is: the output terminal that flake imaging relations is set up module 40 couples initialization internal reference module 50, the output terminal of initialization internal reference module 50 connects homography matrix computing module 60, the output terminal of homography matrix computing module 60 couples the outer moduli piece 70 of initialization, and the output terminal of the outer moduli piece 70 of initialization couples iteration optimization module 80.

It is below the performance of wherein a kind of example of caliberating device 1026a.

Flake imaging relations is set up module 40 for setting up half unit Sphere Measurement Model, and on unit sphere model, sets up flake imaging relations.

The signal of half unit Sphere Measurement Model as shown in Figure 7, is remembered 1 x in space, and its imaging point on fish eye images is (u, v), and its incident angle that points to the incident ray of the unit ball centre of sphere is

wherein θ is the angle of incident ray and unit ball Z axis positive dirction,

the projection of incident ray in unit ball XY plane and the angle of unit ball X-axis positive dirction, by the incident angle of incident ray

flake imaging model to the imaging point (u, v) on fish eye images is described by following equation:

r(θ)＝k ₁θ+k ₂θ ²+k ₃θ ³+k ₄θ ⁴+k ₅θ ⁵+...k _nθ ⁿ (1)

Certain pixel on r presentation video is to the distance of figure principal point, k ₁... k _nit is fish-eye imaging parameters.

Δ _rrepresent fish-eye radial distortion, l ₁... l _n, i ₁... i ₄for radial distortion parameter.

Δ _trepresent fish-eye tangential distortion, m ₁... m _n, j ₁... j ₄for tangential distortion parameter.

X _dfor the position vector of pixel, i.e. (x _d, y _d), u _rfor vector of unit length radially,

for tangential vector of unit length.

$\left(\begin{array}{c}u\\ v\end{array}\right)=\left(\begin{array}{cc}{m}_u& 0\\ 0& m_v\end{array}\right)\left(\begin{array}{c}{x}_d\\ y_d\end{array}\right)+\left(\begin{array}{c}{u}_0\\ v_0\end{array}\right)---\left(5\right)$

(u ₀, v ₀) be the principal point coordinate of image, (m _u, m _v) be respectively the pixel count in unit distance in CCD level and vertical direction.

Wherein Given information is the incident angle of spatial point x

and the corresponding point (u, v) of spatial point x on image, all the other parameter (k ₁, k ₂, k ₃, k ₄, k ₅, l ₁, l ₂, l ₃, i ₁, i ₂, i ₃, i ₄, m ₁, m ₂, m ₃, j ₁, j ₂, j ₃, j ₄, m _u, m _v, u ₀, v ₀) be fish eye lens parameter to be calibrated.It is to be noted, polynomial order in formula (1), (2), (3) can arrive infinite time, in experiment, find, polynomial expression order is got 5 times in formula (1), in formula (2), formula (3) polynomial expression order get 3 times can Accurate Model panorama picture of fisheye lens process.

Initialization internal reference module 50 is for initialization internal reference.

The concrete processing of initialization internal reference is as follows.Read the lens parameters that manufacturer provides: focal distance f and maximum view angle theta _maxmake k ₁=f, r _max=f θ _max.Detect fish eye images border, frontier point carried out to ellipse fitting:

${\left(\frac{u-u_0}{a}\right)}^2+{\left(\frac{v-v_0}{b}\right)}^2=1.$

Then according to ellipse fitting, try to achieve u ₀, v ₀, a, b,

wherein (a, b) is oval length semiaxis, (u ₀, v ₀) be the oval center of circle.And other parameter is set to 0.

Internal reference refers to the parameter of fisheye camera self, and irrelevant with external environment condition, in the present embodiment, the internal reference of indication is: (k ₁, k ₂, k ₃, k ₄, k ₅, l ₁, l ₂, l ₃, i ₁, i ₂, i ₃, i ₄, m ₁, m ₂, m ₃, j ₁, j ₂, j ₃, j ₄, m _u, m _v, u ₀, v ₀) it is to be noted, polynomial order in formula (1), (2), (3) can arrive infinite time, in experiment, find, polynomial expression order is got 5 times in formula (1), in formula (2), formula (3) polynomial expression order get 3 times can Accurate Model panorama picture of fisheye lens process.

Homography matrix computing module 60 is for calculating homography matrix.

Calculate the principle of homography matrix as shown in Figure 5, comprise that gridiron pattern image fetching unit 600, point of crossing choose unit 601, back projection unit 602, homography matrix estimation unit 603, homography matrix and optimize unit 604, point of crossing map unit 605 and point of crossing image coordinate acquiring unit 606.

Annexation between these unit is: the output terminal of gridiron pattern image fetching unit 600 couples point of crossing and chooses unit 601, the output terminal that unit 601 is chosen in point of crossing couples back projection unit 602, the output terminal of back projection unit 602 couples homography matrix estimation unit 603, the output terminal of homography matrix estimation unit 603 couples homography matrix and optimizes unit 604, the output terminal that homography matrix is optimized unit 604 couples point of crossing map unit 605, and the output terminal of point of crossing map unit 605 couples point of crossing image coordinate acquiring unit 606.

Gridiron pattern image fetching unit 600 reads cross-hatch pattern picture.

4 point of crossing that place, gridiron pattern summit is chosen successively in unit 601 on every cross-hatch pattern picture are chosen in point of crossing.

Back projection unit 602 utilizes initialization internal reference, by point of crossing

back projection obtains vector of unit length to unit ball

process is as follows:

$\left(\begin{array}{c}{x}_j^i\\ y_j^i\end{array}\right)=\left(\begin{array}{cc}\frac{1}{m_u}& 0\\ 0& \frac{1}{m_v}\end{array}\right)\left(\begin{array}{c}{u}_j^i-u_0\\ v_j^i-v_0\end{array}\right)$

$r_j^i=\sqrt{{\left(x_j^i\right)}^2+{\left(y_j^i\right)}^2}$

$k_1{\mathrm{\θ}}_j^i+k_2{\left({\mathrm{\θ}}_j^i\right)}^2+k_3{\left({\mathrm{\θ}}_j^i\right)}^3+k_4{\left({\mathrm{\θ}}_j^i\right)}^4+k_5{\left({\mathrm{\θ}}_j^i\right)}^5-r_j^i=0$

Wherein j is j width image, and i is i gridiron pattern point of crossing.

Homography matrix estimation unit 603 is estimated homography matrix.Vector of unit length

can be expressed as: spatial point x on itself and scaling board ⁱbetween there is homograph H _j, by linear algorithm, estimate this homograph H _j, obtain spatial point x ⁱat homograph H _junder corresponding point:

${\hat{x}}_j^i=H_j{x}_p^i/\left|\right|H_j{x}_p^i\left|\right|.$

Homography matrix is optimized unit 604 by LM iteration minimum error function to optimize homography matrix H _j, wherein

it is vector

with

between angle.

Point of crossing map unit 605 is passed through the homography matrix H after optimization by all point of crossing on scaling board _jbe mapped to and on unit ball, obtain corresponding point:

x wherein _p ⁱit is the volume coordinate of i point of crossing on j width cross-hatch pattern picture.

Point of crossing image coordinate acquiring unit 606 transforms to vector of unit length on image: at subpoint the image coordinate of neighbor searching point of crossing

The running that repeats back projection unit 602, homography matrix estimation unit 603 and homography matrix optimization unit 604 just can obtain the H of homography matrix more accurately estimating based on all point of crossing _j.

Outer moduli piece 70 initialization of initialization are joined outward.

Outer ginseng can be by homography matrix H _jinitialization is as follows:

$r_j^1={\mathrm{\λ}}_j{h}_j^1,$ $r_j^2={\mathrm{\λ}}_j{h}_j^2,$ $r_j^3=r_j^1\×r_j^2,$ $t_j={\mathrm{\λ}}_j{h}_j^3$

Wherein,

r _jfor rotation parameter, T _jfor displacement parameter,

be j homography matrix H _ji column vector.

Outer ginseng refers to the parameter between fisheye camera and external environment condition, refers to the parameters R between fisheye camera and scaling board in the present embodiment _jand T _j.

Iteration optimization module 80 minimizes re-projection error for LM iteration

interior participation after being optimized is joined outward.Wherein

for picture point

between pixel distance, M is the point of crossing quantity on every width cross-hatch pattern picture, N is gridiron pattern amount of images.

It is below the performance of the wherein another kind of example of caliberating device 1026a.

The flake imaging relations of the caliberating device of this kind of example is set up module 40 for setting up half unit Sphere Measurement Model, and on unit sphere model, sets up flake imaging relations.

The signal of half unit Sphere Measurement Model as shown in Figure 7, is remembered 1 x in space, and its imaging point on fish eye images is (u, v), and its incident angle that points to the incident ray of the unit ball centre of sphere is

wherein θ is the angle of incident ray and unit ball Z axis positive dirction,

the projection of incident ray in unit ball XY plane and the angle of unit ball X-axis positive dirction, by the incident angle of incident ray

flake imaging model to the imaging point (u, v) on fish eye images is described by following equation:

r(θ)＝k ₁θ+k ₂θ ²+k ₃θ ³+k ₄θ ⁴+k ₅θ ⁵+...k _nθ ⁿ (1)

Certain pixel on r presentation video is to the distance of figure principal point, k ₁... k _nit is fish-eye imaging parameters.

Δ _rrepresent fish-eye radial distortion, l ₁... l _n, i ₁... i ₄for radial distortion parameter.

Δ _trepresent fish-eye tangential distortion, m ₁... m _n, j ₁... j ₄for tangential distortion parameter.

X _dfor the position vector of pixel, i.e. (x _d, y _d), ur is vector of unit length radially, for tangential vector of unit length.

$\left(\begin{array}{c}u\\ v\end{array}\right)=\left(\begin{array}{cc}{m}_u& 0\\ 0& m_v\end{array}\right)\left(\begin{array}{c}{x}_d\\ y_d\end{array}\right)+\left(\begin{array}{c}{u}_0\\ v_0\end{array}\right)---\left(5\right)$

(u ₀, v ₀) be the principal point coordinate of image, (m _u, m _v) be respectively the pixel count in unit distance in CCD level and vertical direction.

Wherein Given information is the incident angle of spatial point x

and the corresponding point (u, v) of spatial point x on image, all the other parameter (k ₁, k ₂, k ₃, k ₄, k ₅, l ₁, l ₂, l ₃, i ₁, i ₂, i ₃, i ₄, m ₁, m ₂, m ₃, j ₁, j ₂, j ₃, j ₄, m _u, m _v, u ₀, v ₀) be fish eye lens parameter to be calibrated.It is to be noted, polynomial order in formula (1), (2), (3) can arrive infinite time, in experiment, find, polynomial expression order is got 5 times in formula (1), in formula (2), formula (3) polynomial expression order get 3 times can Accurate Model panorama picture of fisheye lens process.

Initialization internal reference module 50 is for initialization internal reference.

The concrete processing of initialization internal reference is as follows.Read the lens parameters that manufacturer provides: focal distance f and maximum view angle theta _maxmake k ₁=f, r _max=f θ _max.Detect fish eye images border, frontier point carried out to ellipse fitting:

${\left(\frac{u-u_0}{a}\right)}^2+{\left(\frac{v-v_0}{b}\right)}^2=1.$

Then according to ellipse fitting, try to achieve u ₀, v ₀, a, b,

wherein (a, b) is oval length semiaxis, (u ₀, v ₀) be the oval center of circle.And other parameter is set to 0.

Internal reference refers to the parameter of fisheye camera self, and irrelevant with external environment condition, in the present embodiment, the internal reference of indication is: (k ₁, k ₂, k ₃, k ₄, k ₅, l ₁, l ₂, l ₃, i ₁, i ₂, i ₃, i ₄, m ₁, m ₂, m ₃, j ₁, j ₂, j ₃, j ₄, m _u, m _v, u ₀, v ₀) it is to be noted, polynomial order in formula (1), (2), (3) can arrive infinite time, in experiment, find, polynomial expression order is got 5 times in formula (1), in formula (2), formula (3) polynomial expression order get 3 times can Accurate Model panorama picture of fisheye lens process.

Homography matrix computing module 60 is for calculating homography matrix.

The principle of the calculating homography matrix of the present embodiment as shown in Figure 6, comprises that gridiron pattern image fetching unit 610, point of crossing choose unit 611, back projection unit 612, homography matrix estimation unit 613, homography matrix and optimize unit 614.

Annexation between these unit is: the output terminal of gridiron pattern image fetching unit 610 couples point of crossing and chooses unit 611, the output terminal that unit 611 is chosen in point of crossing couples back projection unit 612, the output terminal of back projection unit 612 couples homography matrix estimation unit 613, and the output terminal of homography matrix estimation unit 613 couples homography matrix and optimizes unit 614.

Gridiron pattern image fetching unit 610 reads cross-hatch pattern picture.

Point of crossing is chosen unit 611 and on every cross-hatch pattern picture, is chosen successively all gridiron pattern point of crossing.

Back projection unit 612 utilizes initialization internal reference, by point of crossing

back projection obtains vector of unit length to unit ball

process is as follows:

$\left(\begin{array}{c}{x}_j^i\\ y_j^i\end{array}\right)=\left(\begin{array}{cc}\frac{1}{m_u}& 0\\ 0& \frac{1}{m_v}\end{array}\right)\left(\begin{array}{c}{u}_j^i-u_0\\ v_j^i-v_0\end{array}\right)$

$r_j^i=\sqrt{{\left(x_j^i\right)}^2+{\left(y_j^i\right)}^2}$

$k_1{\mathrm{\θ}}_j^i+k_2{\left({\mathrm{\θ}}_j^i\right)}^2+k_3{\left({\mathrm{\θ}}_j^i\right)}^3+k_4{\left({\mathrm{\θ}}_j^i\right)}^4+k_5{\left({\mathrm{\θ}}_j^i\right)}^5-r_j^i=0$

Wherein j is j width image, and i is i gridiron pattern point of crossing.

Homography matrix estimation unit 613 is estimated homography matrix.Vector of unit length

can be expressed as: spatial point x on itself and scaling board ⁱbetween there is homograph H _j, by linear algorithm, estimate this homograph H _j, obtain spatial point x ⁱat homograph H _junder corresponding point:

x wherein _p ⁱit is the volume coordinate of i point of crossing on j width cross-hatch pattern picture.

Homography matrix is optimized unit 614 by LM iteration minimum error function to optimize homography matrix H _j, wherein

it is vector

with

between angle.

Outer moduli piece 70 initialization of initialization are joined outward.

Outer ginseng

can be by homography matrix H _jinitialization is as follows:

$r_j^1={\mathrm{\λ}}_j{h}_j^1,$ $r_j^2={\mathrm{\λ}}_j{h}_j^2,$ $r_j^3=r_j^1\×r_j^2,$ $t_j={\mathrm{\λ}}_j{h}_j^3$

Wherein, r _jfor rotation parameter, T _jfor displacement parameter,

be j homography matrix H _ji column vector.

Outer ginseng refers to the parameter between fisheye camera and external environment condition, refers to the parameters R between fisheye camera and scaling board in the present embodiment _jand T _j.

Iteration optimization module 80 minimizes re-projection error for LM iteration interior participation after being optimized is joined outward.Wherein

for picture point

between pixel distance, wherein M is the point of crossing quantity on every width cross-hatch pattern picture, N is gridiron pattern amount of images.

Flake stereoscopic vision module 102a in data collector 10a gathers continuous fish eye images, and these image datas are stored on data storage device 20a, utilize the parameter that flake stereoscopic vision is right, data processing equipment 30a processes gathered data, the measurement of realization to gathered flake image, such as road width, radius of turn, building wide, high, the area of billboard etc.

It should be noted that CCD camera 1024a wherein can be also other forms of for example CMOS camera.

the second embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System

Fig. 2 shows the second embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention.Refer to Fig. 2.The present embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System, comprise data collector 10b, data storage device 20b, data processing equipment 30b.

Annexation between these three devices is: the output terminal connection data memory storage 20b of data collector 10b, the output terminal of the data storage device 20b 30b that connects data processing apparatus.

Data collector 10b comprises fisheye camera stereoscopic vision module 102b and vehicle pose acquisition module 104b.Fisheye camera stereoscopic vision module 102b, by least one stereoscopic vision to forming, i.e. two fisheye cameras.Flake stereoscopic camera stereoscopic vision module 102b comprises fish eye lens 1022b, CCD camera 1024b and caliberating device 1026b.It is upper that fish eye lens 1022b is connected to CCD camera 1024b, and composition can generate the flake image of super wide-angle view.Caliberating device 1026b is for obtaining the imaging parameters of fisheye camera and fisheye camera stereoscopic vision to parameter.The embodiment of caliberating device 1026b (having two kinds) is all described in detail in the first embodiment, does not repeat them here.

Vehicle pose acquisition module 104b is for obtaining the accurate distance of longitude and latitude, attitude information and the Vehicle Driving Cycle of Vehicle Driving Cycle process.Vehicle pose acquisition module 104b can comprise global position system, as gps antenna, gps receiver, GPS base station, and vehicle attitude acquisition device, as gyroscope, accelerometer etc., vehicle operating is apart from acquisition device, as automobile bus or wheel encoder device etc.

Flake stereoscopic vision module 102b in data collector 10b gathers continuous fish eye images, vehicle pose acquisition module 104b obtains the range information of longitude and latitude, attitude and vehicle operating in Vehicle Driving Cycle process, and these image datas and position, attitude, range information store on data storage device 20b.Utilize the parameter that flake stereoscopic vision is right, data processing equipment 30b processes gathered data, realizes the measurement to gathered flake image, such as road width, and building wide, high.The information collecting in conjunction with vehicle pose acquisition module 104b, data processing equipment 30b can calculate accurate terrestrial coordinate and the direction of object in flake image, it is the absolute location information of attention object in image, thereby attention object is marked, to realize the object of city management, generaI investigation and planning.

the 3rd embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System

Fig. 3 shows the 3rd embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System of the present invention.Refer to Fig. 3.The present embodiment based on fish-eye vehicle-mounted mobile Digital Photogrammetric System, comprise data collector 10c, data storage device 20c, data processing equipment 30c.

Annexation between these three devices is: the output terminal connection data memory storage 20c of data collector 10c, the output terminal of the data storage device 20c 30c that connects data processing apparatus.

Data collector 10c comprises fisheye camera stereoscopic vision module 102c, vehicle pose acquisition module 104c and panorama acquisition module 106c.

Fisheye camera stereoscopic vision module 102c by least one stereoscopic vision to forming, i.e. two fisheye cameras.Flake stereoscopic camera stereoscopic vision module 102c comprises fish eye lens 1022c, CCD camera 1024c and caliberating device 1026c.It is upper that fish eye lens 1022c is connected to CCD camera 1024c, and composition can generate the flake image of super wide-angle view.Caliberating device 1026c is for obtaining the imaging parameters of fisheye camera and fisheye camera stereoscopic vision to parameter.The embodiment of caliberating device 1026c (having two kinds) is all described in detail in the first embodiment, does not repeat them here.

Vehicle pose acquisition module 104c is for obtaining longitude and latitude, the attitude information of Vehicle Driving Cycle process, and the accurate distance of Vehicle Driving Cycle.Vehicle pose acquisition module 104c can comprise global position system, as gps antenna, gps receiver, GPS base station, and vehicle attitude acquisition device, as gyroscope, accelerometer etc., vehicle operating is apart from acquisition device, as automobile bus or wheel encoder device etc.

Panorama acquisition module 106c is for gathering the continuous full-view image data of high-quality.

Flake stereoscopic vision module 102c in data collector 10c gathers continuous fish eye images, vehicle pose acquisition module 104c obtains the range information of longitude and latitude, attitude and vehicle operating in Vehicle Driving Cycle process, panorama acquisition module 106c gathers continuous road full-view image, and these image datas and position, attitude, range information store on data storage device 20c.Utilize the parameter that flake stereoscopic vision is right, data processing equipment 30c processes gathered data, realizes the measurement to gathered flake image, such as road width, and building wide, high.The information collecting in conjunction with vehicle pose acquisition module 104c, data processing equipment 30c can calculate accurate terrestrial coordinate and the direction of flake image, i.e. and the absolute location information of measured object can obtain having the full-view image data of GIS information simultaneously.

Above-described embodiment is to provide to those of ordinary skills and realizes and use of the present invention, those of ordinary skills can be without departing from the present invention in the case of the inventive idea, above-described embodiment is made to various modifications or variation, thereby invention scope of the present invention do not limit by above-described embodiment, and it should be the maximum magnitude that meets the inventive features that claims mention.

Claims (9)

1. based on a fish-eye vehicle-mounted mobile Digital Photogrammetric System, comprise data collector, data storage device and data processing equipment, wherein:

Data collector, for gathering image and data, it comprises fisheye camera stereoscopic vision module, for gathering fish eye images information, fisheye camera stereoscopic vision module comprises fish eye lens, camera and caliberating device, wherein:

Fish eye lens, is connected with camera, obtains flake image;

Camera, is connected with fish eye lens, and the flake image that reception fish eye lens obtains also carries out image collection;

Caliberating device, obtains fish-eye imaging parameters and fish eye lens stereoscopic vision to parameter;

Data storage device, couples data collector, the image data arriving for storage of collected;

Data processing equipment, couples data storage device, for the treatment of collected data, realizes the measurement to flake image;

Wherein caliberating device comprises:

Flake imaging relations is set up module, sets up half unit Sphere Measurement Model, and on unit sphere model, sets up flake imaging relations;

Initialization internal reference module, couples flake imaging relations and sets up module, initialization internal reference, and wherein internal reference is the parameter of fisheye camera self, irrelevant with external environment condition;

Homography matrix computing module, couples initialization internal reference module, calculates homography matrix;

The outer moduli piece of initialization, couples homography matrix computing module, and initialization is joined outward, and it is joined is at home and abroad the parameter between fisheye camera and external environment condition;

Iteration optimization module, couples the outer moduli piece of initialization, and LM iteration minimizes re-projection error, and the interior participation after being optimized is joined outward.

2. according to claim 1ly based on fish-eye vehicle-mounted mobile Digital Photogrammetric System, it is characterized in that, data collector also comprises:

Vehicle pose acquisition module, for obtaining position, the attitude of vehicle, the information of range ability;

The information of the position of the data storage device vehicle that also store car pose acquisition module obtains, attitude, range ability.

3. according to claim 1ly based on fish-eye vehicle-mounted mobile Digital Photogrammetric System, it is characterized in that, data collector also comprises:

Panorama acquisition module, for gathering continuous road full-view image.

4. according to claim 1ly based on fish-eye vehicle-mounted mobile Digital Photogrammetric System, it is characterized in that, fisheye camera stereoscopic vision module at least comprises two fisheye cameras.

5. according to claim 1ly based on fish-eye vehicle-mounted mobile Digital Photogrammetric System, it is characterized in that, it is as follows that flake imaging relations is set up the processing of module:

In note space, the imaging point of certain 1 x on fish eye images is (u, v), and the incident angle that spatial point x points to the unit ball centre of sphere is

, wherein θ is the angle of incident ray and unit ball Z axis positive dirction, the projection of incident ray in unit ball XY plane and the angle of unit ball X-axis positive dirction, by the incident angle of incident ray

flake imaging model to the imaging point (u, v) on fish eye images is described by following equation:

r(θ)=k ₁θ+k ₂θ ²+k ₃θ ³+k ₄θ ⁴+k ₅θ ⁵+...k _nθ ⁿ （1）

Certain pixel on r presentation video is to the distance of figure principal point, k ₁... k _nit is fish-eye imaging parameters;

Δ _rrepresent fish-eye radial distortion, l ₁... l _n, i ₁... i ₄for radial distortion parameter;

Δ _trepresent fish-eye tangential distortion, m ₁... m _n, j ₁... j ₄for tangential distortion parameter;

X _dfor the position vector of pixel, i.e. (x _d, y _d), u _rfor vector of unit length radially,

for tangential vector of unit length;

$\left(\begin{array}{c}u\\ v\end{array}\right)=\left(\begin{array}{cc}{m}_u& 0\\ 0& m_v\end{array}\right)\left(\begin{array}{c}{x}_d\\ y_d\end{array}\right)+\left(\begin{array}{c}{u}_0\\ v_0\end{array}\right)---\left(5\right)$

(u ₀, v ₀) be the principal point coordinate of image, (m _u, m _v) be respectively the pixel count in unit distance in CCD level and vertical direction, (k ₁, k ₂, k ₃, k ₄, k ₅, l ₁, l ₂, l ₃, i ₁, i ₂, i ₃, i ₄, m ₁, m ₂, m ₃, j ₁, j ₂, j ₃, j ₄, m _u, m _v, u ₀, v ₀) be fish-eye parameter to be calibrated.

6. according to claim 5ly based on fish-eye vehicle-mounted mobile Digital Photogrammetric System, it is characterized in that, homography matrix computing module comprises:

Gridiron pattern image fetching unit, reads the cross-hatch pattern picture on scaling board;

Unit is chosen in point of crossing, couples gridiron pattern image fetching unit, chooses successively 4 point of crossing at place, gridiron pattern summit on every cross-hatch pattern picture;

Back projection unit, couples point of crossing and chooses unit, utilizes initialization internal reference, by point of crossing

back projection obtains vector of unit length to unit ball wherein j is j width image, and i is i gridiron pattern point of crossing;

Homography matrix estimation unit, couples back projection unit, and estimate sheet is answered matrix H _j, by vector of unit length

be expressed as

spatial point x on vector of unit length and scaling board ⁱbetween there is homograph H _j, by linear algorithm, estimate homograph H _j, obtain the spatial point x on scaling board ⁱat homograph H _junder corresponding point:

x wherein _p ⁱit is the volume coordinate of i point of crossing on j width cross-hatch pattern picture;

Homography matrix is optimized unit, couples homography matrix estimation unit, by LM iteration minimum error function

to optimize homography matrix H _j, wherein

it is vector

with

between angle;

Point of crossing map unit, couples homography matrix and optimizes unit, and the homography matrix H after optimization is passed through in all point of crossing on scaling board _jbe mapped to and on unit ball, obtain corresponding point:

Point of crossing image coordinate acquiring unit, couples point of crossing map unit, and vector of unit length is transformed on image: $\left(\begin{array}{c}{\hat{u}}_j^i\\ {\hat{v}}_j^i\end{array}\right)=\left(\begin{array}{cc}{m}_u& 0\\ 0& m_v\end{array}\right)\left(\begin{array}{c}{\hat{x}}_j^i\\ {\hat{y}}_j^i\end{array}\right)+\left(\begin{array}{c}{u}_0\\ v_0\end{array}\right),$ At subpoint

the image coordinate of neighbor searching point of crossing

(m _u, m _v) be respectively the pixel count in unit distance in CCD level and vertical direction.

7. according to claim 5ly based on fish-eye vehicle-mounted mobile Digital Photogrammetric System, it is characterized in that, homography matrix computing module comprises:

Gridiron pattern image fetching unit, reads the cross-hatch pattern picture on scaling board;

Unit is chosen in point of crossing, couples gridiron pattern image fetching unit, chooses successively all gridiron pattern point of crossing on every cross-hatch pattern picture;

Back projection unit, couples point of crossing and chooses unit, utilizes initialization internal reference, by point of crossing

back projection obtains vector of unit length to unit ball

wherein j is j width image, and i is i gridiron pattern point of crossing;

Homography matrix estimation unit, couples back projection unit, and estimate sheet is answered matrix H _j, by vector of unit length

be expressed as

spatial point x on vector of unit length and scaling board ⁱbetween there is homograph H _j, by linear algorithm, estimate homograph H _j, obtain the spatial point x on scaling board ⁱat homograph H _junder corresponding point:

x wherein _p ⁱit is the volume coordinate of i point of crossing on j width cross-hatch pattern picture;

Homography matrix is optimized unit, couples homography matrix estimation unit, by LM iteration minimum error function

to optimize homography matrix H _j, wherein it is vector

with

between angle.

According to described in claim 6 or 7 based on fish-eye vehicle-mounted mobile Digital Photogrammetric System, it is characterized in that, the processing of the outer moduli piece of initialization is as follows:

Outer ginseng

by homography matrix H _jinitialization is as follows:

$r_j^1={\mathrm{\λ}}_j{h}_j^1,r_j^2={\mathrm{\λ}}_j{h}_j^2,r_j^3=r_j^1\×r_j^2,t_j={\mathrm{\λ}}_j{h}_j^3$

Wherein,

r _jfor rotation parameter, T _jfor displacement parameter,

be j homography matrix H _ji column vector.

9. according to claim 8ly based on fish-eye vehicle-mounted mobile Digital Photogrammetric System, it is characterized in that, the processing of iteration optimization module is as follows:

LM iteration minimizes re-projection error

internal reference after being optimized and outer ginseng, wherein

for picture point

between pixel distance, M is the point of crossing quantity on every width cross-hatch pattern picture, N is gridiron pattern amount of images.

Images