CN101442618A - Method for synthesizing 360 DEG ring-shaped video of vehicle assistant drive - Google Patents

Abstract

The invention discloses a method for synthesizing annular video of 360 degrees for assistant drive of a vehicle. The method adopts a video acquisition unit, an annular video synthesis unit and a display control and storage unit, and has the characteristics of reliable stability, low cost and suitable batch production. The method can acquire video signals in real time for four super-wide-angle fish-eye cameras arranged at front, back, left and right of the vehicle, carries out treatment of fish-eye image correction, annular image mosaic and vision angle conversion on four channels of the video signals through a video decoding unit and a video synthesizing control unit to generate a panoramic displayed plan view of 360 degrees, achieves multiple display modes through the display control and storage unit, and can selectively store correlated video information. The method can provide surrounding scenes for a driver, and eliminate vision blind area thereof, and is favorable for the driver to achieve safe and accurate vehicle driving operations. The method is used for assistant drive for sedans, business vehicles and heavy trucks of various grades, and passenger coaches of different types.

Description

The annular image synthesizing method of 360 degree that is used for vehicle assistant drive

Technical field

The invention belongs to motor vehicles driver assistance and field of video monitoring, relate to a kind of by vehicle all around 4 cameras take simultaneously and carry out the real-time video treatment technology, be particularly related to the annular image synthesizing method of a kind of 360 degree, this method is applicable to car, commercial vehicle, the truck of various class, the driver assistance of dissimilar passenger vehicles.

Background technology

Automobile has become a kind of main transport facility of modern society, and people have higher requirement to traveling security and comfortableness.The initial imagination of the invention of automobile rearview mirror is exactly the situation that makes things convenient for the driver to watch automobile both sides and back, prevents to bump with the external world in reversing process, improves driving and the safety of the third party outside car of self, prolongs the useful life of automobile.But in fact, only rely on driving vision and information that rearview mirror provides, be difficult to satisfy the demand of driver for traffic safety, especially in reversing, the back of vehicle exists very big blind area.

Radar for backing car is that present automobile adopts more a kind of auxiliary driving method, come detection vehicle afterbody object by the transducer that is installed on the vehicle rear bumper, and remind by the audio frequency of different rhythm that the situation of driver's vehicle back, its principle are to rely on the Echo Sounding distance and point out by the sound of different frequency.But do not have vision to come intuitively only according to auditory tone cues, can have error yet the judgement of sound.

Situation when the video backing system that automobile tail is installed single camera allows reversing behind the car is more intuitively visual, when hanging reverse gear shift, this system can connect the camera that is positioned at the tailstock automatically, and situation behind the car is presented on the LCDs before the driver clearly.Video backing system is more directly perceived, practical compared with radar for backing car.There is the blind area in the video backing system of the single camera of automobile tail to other zone of motor vehicle environment, and the dead angle during as vehicle body side, front bumper and turning etc. need further develop its processing system for video.The how tame motor corporation of Japan has proposed and has developed a kind of automobile and look around implementation method at present, comprises the Around View Monitor ring scape surveillance of Nissan Motor, the various visual angles camera chain of Honda company (Multi-View Camera System).The principle of these two kinds of systems is roughly similar, mainly be to carry out video by 4 cameras in automobile all around to synthesize, by methods such as image splicings, form the vertical view that vehicle body is got a bird's eye view, make the driver can be more clearly see situation around the automobile, the driving behaviors such as driving, reversing, left-hand bend of being convenient to keep to the side clearly.At present domesticly also do not propose corresponding automobile and look around implementation method, the automobile that Japanese firm proposes is looked around implementation method and is had problems such as video image is clear inadequately, the image splicing is discontinuous, cost is too high.Demand at following automobile assistant driving system development, the present invention proposes the annular image synthesizing method of a kind of 360 degree (Around Video Compose System-AVCS), or claim that ((Omnidirectional Display System-ODS), this method possesses simultaneously display automobile and gets a bird's eye view the panorama sketch of surrounding environment from the top downwards and switch and select the display automobile function of four side scenes all around the panorama display packing for Panoramic Display System-PDS), omnibearing display method.

Summary of the invention

The object of the present invention is to provide a kind of energy display automobile annular image synthesizing method of 360 degree of environment all around, 4 high-resolution (more than 720 * 480) that all around are provided with respectively by vehicle body, the fish-eye camera of super wide-angle (the horizontal/vertical visual angle surpasses more than 135 degree) is as video acquisition unit, annular video synthesis unit, show that control and memory cell constitute, can show vertical view and the surrounding environment of getting a bird's eye view vehicle body in real time, also can select to show the environment in territory, vehicle body four lateral areas, for the driver provides safe driving required video information, realize safety, accurate vehicle driving operation.

In order to realize above-mentioned task, the annular image synthesizing method of 360 degree that is used for vehicle assistant drive that the present invention proposes:

Gather four tunnel real-time vision signals by video acquisition unit (100), video decoding unit (210) and the synthetic control unit (220) of video delivered in the annular video synthesis unit (200) are handled, wherein the synthetic control unit (220) of video carries out four tunnel vision signals generating the vertical view that 360 degree panoramas show after the processing of correcting fisheye image (221), annular image splicing (222), visual angle conversion (223), SDRAM storage control (224), through showing that control and memory cell (300) realize plurality of display modes, store video information.

Video acquisition module unit (100) is made of the super wide-angle fish-eye camera (102) on the super wide-angle fish-eye camera (101) that is installed in headstock grid place, the left-hand mirror, the super wide-angle fish-eye camera (103) on the right back visor, the super wide-angle fish-eye camera (104) at tailstock top, respectively the real time video signals of collection vehicle four sides; The vision signal frame frequency was 30 frame/seconds, and resolution is 720 * 480; Designed short range infrared LEDs lamp around the super wide-angle fish-eye camera, its can by sensor devices automatically induction down control its open and close, thereby be fit to low-light (level) at night, applied environment that light is dark; Super wide-angle fish-eye camera output composite video signal CVBS is to annular video synthesis unit (200).

Its processing procedure of described annular video synthesis unit (200) is: adopt FLASH and the CPLD bootstrapping that powers on, after system's operation, earlier to dsp chip hardware initialization, interrupt vector initialization, by iic bus the video decoding chip of video decoding unit (210) is carried out initialization again, after the initialization success, produce four road 8bits numeral YUV signal, be sent to the synthetic control unit (220) of video, and handle.

4, according to claims 1 described annular image synthesizing method of 360 degree that is used for vehicle assistant drive, it is characterized in that: show that control and memory cell (300) are made of indicative control unit (301) and SD card storage control unit (302).

Video decoding unit (210) is used the super wide-angle fish-eye camera (101) of video decoding chip to headstock grid place, super wide-angle fish-eye camera (102) on the left-hand mirror, super wide-angle fish-eye camera (103) on the right back visor, the composite video signal CVBS that the super wide-angle fish-eye camera (104) at tailstock top sends decodes, the 8bits numeral YUV signal of output ITU656 standard YUV4:2:2 form, this four tunnel YUV signal is delivered to the first video inputs mouth VP0 of the synthetic control unit (220) of video, the second video inputs mouth VP1, the 3rd video inputs mouth VP2, the 4th video inputs mouth VP3.

Video synthesizes control unit (220) to be realized based on dsp chip, comprise correcting fisheye image unit (221), annular image concatenation unit (222), visual angle converting unit (223) and SDRAM storage control unit (224), video synthesizes control unit (220) the vehicle's surroundings view is generated the vertical view that 360 degree panoramas show through calculation process;

Described video synthesizes control unit (220) processing procedure: adopt correcting fisheye image unit (221) that the image of distortion is proofreaied and correct earlier, four images after the correction are spliced into the panorama sketch of 360 degree through annular image concatenation unit (222), adopt visual angle converting unit (223) to transfer the vertical view of getting a bird's eye view to panorama sketch then, wherein adopt video data and correcting fisheye image unit (221) of SDRAM storage control unit (224) input, annular image concatenation unit (222), the result that visual angle converting unit (223) is handled carries out caching process by outside SDRAM.

Correcting fisheye image unit (221) is made up of mapping (221-1) and two steps of interpolation (221-2), and the piecture geometry fault of adopting fish eye lens to cause is proofreaied and correct;

Described mapping (221-1), its process is as follows: in trimming process, the coordinate of establishing benchmark image is (x, y), the fault image coordinate be (u, v), by the benchmark image coordinate being mapped to the correction that realizes image in the fault image, the 2-d polynomial Mathematical Modeling of its mapping is:

$\left\{\begin{array}{c}u=F_u(x,y)=\underset{i=0}{\overset{n}{\mathrm{\Σ}}}\underset{j=0}{\overset{n-i}{\mathrm{\Σ}}}{a}_{\mathrm{ij}}{x}_i{y}_j\\ v=F_v(x,y)=\underset{i=0}{\overset{n}{\mathrm{\Σ}}}\underset{j=0}{\overset{n-i}{\mathrm{\Σ}}}{b}_{\mathrm{ij}}{x}_i{y}_j\end{array}\right.$

In the formula: n is polynomial item number, a _IjAnd b _IjBe every coefficient;

Because the gray value of image same point should remain unchanged, promptly g (x, y)=f (u, v).After finishing mapping step, be decided to be integer, must adopt interpolation (221-2) to calculate because the x, the y that calculate differ;

Described interpolation (221-2) adopts bilinear interpolation method to realize.

Annular image concatenation unit (222) is made up of coupling (222-1) and stitching (222-2) two steps, generates automobile panorama displayed map all around;

Described coupling (222-1) adopts " two step couplings " method to realize.

Visual angle converting unit (223) utilizes annular image concatenation unit (222) to obtain on the basis of panorama displayed map, carries out the panorama vertical view that contrary perspective projection transformation (223-1) obtains environment surrounding automobile;

Described contrary perspective projection transformation (223-1), implementation step is as follows:

1) S → I mapping is by S plane coordinates point (x _v, y _v, 0), mapping I plane coordinates point (v, u),

$u=\frac{r_{21}{x}_v+r_{22}{y}_v-{\left(R^{-1}{t}_0^c\right)}_2}{r_{11}{x}_v+r_{12}{y}_v-{\left(R^{-1}{t}_0^c\right)}_1}\×f_u+u_0$

$v=-\frac{r_{31}{x}_v+r_{32}{y}_v-{\left(R^{-1}{t}_0^c\right)}_3}{r_{11}{x}_v+r_{12}{y}_v-{\left(R^{-1}{t}_0^c\right)}_1}\&times;f_v+{\mathrm{<figure-callout\; id="0"\; label="v"\; filenames="A200810236552E00241.png"\; state="\{\{state\}\}">v</figure-callout>}}_0$

Wherein $R=\left[\begin{array}{ccc}\cos a\cos \left(b\right)& \sin \left(b\right)& -\sin a\cos \left(b\right)\\ -\cos a\sin \left(b\right)& \cos \left(b\right)& \sin a\sin \left(b\right)\\ \sin a& 0& \cos a\end{array}\right]$ The expression world coordinates is to the rotation relationship of camera coordinate;

$t_0^c={[l,d,h]}^T$ The expression world coordinates is to the translation relation of camera coordinate;

r _IjThe inverse matrix R of expression spin matrix R ^-1The component of the capable j of i row, promptly $R^{-1}=\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right];$

Expression R ^-1With translation matrix Product gained matrix

I component,

$R^{-1}{t}_0^c=\left[\begin{array}{c}{\left(R^{-1}{t}_0^c\right)}_1\\ {\left(R^{-1}{t}_0^c\right)}_2\\ {\left(R^{-1}{t}_0^c\right)}_3\end{array}\right];$

(u ₀, v ₀) expression video camera principal point coordinate; f _u, f _vExpression U, the focal length scale factor of V axle;

2) on the S plane that calculates of previous step some the location of pixels on image coordinate system I (u v) is not integer usually, and this module adopts the bilinear interpolation method of introducing in the image correction module to calculate pixel (x in the re-projection image _v, y _v) pixel value,

Traversal coordinate system (x _v, y _v, 0) and pel array (forming the re-projection image) among the ∈ W, according to above-mentioned two steps, calculate each the some respective pixel location point [u (x on the re-projection image _v, y _v, ⁰), v (x _v, y _v, 0)] gray value of ∈ I.

Indicative control unit (301) reads in the video information of vehicle's surroundings view from outside SDRAM, select output two-path video information through video, respectively two-path video information is carried out video scaling, pass through video superimpose again, as the left and right sides sprite demonstration (301-1) of same display terminal;

Described left and right sides sprite shows that the display mode of (301-1) comprising: show vehicle body panorama birds-eye view and any end view in left and right sides picture mode on same display screen, be used for the keep to the side application scenario of driving a vehicle, moveing backward or parking of automobile; The view that shows the vehicle body left and right sides on same display screen in left and right sides picture mode is used for the application scenario that automobile travels at the parting of the ways or the narrow road section is current; On same display screen, show vehicle body front and back view, be used for the application scenario of vehicle vehicle condition before and after the monitoring of cruising process in left and right sides picture mode;

Described SD card storage control unit (302) is used for the driver with manual or Automatic Styles recording video information, and the length of its video information is adjusted according to the capacity of SD card.

Advantage of the present invention be adopt 4 cheaply the superelevation angle fish-eye camera be installed in the video image of gathering four sides all around of vehicle, realize the vertical view that vehicle 360 degree panoramas show by the synthetic control unit of annular video, and optional majority kind display mode or store video information.

Compare with the prior art of companies such as Nissan Motor, Honda, technique effect of the present invention is embodied in:

1. the present invention adopts that 4 super cheaply wide-angle fish-eye cameras are installed in the headstock grid place, left-hand mirror of vehicle, on the right back visor, tailstock top, can gather the video image of four sides simultaneously, the vision signal frame frequency of being gathered was 30 frame/seconds, and resolution is 720 * 480.

2, super wide-angle fish-eye camera of the present invention, short range infrared LEDs lamp, the application scenario that suitable light is darker have been designed greater than 130 degree in the visual angle around it.

3, the annular video synthesis unit 200 that the present invention developed is made of video decoding unit 210, the synthetic control unit 220 of video.The synthetic control unit 220 of its video is made of correcting fisheye image unit 221, annular image concatenation unit 222, visual angle converting unit 223, SDRAM storage control 224.The vertical view that vehicle 360 degree panoramas show has the characteristics of clear picture, seamless link.

4, demonstration control that the present invention developed and memory cell 300 can realize plurality of display modes by indicative control unit 301 processing, also can select to store the video information that is taken place.Display mode comprises and shows birds-eye view+end view simultaneously, left and right sides view, front and back view.

5. the present invention is made of with memory cell 300 video acquisition unit 100, annular video synthesis unit 200, demonstration control, has reliable and stable, low-cost, as to be fit to batch process characteristics.

Description of drawings

Fig. 1 is a schematic diagram of the present invention;

Fig. 2 overlooks and the camera structure schematic diagram for automobile of the present invention;

Fig. 3 is an annular video synthesis unit structure chart of the present invention;

Fig. 4 is bilinear interpolation algorithm principle figure of the present invention;

Fig. 5 is preceding piece and a back piece schematic diagram in the matching process of the present invention;

Fig. 6 is image splicing transformation of coordinates of the present invention;

Fig. 7 is contrary perspective projection transformation mapping relations figure of the present invention;

Fig. 8 is camera coordinate system of the present invention and bodywork reference frame relation;

Fig. 9 is an indicative control unit structure chart of the present invention.

Wherein: 100-video acquisition unit, 200-annular video synthesis unit, 300-show and control and memory cell, the super wide-angle fish-eye camera in 101-headstock grid place, the super wide-angle fish-eye camera in 102-left-hand mirror place, the super wide-angle fish-eye camera in 103-right back visor place, the super wide-angle fish-eye camera in 104-tailstock top, 210-video decoding unit, 220-video synthesizes control unit, 221-correcting fisheye image unit, 222-annular image concatenation unit, 223-visual angle converting unit, 224-SDRAM storage control unit, 301-indicative control unit, 302-SD card storage control unit.

Below in conjunction with accompanying drawing the present invention is described in further detail.

Embodiment

Referring to Fig. 1, the present invention constitutes the annular image synthesizing method of 360 degree by video acquisition unit 100, annular video synthesis unit 200, demonstration control with memory cell 300.

Referring to Fig. 2, automobile is overlooked and camera comprises the super wide-angle fish-eye camera 104 at the super wide-angle fish-eye camera 101 at headstock grid place, the super wide-angle fish-eye camera 102 on the left-hand mirror, the super wide-angle fish-eye camera 103 on the right back visor, tailstock top, annular video synthesis unit 200, shows that control and memory cell 300 are positioned at before the driver on the deck plate.In darker occasion, as under the night low-light (level) applied environment, the short range infrared LEDs lamp around the camera will be by sensor devices induction its opening and closing of control down automatically.

Referring to Fig. 3, annular video synthesis unit adopts the FLASH bootstrapping that powers on, and is first to dsp chip hardware initialization, interrupt vector initialization after system's operation, by iic bus the video decoding chip of video decoding unit 210 carried out initialization again.After the Video Decoder initialization success, four tunnel composite video signal CVBS that camera is sent decode, produce four road ITU656 standard 8bits numeral YUV signal, the first video inputs mouth VP0, the second video inputs mouth VP1, the 3rd video inputs mouth VP2, the 4th video inputs mouth VP3 that synthesize control unit 220 by video send into the dsp chip processing.Synthetic control unit 220 processing procedures of video are as follows:

Earlier adopt the image of the 221 pairs of distortion in correcting fisheye image unit to proofread and correct respectively, four width of cloth images after the correction are spliced into 360 panorama sketch of spending through annular image concatenation unit 222, adopt visual angle converting unit 223 to transfer the vertical view of getting a bird's eye view to panorama sketch at last.The intermediate object program that the video data of 224 pairs of inputs of SDRAM storage control unit and correcting fisheye image unit 221, annular image concatenation unit 222, visual angle converting unit 223 are handled is carried out caching process by outside SDRAM.Panorama birds-eye view and each lateral plan can show two width of cloth figure in the mode of cutting apart left and right sides picture by the processing of indicative control unit 301 on same display terminal, also can deposit in the SD card more than the outside 2G size in the consecutive image mode by SD card storage control unit 302.

The 221 pairs of piecture geometry faults of adopting fish eye lens to cause in correcting fisheye image unit are proofreaied and correct, and it mainly is made up of mapping 221-1 and two steps of interpolation 221-2.In trimming process, the coordinate of establishing benchmark image for (x, y), the fault image coordinate be (u, v), by the benchmark image coordinate being mapped to the correction of realization image in the fault image.The 2-d polynomial Mathematical Modeling of its mapping is:

$\left\{\begin{array}{c}u=F_u(x,y)=\underset{i=0}{\overset{n}{\mathrm{\&Sigma;}}}\underset{j=0}{\overset{n-i}{\mathrm{\&Sigma;}}}{a}_{\mathrm{ij}}{x}_i{y}_j\\ v=F_v(x,y)=\underset{i=0}{\overset{n}{\mathrm{\&Sigma;}}}\underset{j=0}{\overset{n-i}{\mathrm{\&Sigma;}}}{b}_{\mathrm{ij}}{x}_i{y}_j\end{array}\right.$

In the formula: n is polynomial item number, a _IjAnd b _IjBe every coefficient.

Because the gray value of image same point should remain unchanged, promptly g (x, y)=f (u, v).After finishing mapping step, be decided to be integer, must carry out interpolation arithmetic because the x, the y that calculate differ.Consider factors such as the precision of interpolation and amount of calculation, this step adopts the method for bilinear interpolation.

Referring to Fig. 4, use

Expression rounds downwards, then The expression respectively the row, column coordinate is rounded the location of pixels that obtains downwards, p1, p2 represent respectively (u, v) along u, v direction distance Distance, 0＜p1, p2＜1, as shown in Figure 4.

Then distribute to pixel (x in the re-projection image _u, y _v) pixel value be:

Annular image concatenation unit 222 is by coupling and sew up that two steps form, its objective is to generate automobile panoramic view all around.For two images to be spliced, for ease of describing, by the sequencing of input, we claim first width of cloth image to be " first figure ", and second width of cloth image is " second figure ".Consider the efficiency of coupling, propose to adopt " two step couplings " method.

Referring to Fig. 5, preceding piece and back piece schematic diagram have a r=3 in the matching process among the figure, and the plate to be matched of s=4 (r is the distance of center to four limits, and s is a step-length) is represented its sampled point with the black blockage.We are at r '=s-1 of the central authorities of this plate stack now, the plate of s '=1, and new plate to be matched is the combination of it and original plate.Claim that the new plate that adds is " preceding " of plate to be matched, original plate is " back piece " (dash area) of plate to be matched.As Fig. 5, preceding and back piece have a pixel to overlap, and promptly the center pixel of plate to be matched does not so almost cause double counting, on the contrary the importance of more outstanding this position.

During coupling, will be at second figure according to same rule read pixel value, the coupling of piece before carrying out earlier, the coupling of piece after carrying out again.Use d respectively _f, d _bThe diversity factor of piece and back piece before the expression, the total variances degree d=d of then current coupling plate _f+ d _bBecause the pixel count of preceding piece is A=(2r '+1) ²=(2s-1) ², then the mean difference of preceding each pixel of piece is $\stackrel{\&OverBar;}{d}=\frac{d_f}{{(2s-1)}^2}.$ The tolerance value of representing preceding piece mean difference with t.If d〉t, think that then current plate and plate to be matched can not mate, needn't carry out the coupling of back piece again; By that analogy up to finding out match point.

Referring to Fig. 6, among Fig. 6 (a), be that initial point is set up coordinate system with first figure image origin.Among Fig. 6 (b), be that initial point is set up coordinate system with second figure central point, and it is rotated θ around initial point.Eliminated the relative tilt angle with first figure this moment, is referred to as " smoothing second figure ".Use P ' to represent postrotational optimum, and represent to smooth 4 summits of second figure respectively with A, B, C, D.Black matrix rectangle among the figure is to cover smoothing second minimum rectangle figure, the base parallel coordinate axes, is referred to as " scope rectangle ".If summit, the scope rectangle lower left corner is U.

For guaranteeing O ₁All in splicing figure scope, the origin of coordinates O of new coordinate system should be made as: with O for point and U point ₁Point and U point are the summit, lower-left to rectangle angular vertex, the base parallel coordinate axes.Under new coordinate system, the displacement of first figure is so:

$\mathrm{\&Delta;X}=\left\{\begin{array}{cc}0,& {X\&prime;}_U\&GreaterEqual;0\\ -{X\&prime;}_U,& {X\&prime;}_U<0\end{array}\right.$

$\mathrm{\&Delta;Y}=\left\{\begin{array}{cc}0,& {Y\&prime;}_U\&GreaterEqual;0\\ -{Y\&prime;}_U,& {Y\&prime;}_U<0\end{array}\right.$

If at this moment P ' becomes P ' ' again, then the coordinate of Q (P ' ') point is:

${X\&prime;\&prime;}_P=\left\{\begin{array}{cc}{X}_Q,& {X\&prime;}_U\&GreaterEqual;0\\ X_Q-{X\&prime;}_U,& {X\&prime;}_U<0\end{array}\right.$

${Y\&prime;\&prime;}_P=\left\{\begin{array}{cc}{Y}_Q,& {Y\&prime;}_U\&GreaterEqual;0\\ Y_Q-{Y\&prime;}_U,& {Y\&prime;}_U<0\end{array}\right.$

So, we just can determine to inlay first figure and the coordinate transform of inlaying after second figure is placed into splicing figure fully with such description: with each pixel translation (Δ X, Δ Y) of first figure, and claim this displacement to be " suspension displacement "; With each pixel rotation θ of second figure, the centre coordinate before the rotation is (X _P, Y _P), postrotational centre coordinate is (X ＂ _P, Y ＂ _P).Now will smooth second figure translation, make P " point overlap with the Q point, shown in Fig. 6 (c).

For obtaining the panoramic coordinates vertical view of automobile, the present invention has designed visual angle converting unit 223.Utilize annular image concatenation unit 222 to obtain on the basis of panoramic picture, carry out contrary perspective projection transformation, obtain the panorama vertical view of environment surrounding automobile.Introduce contrary perspective projection transformation (InversePerspective Mapping-abbreviation IPM) method below.

Referring to Fig. 7, contrary perspective projection transformation mapping relations, this process can be regarded as a re-projection process by z=0 planar S among 2 dimension image space I to the 3 dimension Euclidean space W.Wherein:

W={ (x, y, z) } ∈ E ³Represent 3 dimension world coordinate systems.

I={ (u, v) } ∈ E ²Represent 2 dimension image coordinate systems.

Represent the xy plane on the W space.

Referring to Fig. 8, camera coordinate system and bodywork reference frame relation, specific to the vehicle-mounted pick-up head, the car body coordinate is world coordinate system among the present invention, camera in the relation of the position in the bodywork reference frame and bodywork reference frame and image coordinate system shown in Fig. 8 (a) and (b), (c).X wherein _vY _vZ _vBe bodywork reference frame, X _cY _cZ _cBe the camera coordinate system, UOV is the image coordinate system of camera c.The 3D coordinate of camera center in bodywork reference frame is [l, d, h] ^T, the direction of camera is determined by two anglec of rotation α and β, shown in Figure 10 (a) and (b).Concrete IPM realizes having following two steps:

1) S → I mapping is by S plane coordinates point (x _v, y _v, 0), and mapping I plane coordinates point (v, u).

$u=\frac{r_{21}{x}_v+r_{22}{y}_v-{\left(R^{-1}{t}_0^c\right)}_2}{r_{11}{x}_v+r_{12}{y}_v-{\left(R^{-1}{t}_0^c\right)}_1}\&times;f_u+{\mathrm{<figure-callout\; id="0"\; label="u"\; filenames="A200810236552E00241.png"\; state="\{\{state\}\}">u</figure-callout>}}_0$

$v=-\frac{r_{31}{x}_v+r_{32}{y}_v-{\left(R^{-1}{t}_0^c\right)}_3}{r_{11}{x}_v+r_{12}{y}_v-{\left(R^{-1}{t}_0^c\right)}_1}\&times;f_v+{\mathrm{<figure-callout\; id="0"\; label="v"\; filenames="A200810236552E00241.png"\; state="\{\{state\}\}">v</figure-callout>}}_0$

Wherein $R=\left[\begin{array}{ccc}\cos a\cos \left(b\right)& \sin \left(b\right)& -\sin a\cos \left(b\right)\\ -\cos a\sin \left(b\right)& \cos \left(b\right)& \sin a\sin \left(b\right)\\ \sin a& 0& \cos a\end{array}\right]$ The expression world coordinates is to the rotation relationship of camera coordinate;

$t_0^c={[l,d,h]}^T$ The expression world coordinates is to the translation relation of camera coordinate;

r _IjThe inverse matrix R of expression spin matrix R ^-1The component of the capable j of i row, promptly $R^{-1}=\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right];$

Expression R ^-1With translation matrix Product gained matrix

I component,

$R^{-1}{t}_0^c=\left[\begin{array}{c}{\left(R^{-1}{t}_0^c\right)}_1\\ {\left(R^{-1}{t}_0^c\right)}_2\\ {\left(R^{-1}{t}_0^c\right)}_3\end{array}\right];$

(u ₀, v ₀) expression video camera principal point coordinate; f _u, f _vExpression U, the focal length scale factor of V axle.

2) on the S plane that calculates of previous step some the location of pixels on image coordinate system I (u v) is not integer usually, and this module adopts the bilinear interpolation method of introducing in the image correction module to calculate pixel (x in the re-projection image _v, y _v) pixel value.

Traversal coordinate system (x _v, y _v, 0) and pel array (forming the re-projection image) among the ∈ W, according to above-mentioned two steps, calculate each the some respective pixel location point [u (x on the re-projection image _v, y _v, 0), v (x _v, y _v, ⁰)] gray value of ∈ I.

Referring to Fig. 9, read in from outside SDRAM overlook, forward sight, backsight, a left side is looked, the right side is looked video information, select output two-path video signal through video, respectively the two-path video signal is carried out video scaling, pass through video superimpose again, as the left and right sides sprite demonstration of same display terminal.

The present invention can form the vertical view that 360 panoramas show to the scene of vehicle's surroundings or switch the view that shows any side of vehicle, for driver's safe driving or reversing provides real-time video information.

Among the present invention, the correcting fisheye image of realizing being adopted among the present invention with other method, annular image splicing, visual angle conversion method or replace the video decoding chip that adopted among the present invention, dsp chip etc. with other function components and parts and do not influence protection ranges such as proposed by the invention 360 implementation procedures of spending annular image synthesizing method, concrete steps.

Claims (10)

1, the annular image synthesizing method of 360 degree that is used for vehicle assistant drive, it is characterized in that: gather four tunnel real-time vision signals by video acquisition unit (100), video decoding unit (210) and the synthetic control unit (220) of video delivered in the annular video synthesis unit (200) are handled, wherein the synthetic control unit (220) of video carries out correcting fisheye image (221) to four tunnel vision signals, annular image splicing (222), visual angle conversion (223), generate the vertical view that 360 degree panoramas show after the processing of SDRAM storage control (224), through showing that control and memory cell (300) realize plurality of display modes, store video information.

2, according to claims 1 described annular image synthesizing method of 360 degree that is used for vehicle assistant drive, it is characterized in that: video acquisition module unit (100) are made of the super wide-angle fish-eye camera (102) on the super wide-angle fish-eye camera (101) that is installed in headstock grid place, the left-hand mirror, the super wide-angle fish-eye camera (103) on the right back visor, the super wide-angle fish-eye camera (104) at tailstock top, respectively the real time video signals of collection vehicle four sides; The vision signal frame frequency was 30 frame/seconds, and resolution is 720 * 480; Designed short range infrared LEDs lamp around the super wide-angle fish-eye camera, its can by sensor devices automatically induction down control its open and close, thereby be fit to low-light (level) at night, applied environment that light is dark; Super wide-angle fish-eye camera output composite video signal CVBS is to annular video synthesis unit (200).

3, according to claims 1 described annular image synthesizing method of 360 degree that is used for vehicle assistant drive, it is characterized in that:

Its processing procedure of described annular video synthesis unit (200) is: adopt FLASH and the CPLD bootstrapping that powers on, after system's operation, earlier to dsp chip hardware initialization, interrupt vector initialization, by iic bus the video decoding chip of video decoding unit (210) is carried out initialization again, after the initialization success, produce four road 8bits numeral YUV signal, be sent to the synthetic control unit (220) of video, and handle.

4, according to claims 1 described annular image synthesizing method of 360 degree that is used for vehicle assistant drive, it is characterized in that: show that control and memory cell (300) are made of indicative control unit (301) and SD card storage control unit (302).

5, according to claims 3 described annular image synthesizing methods of 360 degree that are used for vehicle assistant drive, it is characterized in that: video decoding unit (210) is used the super wide-angle fish-eye camera (101) of video decoding chip to headstock grid place, super wide-angle fish-eye camera (102) on the left-hand mirror, super wide-angle fish-eye camera (103) on the right back visor, the composite video signal CVBS that the super wide-angle fish-eye camera (104) at tailstock top sends decodes, the 8bits numeral YUV signal of output ITU656 standard YUV4:2:2 form, this four tunnel YUV signal is delivered to the first video inputs mouth VP0 of the synthetic control unit (220) of video, the second video inputs mouth VP1, the 3rd video inputs mouth VP2, the 4th video inputs mouth VP3.

6, according to claims 3 described annular image synthesizing methods of 360 degree that are used for vehicle assistant drive, it is characterized in that: video synthesizes control unit (220) to be realized based on dsp chip, comprise correcting fisheye image unit (221), annular image concatenation unit (222), visual angle converting unit (223) and SDRAM storage control unit (224), video synthesizes control unit (220) the vehicle's surroundings view is generated the vertical view that 360 degree panoramas show through calculation process;

Described video synthesizes control unit (220) processing procedure: adopt correcting fisheye image unit (221) that the image of distortion is proofreaied and correct earlier, four images after the correction are spliced into the panorama sketch of 360 degree through annular image concatenation unit (222), adopt visual angle converting unit (223) to transfer the vertical view of getting a bird's eye view to panorama sketch then, wherein adopt video data and correcting fisheye image unit (221) of SDRAM storage control unit (224) input, annular image concatenation unit (222), the result that visual angle converting unit (223) is handled carries out caching process by outside SDRAM.

7, according to claims 6 described annular image synthesizing methods of 360 degree that are used for vehicle assistant drive, it is characterized in that: correcting fisheye image unit (221) are made up of mapping (221-1) and two steps of interpolation (221-2), and the piecture geometry fault of adopting fish eye lens to cause is proofreaied and correct;

Described mapping (221-1), its process is as follows: in trimming process, the coordinate of establishing benchmark image is (x, y), the fault image coordinate be (u, v), by the benchmark image coordinate being mapped to the correction that realizes image in the fault image, the 2-d polynomial Mathematical Modeling of its mapping is:

$\left\{\begin{array}{c}u=F_u(x,y)=\underset{i=0}{\overset{n}{\mathrm{\&Sigma;}}}\underset{j=0}{\overset{n-i}{\mathrm{\&Sigma;}}}{a}_{\mathrm{ij}}{x}_i{y}_j\\ v=F_v(x,y)=\underset{i=0}{\overset{n}{\mathrm{\&Sigma;}}}\underset{j=0}{\overset{n-i}{\mathrm{\&Sigma;}}}{b}_{\mathrm{ij}}{x}_i{y}_j\end{array}\right.$

In the formula: n is polynomial item number, a _IjAnd b _IjBe every coefficient;

Because the gray value of image same point should remain unchanged, promptly g (x, y)=f (u, v).After finishing mapping step, be decided to be integer, must adopt interpolation (221-2) to calculate because the x, the y that calculate differ;

Described interpolation (221-2) adopts bilinear interpolation method to realize.

8, according to claims 6 described annular image synthesizing methods of 360 degree that are used for vehicle assistant drive, it is characterized in that: annular image concatenation unit (222) is made up of coupling (222-1) and stitching (222-2) two steps, generates automobile panorama displayed map all around;

Described coupling (222-1) adopts " two step couplings " method to realize.

9, according to claims 6 described annular image synthesizing methods of 360 degree that are used for vehicle assistant drive, it is characterized in that: visual angle converting unit (223) utilizes annular image concatenation unit (222) to obtain on the basis of panorama displayed map, carries out the panorama vertical view that contrary perspective projection transformation (223-1) obtains environment surrounding automobile;

Described contrary perspective projection transformation (223-1), implementation step is as follows:

1) S → I mapping is by S plane coordinates point (x _v, y _v, 0), mapping I plane coordinates point (v, u),

$u=\frac{r_{21}{x}_v+r_{22}{y}_v-{\left(R^{-1}{t}_0^c\right)}_2}{r_{11}{x}_v+r_{12}{y}_v-{\left(R^{-1}{t}_0^c\right)}_1}\&times;f_u+u_0$

$v=-\frac{r_{31}{x}_v+r_{32}{y}_v-{\left(R^{-1}{t}_0^c\right)}_3}{r_{11}{x}_v+r_{12}{y}_v-{\left(R^{-1}{t}_0^c\right)}_1}\&times;f_v+u_0$

Wherein $R=\left[\begin{array}{ccc}\cos a\cos \left(b\right)& \sin \left(b\right)& -\sin a\cos \left(b\right)\\ -\cos a\sin \left(b\right)& \cos \left(b\right)& \sin a\sin \left(b\right)\\ \sin a& 0& \cos a\end{array}\right]$ The expression world coordinates is to the rotation relationship of camera coordinate;

$t_0^c={[l,d,h]}^T$ The expression world coordinates is to the translation relation of camera coordinate;

r _IjThe inverse matrix R of expression spin matrix R ^-1The component of the capable j of i row, promptly $R^{-1}=\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right];$

Expression R ^-1With translation matrix

Product gained matrix

I component,

$R^{-1}{t}_0^c=\left[\begin{array}{c}{\left(R^{-1}{t}_0^c\right)}_1\\ {\left(R^{-1}{t}_0^c\right)}_2\\ {\left(R^{-1}{t}_0^c\right)}_3\end{array}\right];$

(u ₀, v ₀) expression video camera principal point coordinate; f _u, f _vExpression U, the focal length scale factor of V axle;

2) on the S plane that calculates of previous step some the location of pixels on image coordinate system I (u v) is not integer usually, and this module adopts the bilinear interpolation method of introducing in the image correction module to calculate pixel (x in the re-projection image _v, y _v) pixel value,

Traversal coordinate system (x _v, y _v, 0) and pel array (forming the re-projection image) among the ∈ W, according to above-mentioned two steps, calculate each the some respective pixel location point [u (x on the re-projection image _v, y _v, 0), v (x _v, y _v, 0)] gray value of ∈ I.

10, according to claims 4 described annular image synthesizing methods of 360 degree that are used for vehicle assistant drive, it is characterized in that: indicative control unit (301) reads in the video information of vehicle's surroundings view from outside SDRAM, select output two-path video information through video, respectively two-path video information is carried out video scaling, pass through video superimpose again, as the left and right sides sprite demonstration (301-1) of same display terminal;

Described left and right sides sprite shows that the display mode of (301-1) comprising: show vehicle body panorama birds-eye view and any end view in left and right sides picture mode on same display screen, be used for the keep to the side application scenario of driving a vehicle, moveing backward or parking of automobile; The view that shows the vehicle body left and right sides on same display screen in left and right sides picture mode is used for the application scenario that automobile travels at the parting of the ways or the narrow road section is current; On same display screen, show vehicle body front and back view, be used for the application scenario of vehicle vehicle condition before and after the monitoring of cruising process in left and right sides picture mode;

Described SD card storage control unit (302) is used for the driver with manual or Automatic Styles recording video information, and the length of its video information is adjusted according to the capacity of SD card.

Images