CN108263283B - Method for calibrating and splicing panoramic all-round looking system of multi-marshalling variable-angle vehicle - Google Patents
Method for calibrating and splicing panoramic all-round looking system of multi-marshalling variable-angle vehicle Download PDFInfo
- Publication number
- CN108263283B CN108263283B CN201810074955.XA CN201810074955A CN108263283B CN 108263283 B CN108263283 B CN 108263283B CN 201810074955 A CN201810074955 A CN 201810074955A CN 108263283 B CN108263283 B CN 108263283B
- Authority
- CN
- China
- Prior art keywords
- carriage
- panoramic
- angle
- image
- splicing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims description 22
- 238000012545 processing Methods 0.000 claims abstract description 7
- 230000002093 peripheral effect Effects 0.000 claims abstract description 5
- 238000013480 data collection Methods 0.000 claims abstract description 3
- 238000012937 correction Methods 0.000 claims description 29
- 238000013507 mapping Methods 0.000 claims description 27
- 240000004050 Pentaglottis sempervirens Species 0.000 claims description 24
- 235000004522 Pentaglottis sempervirens Nutrition 0.000 claims description 24
- 230000004927 fusion Effects 0.000 claims description 21
- 238000001514 detection method Methods 0.000 claims description 14
- 230000007704 transition Effects 0.000 claims description 9
- 239000004744 fabric Substances 0.000 claims description 7
- 230000009466 transformation Effects 0.000 claims description 7
- 230000000007 visual effect Effects 0.000 claims description 7
- 238000007500 overflow downdraw method Methods 0.000 claims description 6
- 230000003068 static effect Effects 0.000 claims description 6
- 238000013519 translation Methods 0.000 claims description 4
- 241000251468 Actinopterygii Species 0.000 claims description 3
- 238000005286 illumination Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 241001300198 Caperonia palustris Species 0.000 description 1
- 235000000384 Veronica chamaedrys Nutrition 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
- G06T3/047—Fisheye or wide-angle transformations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/10—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
- B60R2300/105—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using multiple cameras
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
- B60R2300/303—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using joined images, e.g. multiple camera images
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/60—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
- B60R2300/607—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective from a bird's eye viewpoint
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/80—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
- B60R2300/806—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for aiding parking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Mechanical Engineering (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
- Studio Devices (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
The invention provides a panoramic all-round parking auxiliary system for multi-grouped variable-angle vehicles, which comprises a fisheye camera, a hinged disk angle sensing device and a data acquisition processor, and is characterized in that: the panoramic all-around parking auxiliary system utilizes video pictures collected by two or three fisheye cameras with the azimuth field angles of at least 180 degrees and angle information collected by the angle sensing device of the hinged disk, which are arranged on each vehicle compartment, to be processed by a data collection processor to obtain seamless spliced panoramic all-around images of the peripheral area of the multi-grouped variable-angle vehicle; the angle sensing device of the hinged disk between the carriages is used for measuring the angle information of two carriages hinged by the angle sensing device in real time and transmitting the measured angle to the data acquisition processor in real time for processing. According to the invention, by acquiring and analyzing the hinge angle of the vehicle hinge disc, when the carriage rotates relatively, namely the position of the camera changes relatively, the panoramic stitching parameters are modified in real time, and multi-grouping real-time seamless panoramic stitching is realized.
Description
Technical Field
The invention belongs to the field of vehicle safety assistance and automotive electronics, and particularly relates to a calibration and splicing method for a multi-marshalling variable-angle vehicle panoramic all-round-looking system.
Background
With the rapid increase of automobile reserves, road traffic safety gradually becomes a significant social problem. The panoramic all-round looking system can effectively assist a driver and reduce the accident rate of the vehicle under the condition of low-speed running. At present, most of panoramic all-round parking auxiliary systems in the market are for passenger cars, and four cameras are respectively arranged below a front bumper Logo, below a left rearview mirror and a right rearview mirror of an automobile and at a rear license plate lamp; a small number of panoramic systems for commercial vehicles are provided with four cameras which are respectively arranged at the middle points of the top of the vehicle in four directions. No matter it is passenger car or commercial car panorama system, the relative position of camera is fixed unchangeable in the driving process.
The multi-grouped variable-angle vehicle is provided with a plurality of carriages, and more than 4 cameras are required to be installed so as to realize panoramic splicing; the multi-group variable-angle vehicle is large in vehicle body, the motion of an image object and the change of light rays collected by each camera are different, and a better white balance algorithm is required to realize image fusion; the angle of the multi-group variable-angle vehicle between carriages changes in real time in the running process, so that the relative position between cameras mounted on different carriages also changes in real time. Therefore, the existing panoramic looking-around scheme can not meet the panoramic looking-around requirement of seamless splicing of multi-marshalling angle-variable vehicles.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: the method for calibrating and splicing the seamless splicing panoramic all-around vision system for the multi-marshalling variable-angle vehicles is provided, and an auxiliary tool is provided for driving and parking of the vehicles.
The invention adopts the technical scheme to solve the technical problems, and the panoramic all-round parking auxiliary system for multi-grouped variable-angle vehicles comprises a fisheye camera, a hinged disk angle sensing device and a data acquisition processor, and is characterized in that:
the panoramic all-around parking auxiliary system utilizes video pictures collected by two or three fisheye cameras with the azimuth field angles of at least 180 degrees and angle information collected by the angle sensing device of the hinged disk, which are arranged on each vehicle compartment, to be processed by a data collection processor to obtain seamless spliced panoramic all-around images of the peripheral area of the multi-grouped variable-angle vehicle;
the angle sensing device of the hinged disk between the carriages is used for measuring the angle information of two carriages hinged by the angle sensing device in real time, is connected with the data acquisition processor in a bus mode, and transmits the measured angle to the data acquisition processor in real time for processing.
Furthermore, the data acquisition processor comprises a fisheye image distortion correction unit, an angular point detection unit, a perspective transformation unit, a single carriage image splicing unit, a single carriage image fusion unit, a hinged disc angle acquisition unit, a multi-carriage image splicing unit and a multi-carriage image fusion unit; wherein:
the fisheye image distortion correction unit is used for calibrating camera parameters and distortion correction parameters according to the internal structure of each fisheye camera and the established distortion model to obtain a mapping relation from the fisheye image to the distortion correction image;
the angular point detection unit detects and calibrates angular points in images captured by the fisheye cameras; the perspective transformation unit projects images from different fisheye cameras to the same coordinate system to form a bird's-eye view according to the ideal corner position obtained by measuring a scene and the actual corner position obtained by detection, and establishes a mapping relation from the correction image to the bird's-eye view;
the single carriage image splicing unit is used for matching characteristic points of overlapped areas in the aerial views obtained by converting the adjacent cameras, splicing all the camera aerial views of the single carriage into a single carriage panoramic aerial view, finally generating a mapping relation between the single carriage panoramic aerial view and each camera fish-eye distortion map, and generating and storing a lookup table;
the single-carriage image fusion unit is used for delimiting a transition area at the joint of the images of the adjacent cameras in the panoramic aerial view and eliminating a splicing seam by adopting a weighted fusion method;
the hinged disk angle acquisition unit is connected with the hinged disk angle sensing device through a bus, and receives and analyzes relative angle information between carriages;
the multi-carriage image splicing unit cuts and splices the individual aerial view of each carriage by taking a hinge point as a reference, rotates the individual aerial view of the corresponding carriage according to the angle information of the hinge plate, and simultaneously translates the global aerial view up and down to adapt to the screen display requirement;
and the multi-carriage image fusion unit is used for carrying out global white balance on each camera and carrying out weighted fusion on each edge of each cut single carriage aerial view so as to eliminate a splicing seam for splicing the multi-carriage aerial view.
The invention also provides a method for calibrating and splicing the multi-marshalling variable-angle vehicle panoramic all-round-looking system, which is characterized by comprising the following steps of:
s1, the fisheye image distortion correction unit is used for calibrating camera parameters and distortion correction parameters according to the internal structure of each fisheye camera and the established distortion model to obtain a mapping relation from the fisheye image to the distortion correction image;
s2, detecting a calibration cloth corner point in the image captured by each fisheye camera;
s3, projecting images from different fisheye cameras to the same coordinate system to form a bird 'S-eye view according to the actual corner position obtained by detecting S2 and the ideal corner position obtained by combining a measuring scene, and establishing a mapping relation from a correction image to the bird' S-eye view;
s4, performing feature point matching on overlapped areas in the aerial views obtained by converting the adjacent cameras by the single carriage image splicing unit, splicing all the aerial views of the cameras of the single carriage into a single carriage panoramic aerial view for the first time, finally generating a mapping relation between the single carriage panoramic aerial view and each camera fish eye distortion image, and storing the mapping relation in a lookup table form in a storage unit;
s5, after a lookup table is obtained, obtaining mapping relation information of pixel points of original video pictures collected by the fisheye cameras and pixel points of the single-carriage panoramic aerial view from the storage unit in a table reading mode, and mapping the video pictures collected by all the fisheye cameras into the single-carriage panoramic aerial view in real time;
s6, defining a transition area at the joint of the images of the adjacent cameras in the panoramic aerial view, and eliminating splicing seams by adopting a weighted fusion method;
and S7, cutting the individual bird 'S-eye view of each carriage by taking a hinge point as a reference, and splicing to form the static multi-carriage panoramic bird' S-eye view.
S8, receiving the information of the angle sensing device of the hinged disk through the bus, analyzing to obtain the relative angle information between the carriages, rotating the independent aerial view of the corresponding carriage according to the angle information of the hinged disk, and translating the global aerial view up and down to adapt to the screen display requirement;
s9, carrying out global white balance on the images obtained by capturing all the cameras, and eliminating the illumination influence; and performing weighted fusion on each edge of each cut individual car aerial view to eliminate a splicing seam for splicing the multiple car aerial views.
Further, the S1 adopts a polynomial model as a camera distortion model, and completes camera calibration by using a single fisheye image containing checkerboards.
Further, S2 first detects the corners in the checkerboard array calibration cloth located at the center of the field of view of the fisheye camera, then defines the regions of interest for detecting the black large squares on both sides of the field of view according to the positions of the corners, performs quadrilateral fitting in the regions of interest, eliminates interfering with the quadrilateral contours by limiting the quadrilateral positions, the quadrilateral areas, the quadrilateral shapes, the adjacent side length ratios, and the diagonal ratios, obtains the black large square outline and four vertex coordinates, and determines the positions of the vertices of the black large squares by using a sub-pixel corner detection method around the vertex coordinates.
Further, in S7, the front and rear car birds-eye views are not clipped, the middle car birds-eye view is not clipped in the width direction of the car body, the front car birds-eye view is overlapped with the front-middle car hinge point position in the middle car birds-eye view, the rear car birds-eye view is overlapped with the rear-middle car hinge point position in the middle car birds-eye view, and the two cars are spliced to form the static multi-car panorama bird-eye view.
Further, in S8, the bird' S eye view of the car is not rotated, the front car rotates around the front-middle hinge point according to the angle information of the front-middle hinge disk, and the rear car rotates around the rear-middle hinge point according to the angle information of the rear-middle hinge disk; when the current carriage or the rear carriage exceeds the display range, the panoramic mosaic image is wholly translated to the direction opposite to the steering direction, and when the middle carriage exceeds the display range after translation, the panoramic mosaic image is wholly contracted.
The invention has the beneficial effects that:
1. the existing panoramic viewing system aims at the condition that the relative position of a camera is not changed, and panoramic stitching is realized by using fixed parameters after first calibration and stitching are completed. When a multi-marshalling variable-angle vehicle turns, the positions of the cameras of different carriages are changed relatively, if panoramic stitching is carried out by adopting fixed parameters, severe image dislocation can be generated at the stitching position of the aerial view of the different carriages, and stitching can not be realized. According to the invention, by acquiring and analyzing the hinge angle of the vehicle hinge disc, when the carriage rotates relatively, namely the position of the camera changes relatively, the panoramic stitching parameters are modified in real time, and multi-grouping real-time seamless panoramic stitching is realized.
2. The existing panoramic all-round looking system mostly adopts 4 cameras to carry out panoramic splicing, and a few adopt 6 cameras to carry out panoramic splicing, and do not adopt 8 or more cameras to carry out splicing. However, three compartments of a multi-marshalling vehicle need 8 cameras to complete panoramic stitching. The invention adopts a direct connection transmission method of CVBS video signals, and solves the problem that the transmission bandwidth in the current embedded scheme does not support the simultaneous input of 8 paths of cameras; the single-carriage panoramic stitching is carried out by adopting a lookup table method, the multi-carriage stitching is realized by rotating the single-carriage panoramic aerial view, the calculation amount is reduced, and the real-time performance is ensured.
3. When the multi-grouping panoramic aerial view is displayed, if the middle carriage and the aerial view thereof are placed at the center of the display screen, when the relative angle of the front carriage or the rear carriage is large, the front carriage or the rear carriage partially exceeds the display range of the screen, and the environmental information around the head or the tail of the vehicle cannot be displayed in the display screen. When the situation occurs, the whole multi-group aerial view is moved to the direction opposite to the steering direction; when the middle carriage deviates from the display area after moving for a certain distance, the multi-group aerial view is integrally contracted, so that the vehicle perimeter panoramic information is always kept in the display range.
Drawings
Fig. 1 is a system layout diagram.
Fig. 2 is a flowchart of a panorama stitching method.
Fig. 3 is a detailed method of single-carriage panoramic splicing.
Fig. 4 is a detailed method of multi-car panoramic stitching.
FIG. 5 is a schematic view of a single car splice seam.
FIG. 6 is a schematic view of a multi-car splice seam.
Detailed Description
In order to make the technical solutions of the present invention more comprehensible to those skilled in the art of panoramic parking assist systems, the present invention is further described in detail below with reference to fig. 1 to 6.
The embodiment provides a panoramic all-round parking auxiliary system for multi-grouping variable-angle vehicles, which comprises fisheye cameras, an articulated disc angle sensing device and a data acquisition processor, wherein the panoramic all-round parking auxiliary system utilizes video pictures acquired by two or three fisheye cameras with at least 180-degree azimuth field angles and angle information acquired by the articulated disc angle sensing device, which are installed on each carriage, to process through the data acquisition processor to obtain seamless splicing panoramic all-round images of peripheral areas of the multi-grouping variable-angle vehicles.
The angle sensing device of the hinged disk between the carriages is used for measuring the angle information of two carriages hinged by the angle sensing device in real time, is connected with the data acquisition processor in a bus mode, and transmits the measured angle to the data acquisition processor in real time for processing.
The data acquisition processor comprises a fisheye image distortion correction unit, an angular point detection unit, a perspective transformation unit, a single carriage image splicing unit, a single carriage image fusion unit, a hinged disc angle acquisition unit, a multi-carriage image splicing unit and a multi-carriage image fusion unit; wherein:
the fisheye image distortion correction unit is used for calibrating camera parameters and distortion correction parameters according to the internal structure of each fisheye camera and the established distortion model to obtain a mapping relation from the fisheye image to the distortion correction image;
the angular point detection unit detects and calibrates angular points in images captured by the fisheye cameras; the perspective transformation unit projects images from different fisheye cameras to the same coordinate system to form a bird's-eye view according to the ideal corner position obtained by measuring a scene and the actual corner position obtained by detection, and establishes a mapping relation from the correction image to the bird's-eye view;
the single carriage image splicing unit is used for matching characteristic points of overlapped areas in the aerial views obtained by converting the adjacent cameras, splicing all the camera aerial views of the single carriage into a single carriage panoramic aerial view, finally generating a mapping relation between the single carriage panoramic aerial view and each camera fish-eye distortion map, and generating and storing a lookup table;
the single-carriage image fusion unit is used for delimiting a transition area at the joint of the images of the adjacent cameras in the panoramic aerial view and eliminating a splicing seam by adopting a weighted fusion method;
the hinged disk angle acquisition unit is connected with the hinged disk angle sensing device through a bus, and receives and analyzes relative angle information between carriages;
the multi-carriage image splicing unit cuts and splices the individual aerial view of each carriage by taking a hinge point as a reference, rotates the individual aerial view of the corresponding carriage according to the angle information of the hinge plate, and simultaneously translates the global aerial view up and down to adapt to the screen display requirement;
and the multi-carriage image fusion unit is used for carrying out global white balance on each camera and carrying out weighted fusion on each edge of each cut single carriage aerial view so as to eliminate a splicing seam for splicing the multi-carriage aerial view.
As shown in the drawings, the present embodiment provides a panoramic all-round parking assist system for three-grouped variable-angle vehicles, which obtains a seamless spliced panoramic all-round image of a peripheral area of a multi-grouped variable-angle vehicle by processing a video image acquired by 8 fisheye cameras with azimuth field angles of at least 180 degrees mounted on the three-grouped variable-angle vehicle through a data acquisition processor; wherein the camera arrangement scheme is as follows: for a three-section marshalling angle-variable vehicle as an example, 3 cameras are respectively arranged on a front compartment and a rear compartment, the front compartment is positioned at the midpoint positions of the front side, the left side and the right side of the roof of the compartment, and the rear compartment is positioned at the midpoint positions of the rear side, the left side and the right side of the roof of the compartment; on the middle compartment, 2 cameras are arranged and are positioned at the middle points of the left side and the right side of the roof of the compartment.
The angle sensing device of the hinged disk between the carriages is as follows: and angle information of two articulated carriages of the device is measured in real time, the device is connected with the data acquisition processor in a bus mode, and the measured angle is transmitted to the data acquisition processor in real time for processing.
Taking a three-section carriage vehicle as an example, two hinged disc angle sensing devices are provided for measuring the relative angle of the front and middle carriages and the relative angle of the middle and rear carriages respectively.
In the embodiment, the multi-carriage image fusion unit does not cut the bird's-eye views of the front carriage and the rear carriage, does not cut the bird's-eye view of the middle carriage in the width direction of the vehicle body, and cuts the bird's-eye view of the middle carriage in the length direction of the vehicle body to a position where the front hinge disc and the rear hinge disc are slightly expanded outwards; the middle carriage does not rotate, the front carriage rotates around a front hinge point according to the angle information of the front hinge disk, and the rear carriage rotates around a rear hinge point according to the angle information of the rear hinge disk; the splicing line of the three-carriage aerial view is the edge of the intermediate carriage aerial view: the edge of the three-carriage aerial view is weighted and fused, so that the gradual change effect is realized, and the purposes of eliminating splicing lines and realizing smooth transition are achieved; when the rotating angle is large and the front carriage or the rear carriage exceeds the display range, the whole panoramic mosaic image is translated in the direction opposite to the steering direction, and when the middle carriage exceeds the display range after translation, the whole panoramic mosaic image is shrunk.
The data acquisition processor in the embodiment is formed by a main embedded system and three sub-embedded systems in a cooperative manner, wherein the three sub-embedded systems process image data of a carriage and transmit the data to the main embedded system in a CVBS signal transmission manner, and the main embedded system integrates various information to complete panoramic image splicing.
The embodiment also provides a method for calibrating and splicing by using the multi-group variable-angle vehicle panoramic all-around vision system, which comprises the following steps:
and S1, the fisheye image distortion correction unit is used for calibrating camera parameters and distortion correction parameters according to the internal structure of each fisheye camera and the established distortion model to obtain the mapping relation from the fisheye image to the distortion correction image.
In this embodiment, a polynomial distortion model is used, and the model describes a pixel point correspondence between the corrected ideal image and the original distortion image.
Assumed point (x)u,yu) Is a point in the corrected ideal diagram, (x)d,yd) Is a point in the original distortion map, and the nonlinear correspondence between the two can be represented by the following formula:
xu＝(xd-Cx)F(Λ)+Cx
yu＝(yd-Cy)F(Λ)+Cy
wherein lambda is ═ lambda1,λ2,...,λL]Is the distortion coefficient, Cx,CyIs the center of distortion, and L is typically 2-4. The form of F is as follows.
Where γ is the diagonal length.
In practical application, a fisheye camera to be calibrated is used for shooting a calibration image of a checkerboard corner point, so that the checkerboard occupies the whole visual field as much as possible. Detecting the angular points in the calibration image by using an angular point detection method, then establishing ideal angular point coordinates, and then optimizing a distortion coefficient and a distortion center by using a two-step optimization method, so that coordinates obtained by calculating the angular point coordinates in the fisheye image through parameters obtained by optimization approach to the ideal angular point coordinates. The distortion coefficients and distortion centers are preserved.
And S2, placing calibration cloth around the vehicle, and detecting calibration cloth corner points in the images captured by the fisheye cameras.
The angular point detection method comprises the following steps:
1) detecting angular points in the checkerboard array calibration cloth positioned in the center of the visual field of the fisheye camera;
2) and defining an interested area for detecting the large black square grids by using the relative position relationship between the large black square grids and the checkerboard array. Wherein, in the front and back visual angles, the large black square grids are tightly close to two sides of the checkerboard array; whereas in left and right viewing angles, the large black squares are at the very edge of the field of view. Therefore, the region of interest defining parameters of the front and back visual angles and the left and right visual angles are different;
3) carrying out image preprocessing on the region of interest, including self-adaptive binarization processing and closed operation;
4) and in the region of interest, positioning the large black square outline by adopting a quadrilateral outline fitting method. In the process, other contours except the target contour can be generated, the interference quadrilateral contour is eliminated by limiting the quadrilateral position, the quadrilateral area, the quadrilateral shape, the adjacent side length ratio and the diagonal ratio to obtain a black large square lattice contour and four vertex coordinates, and a sub-pixel angular point detection method is used around the vertex coordinates to determine the accurate black large square lattice vertex position.
And S3, projecting images from different fisheye cameras to the same coordinate system to form a bird 'S-eye view according to the actual corner position obtained by detecting the S2 and the ideal corner position obtained by combining the measured scene, and establishing a mapping relation from the correction image to the bird' S-eye view.
S4, the single carriage image splicing unit performs feature point matching on the overlapped area in the aerial view obtained by converting the adjacent cameras, splices all the camera aerial views of the single carriage into a single carriage panoramic aerial view for the first time, generates the mapping relation between the single carriage panoramic aerial view and each camera fish eye distortion map, and stores the mapping relation in a lookup table form in the storage unit.
And S5, after the lookup table is obtained, obtaining the mapping relation information of the pixel points of the original video images collected by the fisheye cameras and the pixel points of the single-carriage panoramic aerial view from the storage unit by reading the lookup table, and mapping the video images collected by all the fisheye cameras into the single-carriage panoramic aerial view in real time.
In practical application, the distortion correction image is not generated in the fisheye correction stage, the single-camera bird's-eye view is not generated in the perspective transformation stage, and the single-carriage panoramic bird's-eye view is directly generated in the single-carriage image splicing unit. And after the single-carriage bird's-eye view is obtained for the first time, the lookup table is obtained, and then the collected fisheye data can be directly spliced into the single-carriage bird's-eye view only by moving the image data according to the lookup table.
S6, defining a transition area at the joint of the images of the adjacent cameras in the panoramic aerial view, and eliminating splicing seams by adopting a weighted fusion method;
an oblique angle is adopted between adjacent cameras as a splicing seam, and a transition area is divided into a rectangular area taking two end points of the splicing seam as diagonal vertexes. The method adopts the weighted fusion of the different angles.
And S7, cutting the individual bird 'S-eye view of each carriage by taking a hinge point as a reference, and splicing to form the static multi-carriage panoramic bird' S-eye view.
The aerial views of the front carriage and the rear carriage are not cut, the aerial view of the middle carriage is not cut in the width direction of the vehicle body, and the aerial view of the middle carriage is cut in the length direction of the vehicle body to a position where the front hinge disc and the rear hinge disc are expanded outwards a little. And overlapping the front-middle carriage hinge point position in the front carriage aerial view and the middle carriage aerial view, overlapping the rear-middle carriage hinge point position in the rear carriage aerial view and the middle carriage aerial view, and splicing into a static multi-carriage panoramic aerial view.
S8, receiving the information of the hinged disk angle sensing device through the bus, analyzing to obtain the relative angle information between the carriages, rotating the independent aerial view of the corresponding carriage according to the angle information of the hinged disk, and translating the global aerial view up and down to meet the screen display requirement.
The aerial view of the middle carriage does not rotate, the front carriage rotates around the front-middle hinge point according to the angle information of the front-middle hinge disk, and the rear carriage rotates around the rear-middle hinge point according to the angle information of the rear-middle hinge disk. When the rotating angle is large and the front carriage or the rear carriage exceeds the display range, the whole panoramic mosaic image is translated in the direction opposite to the steering direction, and when the middle carriage exceeds the display range after translation, the whole panoramic mosaic image is shrunk.
And S9, carrying out global white balance on the images obtained by capturing all the cameras, and eliminating the illumination influence. And performing weighted fusion on each edge of each cut individual car aerial view to eliminate a splicing seam for splicing the multiple car aerial views.
The splicing line of the multi-carriage panoramic aerial view is the edge of the intermediate carriage aerial view: and adopting an image transparency change method for the edges of the three-carriage aerial view, taking the distance from the edges of the images as a standard, the image edge transparency as 1, and taking the position transparency at a distance from the edges of the images as 0, thereby realizing the gradual change effect and achieving the purposes of eliminating splicing lines and realizing smooth transition.
Equivalent changes or modifications of the principles and concepts disclosed herein may be made within the scope of the invention, the invention is not limited to the details shown and described herein, and it is understood that various changes or modifications may be made without departing from the spirit and scope of the invention.
Although the present invention has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that the present invention is not limited to the specific embodiments, but rather, various modifications, alterations, and substitutions can be made without departing from the spirit and scope of the present application.
Claims (3)
1. The utility model provides a panorama look around parking auxiliary system for organizing more variable angle vehicle, includes fisheye camera, articulated disk angle sensing device and data acquisition treater, its characterized in that:
the panoramic all-around parking auxiliary system utilizes video pictures collected by two or three fisheye cameras with the azimuth field angles of at least 180 degrees and angle information collected by the angle sensing device of the hinged disk, which are arranged on each vehicle compartment, to be processed by a data collection processor to obtain seamless spliced panoramic all-around images of the peripheral area of the multi-grouped variable-angle vehicle;
the angle sensing device of the hinged disc between the carriages is used for measuring the angle information of two carriages hinged by the angle sensing device in real time, is connected with the data acquisition processor in a bus mode, and transmits the measured angle to the data acquisition processor in real time for processing;
the data acquisition processor comprises a fisheye image distortion correction unit, an angular point detection unit, a perspective transformation unit, a single carriage image splicing unit, a single carriage image fusion unit, a hinged disc angle acquisition unit, a multi-carriage image splicing unit and a multi-carriage image fusion unit; wherein:
the fisheye image distortion correction unit is used for calibrating camera parameters and distortion correction parameters according to the internal structure of each fisheye camera and the established distortion model to obtain a mapping relation from the fisheye image to the distortion correction image;
the angular point detection unit detects and calibrates angular points in images captured by the fisheye cameras; the perspective transformation unit projects images from different fisheye cameras to the same coordinate system to form a bird's-eye view according to the ideal corner position obtained by measuring a scene and the actual corner position obtained by detection, and establishes a mapping relation from the correction image to the bird's-eye view;
the single carriage image splicing unit is used for matching characteristic points of overlapped areas in the aerial views obtained by converting the adjacent cameras, splicing all the camera aerial views of the single carriage into a single carriage panoramic aerial view, finally generating a mapping relation between the single carriage panoramic aerial view and each camera fish-eye distortion map, and generating and storing a lookup table;
the single-carriage image fusion unit is used for delimiting a transition area at the joint of the images of the adjacent cameras in the panoramic aerial view and eliminating a splicing seam by adopting a weighted fusion method;
the hinged disk angle acquisition unit is connected with the hinged disk angle sensing device through a bus, and receives and analyzes relative angle information between carriages;
the multi-carriage image splicing unit cuts and splices the individual aerial view of each carriage by taking a hinge point as a reference, rotates the individual aerial view of the corresponding carriage according to the angle information of the hinge plate, and simultaneously translates the global aerial view up and down to adapt to the screen display requirement;
and the multi-carriage image fusion unit is used for carrying out global white balance on each camera and carrying out weighted fusion on each edge of each cut single carriage aerial view so as to eliminate a splicing seam for splicing the multi-carriage aerial view.
2. The method for calibrating and splicing the panoramic all-round parking assist system for the multi-group variable-angle vehicle as claimed in claim 1 is characterized by comprising the following steps:
s1, the fisheye image distortion correction unit is used for calibrating camera parameters and distortion correction parameters according to the internal structure of each fisheye camera and the established distortion model to obtain a mapping relation from the fisheye image to the distortion correction image; the S1 adopts a polynomial model as a camera distortion model, and completes camera calibration by using a single fisheye image containing checkerboards;
s2, detecting a calibration cloth corner point in the image captured by each fisheye camera; s2, firstly detecting the angular points in the checkerboard array calibration cloth positioned in the center of the visual field of the fisheye camera, then defining an interested area for detecting black large squares on two sides of the visual field according to the angular point positions, carrying out quadrilateral fitting in the interested area, eliminating interference quadrilateral outlines by limiting quadrilateral positions, quadrilateral areas, quadrilateral shapes, adjacent side length ratios and diagonal ratios to obtain black large square grid outlines and four vertex coordinates, and determining the positions of the vertices of the black large square grids by using a sub-pixel angular point detection method around the vertex coordinates;
s3, projecting images from different fisheye cameras to the same coordinate system to form a bird 'S-eye view according to the actual corner position obtained by detecting S2 and the ideal corner position obtained by combining a measuring scene, and establishing a mapping relation from a correction image to the bird' S-eye view;
s4, performing feature point matching on overlapped areas in the aerial views obtained by converting the adjacent cameras by the single carriage image splicing unit, splicing all the aerial views of the cameras of the single carriage into a single carriage panoramic aerial view for the first time, finally generating a mapping relation between the single carriage panoramic aerial view and each camera fish eye distortion image, and storing the mapping relation in a lookup table form in a storage unit;
s5, after a lookup table is obtained, obtaining mapping relation information of pixel points of original video pictures collected by the fisheye cameras and pixel points of the single-carriage panoramic aerial view from the storage unit in a table reading mode, and mapping the video pictures collected by all the fisheye cameras into the single-carriage panoramic aerial view in real time;
s6, defining a transition area at the joint of the images of the adjacent cameras in the panoramic aerial view, and eliminating splicing seams by adopting a weighted fusion method;
s7, cutting the individual aerial view of each carriage by taking a hinge point as a reference, and splicing to form a static multi-carriage panoramic aerial view;
s8, receiving the information of the angle sensing device of the hinged disk through the bus, analyzing to obtain the relative angle information between the carriages, rotating the independent aerial view of the corresponding carriage according to the angle information of the hinged disk, and translating the global aerial view up and down to adapt to the screen display requirement; the front carriage rotates around a front-middle hinge point according to the angle information of the front-middle hinge disc, and the rear carriage rotates around a rear-middle hinge point according to the angle information of the rear-middle hinge disc; when the front carriage or the rear carriage exceeds the display range, the whole panoramic mosaic image is translated in the direction opposite to the steering direction, and when the middle carriage exceeds the display range after translation, the whole panoramic mosaic image is contracted;
s9, carrying out global white balance on the images obtained by capturing all the cameras, and eliminating the illumination influence; and performing weighted fusion on each edge of each cut individual car aerial view to eliminate a splicing seam for splicing the multiple car aerial views.
3. The method of claim 2, wherein in step S7, the front and rear car birds-eye views are not cut, the middle car birds-eye view is not cut in the width direction of the car body, the front car birds-eye view is overlapped with the front-middle car hinge point in the middle car birds-eye view, and the rear car birds-eye view is overlapped with the rear-middle car hinge point in the middle car birds-eye view, and the two cars are spliced into a static multi-car panorama.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810074955.XA CN108263283B (en) | 2018-01-25 | 2018-01-25 | Method for calibrating and splicing panoramic all-round looking system of multi-marshalling variable-angle vehicle |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810074955.XA CN108263283B (en) | 2018-01-25 | 2018-01-25 | Method for calibrating and splicing panoramic all-round looking system of multi-marshalling variable-angle vehicle |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108263283A CN108263283A (en) | 2018-07-10 |
| CN108263283B true CN108263283B (en) | 2020-01-14 |
Family
ID=62776812
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810074955.XA Active CN108263283B (en) | 2018-01-25 | 2018-01-25 | Method for calibrating and splicing panoramic all-round looking system of multi-marshalling variable-angle vehicle |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108263283B (en) |
Families Citing this family (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110826364B (en) * | 2018-08-09 | 2024-02-02 | 上海汽车集团股份有限公司 | Library position identification method and device |
| CN109544647A (en) * | 2018-11-30 | 2019-03-29 | 郑州天迈科技股份有限公司 | Calibration cloth, place and method for 360 ° of panoramic parking assist systems |
| CN109532671A (en) * | 2018-12-12 | 2019-03-29 | 广州小鹏汽车科技有限公司 | Track of vehicle processing method, device, electronic equipment and automobile |
| CN109767473B (en) * | 2018-12-30 | 2022-10-28 | 惠州华阳通用电子有限公司 | Panoramic parking device calibration method and device |
| CN109905613B (en) * | 2019-01-21 | 2021-05-25 | 广州市安晓科技有限责任公司 | Automobile panoramic stitching method, device and medium for avoiding door opening interference |
| CN110060200B (en) * | 2019-03-18 | 2023-05-30 | 创新先进技术有限公司 | Image perspective transformation method, device and equipment |
| CN111856871A (en) * | 2019-04-26 | 2020-10-30 | 东莞潜星电子科技有限公司 | Vehicle-mounted 3D panoramic all-around display method |
| CN111912374A (en) * | 2019-05-08 | 2020-11-10 | 中车株洲电力机车研究所有限公司 | Vehicle articulation angle measuring system and method and application vehicle |
| CN110378201A (en) * | 2019-06-05 | 2019-10-25 | 浙江零跑科技有限公司 | A kind of hinged angle measuring method of multiple row vehicle based on side ring view fisheye camera input |
| CN110363085B (en) * | 2019-06-10 | 2021-11-09 | 浙江零跑科技股份有限公司 | Method for realizing looking around of heavy articulated vehicle based on articulation angle compensation |
| CN110781883B (en) * | 2019-09-18 | 2024-03-19 | 江苏大学 | Parking space size recognition system and method based on 360-degree looking-around camera |
| CN110751598B (en) * | 2019-10-17 | 2023-09-26 | 长沙智能驾驶研究院有限公司 | Vehicle hinge point coordinate calibration method and device, computer equipment and storage medium |
| CN111179168B (en) * | 2019-12-12 | 2023-11-17 | 河南嘉晨智能控制股份有限公司 | Vehicle-mounted 360-degree panoramic all-around monitoring system and method |
| CN110719411B (en) * | 2019-12-16 | 2020-04-03 | 长沙智能驾驶研究院有限公司 | Panoramic all-around view image generation method of vehicle and related equipment |
| CN111369439B (en) * | 2020-02-29 | 2023-05-23 | 华南理工大学 | Panoramic all-around image real-time splicing method for automatic parking space identification based on all-around |
| CN111361502B (en) * | 2020-03-23 | 2021-04-13 | 长沙立中汽车设计开发股份有限公司 | Implementation method for variable-angle panoramic looking-around system of articulated engineering vehicle |
| CN111559314B (en) * | 2020-04-27 | 2021-08-24 | 长沙立中汽车设计开发股份有限公司 | Depth and image information fused 3D enhanced panoramic looking-around system and implementation method |
| CN111546986A (en) * | 2020-04-30 | 2020-08-18 | 北京大椽科技有限公司 | Trailer panoramic looking-around method |
| CN111942391A (en) * | 2020-09-15 | 2020-11-17 | 徐工集团工程机械股份有限公司道路机械分公司 | Hinged engineering machinery, panoramic looking-around system and calibration method thereof |
| CN112348741A (en) * | 2020-11-03 | 2021-02-09 | 明见（厦门）技术有限公司 | Panoramic image splicing method, panoramic image splicing equipment, storage medium, display method and display system |
| CN112967183A (en) * | 2021-01-30 | 2021-06-15 | 惠州华阳通用电子有限公司 | Panoramic picture splicing method and storage medium |
| CN112793564B (en) * | 2021-02-07 | 2022-07-22 | 杨东杰 | Autonomous parking auxiliary system based on panoramic aerial view and deep learning |
| CN112793509B (en) * | 2021-04-14 | 2021-07-23 | 天津所托瑞安汽车科技有限公司 | Blind area monitoring method, device and medium |
| CN113255468A (en) * | 2021-05-06 | 2021-08-13 | 东风汽车集团股份有限公司 | Vehicle-mounted panoramic image calibration block identification optimization method and system and readable storage medium |
| CN113538377B (en) * | 2021-07-15 | 2022-08-12 | 河北三国新能源科技有限公司 | Driving test vehicle quarter turn line pressing detection method and system based on panoramic looking-around |
| CN113870101B (en) * | 2021-12-02 | 2022-03-08 | 交通运输部公路科学研究所 | Panoramic all-around image splicing method and device for articulated vehicle |
| CN115412676A (en) * | 2022-09-13 | 2022-11-29 | 清华大学 | Panoramic looking-around system based on weight mask and panoramic image fusion method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103763517A (en) * | 2014-03-03 | 2014-04-30 | 惠州华阳通用电子有限公司 | Vehicle-mounted around view display method and system |
| CN104732542A (en) * | 2015-03-27 | 2015-06-24 | 安徽省道一电子科技有限公司 | Image processing method for panoramic vehicle safety system based on multi-camera self calibration |
| CN104851076A (en) * | 2015-05-27 | 2015-08-19 | 武汉理工大学 | Panoramic 360-degree-view parking auxiliary system for commercial vehicle and pick-up head installation method |
| CN105096317A (en) * | 2015-07-03 | 2015-11-25 | 吴晓军 | Fully automatic calibration method for high performance camera under complicated background |
| CN106408511A (en) * | 2016-09-22 | 2017-02-15 | 苏州大学 | Overlook conversion method, overlook image acquisition method and mapping table construction method of fisheye image |
| CN107154022A (en) * | 2017-05-10 | 2017-09-12 | 北京理工大学 | A kind of dynamic panorama mosaic method suitable for trailer |
-
2018
- 2018-01-25 CN CN201810074955.XA patent/CN108263283B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103763517A (en) * | 2014-03-03 | 2014-04-30 | 惠州华阳通用电子有限公司 | Vehicle-mounted around view display method and system |
| CN104732542A (en) * | 2015-03-27 | 2015-06-24 | 安徽省道一电子科技有限公司 | Image processing method for panoramic vehicle safety system based on multi-camera self calibration |
| CN104851076A (en) * | 2015-05-27 | 2015-08-19 | 武汉理工大学 | Panoramic 360-degree-view parking auxiliary system for commercial vehicle and pick-up head installation method |
| CN105096317A (en) * | 2015-07-03 | 2015-11-25 | 吴晓军 | Fully automatic calibration method for high performance camera under complicated background |
| CN106408511A (en) * | 2016-09-22 | 2017-02-15 | 苏州大学 | Overlook conversion method, overlook image acquisition method and mapping table construction method of fisheye image |
| CN107154022A (en) * | 2017-05-10 | 2017-09-12 | 北京理工大学 | A kind of dynamic panorama mosaic method suitable for trailer |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108263283A (en) | 2018-07-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108263283B (en) | Method for calibrating and splicing panoramic all-round looking system of multi-marshalling variable-angle vehicle | |
| CN110381255B (en) | Vehicle-mounted video monitoring system and method applying 360-degree panoramic looking-around technology | |
| US11553153B2 (en) | Display system and method | |
| CN109435852B (en) | Panoramic auxiliary driving system and method for large truck | |
| CN109741455B (en) | Vehicle-mounted stereoscopic panoramic display method, computer readable storage medium and system | |
| CN111369439B (en) | Panoramic all-around image real-time splicing method for automatic parking space identification based on all-around | |
| US9858639B2 (en) | Imaging surface modeling for camera modeling and virtual view synthesis | |
| CN109087251B (en) | Vehicle-mounted panoramic image display method and system | |
| CN108765496A (en) | A kind of multiple views automobile looks around DAS (Driver Assistant System) and method | |
| CN104851076A (en) | Panoramic 360-degree-view parking auxiliary system for commercial vehicle and pick-up head installation method | |
| CN101442618A (en) | Method for synthesizing 360 DEG ring-shaped video of vehicle assistant drive | |
| CN112224132A (en) | Vehicle panoramic all-around obstacle early warning method | |
| CN110363085B (en) | Method for realizing looking around of heavy articulated vehicle based on articulation angle compensation | |
| CN112233188B (en) | Calibration method of data fusion system of laser radar and panoramic camera | |
| CN113362228A (en) | Method and system for splicing panoramic images based on improved distortion correction and mark splicing | |
| CN110689506A (en) | Panoramic stitching method, automotive panoramic stitching method and panoramic system thereof | |
| CN107364393A (en) | Display methods, device, storage medium and the electronic equipment of vehicle rear view image | |
| CN112348741A (en) | Panoramic image splicing method, panoramic image splicing equipment, storage medium, display method and display system | |
| CN115239922A (en) | AR-HUD three-dimensional coordinate reconstruction method based on binocular camera | |
| CN114640801A (en) | Vehicle-end panoramic view angle auxiliary driving system based on image fusion | |
| CN112802109B (en) | Method for generating aerial view panorama of automobile | |
| CN111626227A (en) | Method for realizing vehicle bottom perspective panoramic system based on binocular vision | |
| CN110667477A (en) | Vehicle night video processing system and method | |
| CN113850905B (en) | Panoramic image real-time stitching method for circumferential scanning type photoelectric early warning system | |
| CN113724133B (en) | 360-degree circular splicing method for non-rigid body connected trailer |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |