CN106952311A - Assisted parking system and method based on panoramic stitching data mapping table - Google Patents

Abstract

The invention discloses an auxiliary parking method and system based on a panoramic splicing data mapping table. Calibration cloth is arranged on the ground around the vehicle body, and the layout and size information of the checkerboard area and the public coverage area on the calibration cloth are used to install the vehicle on the vehicle body. The images in the coverage areas of the adjacent cameras on the body are roughly registered; then the images in the coverage areas of the adjacent cameras are fine-tuned by using the object features in the common coverage area, and the non-similarity of the image features is used. When the non-similarity sum of the image features collected by the camera reaches the minimum, find the optimal image registration position; use the panoramic stitching template image to generate a panoramic stitching data mapping table, and generate a panoramic video in real time according to the file containing the data mapping relationship table image. The use of the present invention can effectively improve the registration accuracy of the panoramic calibration image, improve the splicing and fusion effect of the panoramic video image, and effectively meet the real-time requirements of hardware.

Description

Assisted parking system and method based on panoramic stitching data mapping table

technical field

The invention relates to the technical field of image processing, in particular to an auxiliary parking system and method based on a panoramic stitching data mapping table.

Background technique

The panoramic assisted parking system collects images around the vehicle body in real time through four fisheye cameras installed on the front, rear, left, and right sides of the vehicle body. The image processing unit performs distortion correction on the four fisheye images, and then transforms the four fisheye images into one Under the spatial coordinate system, a panoramic image is then generated and transmitted to the center console display device. The user can intuitively see the environment around the car body in the car, which can effectively improve driving safety.

Since the input original image of the panoramic assisted parking system is obtained by multiple ultra-wide-angle vehicle cameras, and requires distortion correction and homography transformation for system calibration, this will lead to large differences in object deformation and color. No registration algorithm can achieve the ideal registration effect. Finding an optimal image registration position is a difficult point in the implementation of the panoramic assisted parking system. Due to the special application requirements of the system, the panoramic assisted parking system needs the hardware device of the system to meet the real-time requirements in practical applications.

Contents of the invention

In order to solve the deficiencies of the prior art, the present invention provides an auxiliary parking method based on a panoramic stitching data mapping table, with the purpose of generating a panoramic stitching data mapping table using a panoramic stitching template image.

Another object of the present invention is to provide an auxiliary parking system based on a panoramic stitching data mapping table, which can generate panoramic video images in real time according to a file containing a data mapping table.

An assisted parking method based on a panoramic stitching data mapping table, including:

Set a calibration cloth on the ground around the vehicle body, and use the layout and size information of the checkerboard area and the public coverage area on the calibration cloth to roughly register the images in the coverage area of the adjacent camera installed on the vehicle body;

Then use the object features in the common coverage area to fine-tune the images in the coverage areas of adjacent cameras, and use the dissimilarity of image features to minimize the sum of the dissimilarity of all image features collected by different cameras, find Optimal image registration position;

The panorama stitching data mapping table is generated by using the panorama mosaic template image, and the panorama video image can be generated in real time according to the file containing the data mapping relationship table.

Further, 4-way fisheye ultra-wide-angle cameras are installed on the front, left, rear and right of the vehicle body to be calibrated.

Further, use the camera to obtain four fisheye calibration images A ₀ (x ₀₀ ,y ₀₀ ), A ₁ (x ₁₀ ,y ₁₀ ), A ₂ (x ₂₀ ,y ₂₀ ), A ₃ (x ₃₀ ,y ₃₀ ), perform distortion correction on these four fisheye calibration images, and obtain four distortion-corrected fisheye calibration images B ₀ (x ₀₁ ,y ₀₁ ), B ₁ (x ₁₁ , y ₁₁ ), B ₂ (x ₂₁ ,y ₂₁ ), B ₃ (x ₃₁ ,y ₃₁ ).

Further, identify the checkerboard interior corners in B ₀ (x ₀₁ ,y ₀₁ ), B ₁ (x ₁₁ ,y ₁₁ ), B ₂ (x ₂₁ ,y ₂₁ ), B ₃ (x ₃₁ ,y ₃₁ ) images respectively point, get the world coordinates and image coordinates of the inner corner points of the checkerboard in the image, and calculate B ₀ (x ₀₁ ,y ₀₁ ), B ₁ (x ₁₁ ,y ₁₁ ) according to the world coordinates and image coordinates of the inner corner points of the checkerboard , B ₂ (x ₂₁ , y ₂₁ ), B ₃ (x ₃₁ , y ₃₁ ) image homography transformation matrix H ₀ , H ₁ , H ₂ , H ₃ , according to the homography transformation matrix, image B ₀ (x ₀₁ ,y ₀₁ ), B ₁ (x ₁₁ ,y ₁₁ ), B ₂ (x ₂₁ ,y ₂₁ ), and B ₃ (x ₃₁ ,y ₃₁ ) perform bird’s-eye view transformation respectively to obtain four bird’s-eye view calibration images C ₀ (x ₀₂ ,y ₀₂ ), C ₁ (x ₁₂ ,y ₁₂ ), C ₂ (x ₂₂ ,y ₂₂ ), C ₃ (x ₃₂ ,y ₃₂ ).

Furthermore, the homography transformation needs to ensure that the image size of a single checkerboard involved in the four bird's-eye view calibration images is the same, and the direction of the checkerboard in the bird's-eye view calibration image is consistent with the direction of the panorama stitching.

Further, the method of generating the panoramic stitching template image is as follows: according to the actual size of the checkerboard area of the calibration cloth, the actual vertical distance between the checkerboard areas, the image size information of the checkerboard area, and the display distance of the panoramic bird's-eye view calibration image, Generate panorama stitching template images.

Further, the panorama _stitching template image is composed of five blocks: the central area R _cent , the front area _RA , the left area _RB , the rear area RC , and the right area RD. The central area _{R cent} is a car model image, and the front area Region R _A , left region R _B , rear region R _C , and right region R _D respectively correspond to the bird's-eye view calibration images C ₀ (x ₀₂ ,y ₀₂ ), C ₁ (x ₁₂ ,y ₁₂ ), C ₂ (x ₂₂ , y ₂₂ ) and C ₃ (x ₃₂ , y ₃₂ ), corresponding to each other, according to the size of the checkerboard area, determine the splicing gap of the two bird's-eye view calibration images, and the rectangular area corresponding to the splicing gap is the overlapping of the two bird's-eye view calibration images area.

Further, according to the position of the stitching line and the image position of the checkerboard in the bird's-eye view image, the four bird's-eye view calibration images are stitched into a complete panoramic bird's-eye view calibration image D(x ₃ , y ₃ ).

Further, the finding of the optimal image registration position specifically includes: determining the overlapping area of adjacent bird's-eye view calibration images, performing feature extraction on the overlapping area images of adjacent bird's-eye view calibration images, and according to the image features of the two overlapping areas , fine-tune the positions of the two bird's-eye view calibration images, and search for the best registration position of two adjacent bird's-eye view images, and finally make all the non-similarities of image features collected by different cameras in the four overlapping areas and reach a minimum.

Further, a panorama stitching map file is generated, and the panorama stitching map consists of the following parts:

F0: Pixel coordinate mapping table M ₀ between the bird's-eye view calibration image C ₀ (x ₀₂ ,y ₀₂ ) and the front fish-eye calibration image A ₀ (x ₀₀ ,y ₀₀ );

F1: Pixel coordinate mapping table M ₁ between the bird's-eye view calibration image C ₁ (x ₁₂ ,y ₁₂ ) and the left fish-eye calibration image A ₁ (x ₁₀ ,y ₁₀ );

F2: Pixel coordinate mapping table M ₂ between the bird's-eye calibration image C ₂ (x ₂₂ , y ₂₂ ) and the rear fish-eye calibration image A ₂ (x ₂₀ , y ₂₀ );

F3: Pixel coordinate mapping table M ₃ between the bird's-eye calibration image C ₃ (x ₃₂ , y ₃₂ ) and the right fish-eye calibration image A ₃ (x ₃₀ , y ₃₀ );

F4: Mapping table M ₄ of the region between the panoramic bird's-eye view calibration image D(x ₃ ,y ₃ ) and the bird's-eye view calibration image in 4 directions. The mapping table M ₄ can be expressed as formula (1), where R ₀ , R ₁ , R ₂ , R ₃ are bird's-eye view calibration images C ₀ (x ₀₂ ,y ₀₂ ), C ₁ (x ₁₂ ,y ₁₂ ), C ₂ (x ₂₂ ,y ₂₂ ), C ₃ (x ₃₂ ,y ₃₂ ) , and does not belong to the fusion area of adjacent bird's-eye calibration images in the panoramic video image, the fusion area of adjacent bird-eye calibration images is the sub-image of the overlapping area of adjacent bird's-eye calibration images, R ₄ , R ₅ , R ₆ , R ₇ belongs to the fusion area of adjacent bird's-eye view calibration images, and performs fusion processing on the overlapping area images in the fusion area, where R ₄ is the bird's-eye view calibration images C ₀ (x ₀₂ ,y ₀₂ ), C ₁ (x ₁₂ ,y ₁₂ ) R ₅ is the overlapping area sub-image of the bird's-eye view calibration image C ₁ (x ₁₂ ,y ₁₂ ), C ₂ (x ₂₂ ,y ₂₂ ), R ₆ is the bird's-eye view calibration image C ₂ (x ₂₂ ,y y ₂₂ ), the overlapping area sub-images of C ₃ (x ₃₂ , y ₃₂ ), R ₇ is the overlapping area sub-image of the bird's-eye view calibration image C ₀ (x ₀₂ , y ₀₂ ), C ₃ (x ₃₂ , y ₃₂ );

F5: Panoramic bird's-eye view calibration image D(x ₃ ,y ₃ ) and 4-direction bird's-eye view calibration images C ₀ (x ₀₂ ,y ₀₂ ), C ₁ (x ₁₂ ,y ₁₂ ), C ₂ (x ₂₂ ,y ₂₂ ) , C ₃ (x ₃₂ , y ₃₂ ), the pixel point coordinate mapping table M ₅ , and for any pixel point (x′ ₃ , y′ ₃ ) in the adjacent bird’s-eye view calibration image fusion area, it needs to be explained (x′ ₃ ,y′ ₃ ) and the coordinate mapping relationship between the two bird's-eye view image pixels;

F6: Determine the fusion factors h ₁ and h ₂ of pixels in the fusion regions R ₄ , R ₅ , R ₆ , and R ₇ of adjacent bird’s-eye view calibration images, where h ₁ and h ₂ are the corresponding coordinates of the overlapping parts of the bird’s-eye view images Lower the weight of the pixel, store the coordinates (x′ ₃ , y′ ₃ ) of any pixel involved in the fusion area R ₄ , R ₅ , R ₆ , R ₇ and the fusion factor of the pixel into the mapping table M ₆ .

Further, the calibration cloth is composed of a checkerboard area and a public coverage area. The public coverage area is mainly used as a marker for panoramic stitching, and the front calibration cloth, For the rear calibration cloth, left calibration cloth and right calibration cloth, ensure that the checkerboard rectangular areas of adjacent calibration cloths are perpendicular to each other, and ensure that the calibration cloth is within the visual range of the front, rear, left, and right fisheye cameras. The direction of the calibration cloth, using the camera to obtain the calibration image information around the car.

Further, the front region R _A of the panoramic stitching template image is composed of regions R ₀ and R ₇ , the left region R _B is composed of regions R ₁ and R ₄ , the rear region R _C is composed of regions R ₂ and R ₅ , and the right region R B is composed of regions R 2 and R 5 . Square region _RD is composed of regions R ₃ , R ₆ .

Further, the implementation steps of generating panoramic video images in real time: After obtaining the 4-way fisheye real-time images around the vehicle body in real time, for the pixels in the R ₀ , R ₁ , R ₂ , and R ₃ areas, according to the data mapping table generated by the calibration system File, mapped to the coordinates of the original fisheye image to which it belongs, to determine the output pixel value, for the pixels in the R ₄ , R ₅ , R ₆ , and R ₇ areas, map to the original fisheye image to be involved in the calculation according to the data mapping table file According to the fusion factor determined by the mapping table, the pixel is fused to determine the output pixel value.

Assisted parking system based on panoramic stitching data mapping table, including:

The image acquisition module uses 4-way fisheye ultra-wide-angle cameras installed in the front, left, rear and right of the vehicle body to acquire images and perform synchronous processing on the images;

Judgment module, used to judge whether the image after synchronous processing needs to be calibrated, if necessary, the image after synchronous processing is transmitted to the system calibration module, otherwise, it is transferred to the panorama generation module;

The system calibration module performs distortion correction on the synchronously processed front video image, rear video image, left video image and right video image respectively, and then performs bird's-eye view transformation respectively, performs panoramic stitching on the bird's-eye transformed images and generates a panoramic stitching data map surface;

The panorama generation module is used to remap and transform the video image, and then process the transformed image and transmit it to the display module for image display.

Compared with prior art, the beneficial effect of the present invention is:

The panoramic assisted parking system of the present invention uses the calibration cloth information to roughly register the bird's-eye view calibration image, uses the object characteristics in the public coverage area to fine-tune the image, and realizes the optimal registration of the bird's-eye view calibration image. A template for generating a panorama stitching data mapping table using the panorama stitching data mapping.

The use of the present invention can effectively improve the registration accuracy of the panoramic calibration image, improve the splicing and fusion effect of the panoramic video image, and effectively meet the real-time requirements of hardware.

The invention solves the key problem in the panoramic parking system in combination with the actual demand in the development of panoramic parking products.

Description of drawings

The accompanying drawings constituting a part of the present application are used to provide further understanding of the present application, and the schematic embodiments and descriptions of the present application are used to explain the present application, and do not constitute improper limitations to the present application.

Fig. 1 is a module block diagram of a panoramic assisted parking system;

Figure 2 is a schematic diagram of the location of the calibration cloth;

Figure 3 is a panorama stitching template diagram;

Fig. 4 is a view of the display range of panoramic stitching bird's-eye view images.

detailed description

It should be pointed out that the following detailed description is exemplary and intended to provide further explanation to the present application. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terminology used here is only for describing specific implementations, and is not intended to limit the exemplary implementations according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural, and it should also be understood that when the terms "comprising" and/or "comprising" are used in this specification, they mean There are features, steps, operations, means, components and/or combinations thereof.

As introduced in the background technology, there are deficiencies in the prior art. In order to solve the above technical problems, the present application proposes a parking assistance method and system based on a panoramic stitching data mapping table.

The working process of the panoramic assisted parking system can be divided into two subsystems. One of the working subsystems is the calibration system, which is calibrated after the system is installed for the first time. Image processing is performed on the 4-channel fisheye image to generate a panoramic stitching image. The final The purpose is to generate a data mapping table between the original fisheye image and the panoramic stitching image. Another working subsystem is to use the data mapping table generated by the calibration system after the system is calibrated and the 4-channel fisheye real-time image is obtained. Panoramic live images.

In a typical implementation of the present application, the embodiment of the present invention adopts the following system design scheme: use a PC to calibrate the panoramic assisted parking system, and determine the data mapping between the calibration image and the panoramic stitching image acquired by the fisheye ultra-wide-angle camera table, and write the data mapping table into a text file, and then copy the file to a storage device using a USB interface. The hardware device of the panoramic assisted parking system is completed by reading the file containing the data mapping table in the USB port Initialization of the panoramic parking assistant. When the panoramic assisted parking equipment is started next time, the system reads the information unit containing the data mapping table in the flash memory, uses the data mapping table to process the real-time four-way fisheye video image, generates a real-time panoramic video image, and converts it to shown on the display. A working block diagram of an image system for panoramic parking assistance in an embodiment of the present invention is shown in Fig. 1 .

Fix 4 fisheye super wide-angle car cameras on the front, rear, left, and right sides of the car. For the front and rear cameras of the car, they can be installed near the top of the license plate. The installation angle is inclined downward. For the cameras on the left and right of the car, you can Installed near the rearview mirror, the installation angle is vertically downward. According to the length and width ratio of the vehicle, determine the size and layout of the calibration cloth. There are checkerboard and stitching markers drawn on the calibration cloth, and the calibration cloth is placed around the car so that the front, rear and sides of the vehicle are respectively placed for calibration. The checkerboard markers, and the splicing markers are placed in the common coverage area of the two cameras. The schematic diagram of the calibration cloth position is shown in Figure 2.

Use the fisheye camera calibration tool to calibrate the fisheye camera to be used in the system. You can use opencv or matlab calibration toolbox to calibrate the fisheye camera. Here, choose matlab calibration toolbox.

In another typical implementation of the present application, the calibration step: perform image processing on the 4-way fisheye calibration image to generate a panoramic stitching image, and the ultimate goal is to generate a data mapping table between the original fisheye image and the panoramic stitching image , the implementation steps are described as follows:

Step (1): Fix the 4-way fisheye camera at the front, left, rear and right of the vehicle body to be calibrated, and place (draw) the calibration cloth on the ground as shown in Figure 2, where the four calibration cloths are The layout and dimensions of the checkerboard areas are identical.

Step (2): Obtain four fisheye calibration images A ₀ (x ₀₀ ,y ₀₀ ), A ₁ (x ₁₀ ,y ₁₀ ), A ₂ (x ₂₀ ,y ₂₀ ) of the front, left, rear and right of the car ), A ₃ (x ₃₀ ,y ₃₀ ); according to the distortion correction model, the distortion correction is performed on the front, left, rear and right fisheye calibration images, and four distortion-corrected fisheye calibration images B ₀ (x ₀₁ , y ₀₁ ), B ₁ (x ₁₁ ,y ₁₁ ), B ₂ (x ₂₁ ,y ₂₁ ), B ₃ (x ₃₁ ,y ₃₁ ).

Step (3): Identify the inner corners of the checkerboard in the fisheye calibration image after distortion correction respectively, and obtain the image coordinates of the four vertices of the rectangular area formed by the inner corners of the checkerboard, and the upper left vertex of the rectangular area formed by all the inner corners of the checkerboard The image coordinates corresponding to the upper right vertex, the lower left vertex, and the lower right vertex are P ₀ , P ₁ , P ₂ , and P ₃ , respectively. If there are a checkerboard squares in the horizontal direction and b checkerboard squares in the vertical direction in the image. If P ₄ (x ₀ , y ₀ ) is the coordinates corresponding to the upper left vertex of the rectangular area formed by the inner corners of the checkerboard in the bird's-eye view image, then the upper right vertex, lower left vertex, and lower right vertex of the rectangular area formed by the inner corners of the checkerboard are in The corresponding coordinates in the bird’s-eye view image are P ₅ ((a-2)×u+x ₀ ,y ₀ ), P ₆ (x ₀ ,(b-2)×u+y ₀ ), P ₇ ((a- 2)×u+x ₀ ,(b-2)×u+y ₀ ), where u is the image size of a single checkerboard square after bird’s-eye view transformation, C′ ₀ , C′ ₁ , C′ ₂ , C′ ₃ respectively To rotate the images B ₀ , B ₁ , B ₂ , and B ₃ counterclockwise by 0 degrees, 90 degrees, 180 degrees, and 270 degrees around the upper left vertex, calculate the values of P ₄ , P ₅ , P ₆ , and P ₇ The corresponding pixel points P′ ₀ , P′ ₁ , P′ ₂ , P′ ₃ in the images C′ ₀ , C′ ₁ , C′ ₂ , and C′ ₃ , according to the image coordinates P ₀ , P ₁ of the inner corner points of the checkerboard , P ₂ , P ₃ and the world coordinates P′ ₀ , P′ ₁ , P′ ₂ , P′ ₃ of the inner corner points of the checkerboard can calculate the homography transformation matrix of the image; according to the homography transformation matrix, for each After distortion correction, the fisheye calibration images are subjected to bird's-eye view transformation respectively, and four bird's-eye-view images C ₀ (x ₀₂ ,y ₀₂ ), C ₁ (x ₁₂ ,y ₁₂ ), C ₂ (x ₂₂ ,y ₂₂ ), C ₃ (x ₃₂ , y ₃₂ ), respectively determine the fusion areas of the four bird's-eye-view images.

Step (4): according to formula (2), (3), (4), (5), (6), (7), (8), (9), (10) generate as shown in accompanying drawing 3 Stitching template images, where w _lr , w _rr , h _fr , h _rr are related to the display distance of the panoramic bird's-eye view image, as shown in Figure 4.

w _v =a×u (10)

Stitch the four bird's-eye view images into a complete panoramic bird's-eye view image D(x ₃ , y ₃ ), and determine the overlapping area of any pair of bird's-eye view calibration images.

Step (5): Perform binarization on the overlapping areas of adjacent bird’s-eye view calibration images, select a target template image A _mh on one of the feature images, with a size of M _mh ×N _mh , and a target template image on the other feature image The search sub-image B _mh covered by the template A _mh has a size of L _mh ×W _mh , and the maximum value of M _mh and L _mh is r, and the maximum value of N _mh and W _mh is c. Existing pixel points, and its pixel value is 0, which means that this pixel point is not a feature point on the image. Use the formula (11) to calculate the dissimilarity P mh between the target template A _mh and the search sub-image B _mh , and obtain the minimum value of P _mh when matching the template A _mh and the sub-image B _mh covered by the template, and complete the _comparison between the two images. The registration of the image is established through the matching relationship of the features to establish the registration mapping transformation between the images. Finally, in the four overlapping areas of the panoramic stitching image, the sum of the dissimilarities of all image features collected by different cameras is minimized.

Step (6): According to the homography transformation matrix calculated in step (3) and the used fisheye image distortion correction model, the panoramic stitching mapping tables M ₀ , M ₁ , M ₂ and M ₃ can be generated, and according to the panoramic stitching template The image can generate mapping tables M ₄ , M ₅ . The fusion area image fusion adopts the feathering method, and the specific fusion methods are as formula (12), formula (13), formula (14), where d ₁ and d ₂ are the distances from the pixels in the spliced fusion area to the overlapping boundaries AC and AB respectively, When calculating the pixel values of the fusion regions R ₄ , R ₅ , R ₆ , and R ₇ pixels, the overlapping images I _s involved in the calculation are A ₀ , A ₁ , A ₂ , and A ₃ respectively, and the overlapping images I _t involved in the calculation are respectively are A ₁ , A ₂ , A ₃ and A ₀ . The coordinates of the pixels involved in the fusion regions R ₄ , R ₅ , R ₆ , and R ₇ and the fusion gradient factors of the pixels are stored in the mapping table M ₆ . Write the information including the panoramic stitching mapping relation tables M ₀ , M ₁ , M ₂ , M ₃ , M ₄ , M ₅ and M ₆ into the panorama stitching mapping table file.

h ₁ +h ₂ =1 (13)

E(x,y)=h ₁ I _s (x,y)+h ² I _t (x,y) (14)

If (x′ ₃ , y′ ₃ ) is the pixel point in the panoramic video image, after the calibration is completed, after the 4-way fisheye real-time image is obtained, use the data mapping table generated by the calibration system to generate a panoramic real-time image, and the implementation steps Described as follows:

Step (1): read the panorama stitching map file, and initialize the variables related to the panorama stitching map;

Step (2): According to the mapping table M ₄ , the area to which the pixel point (x′ ₃ , y′ ₃ ) belongs can be determined;

Step (3): If the pixel point (x′ ₃ , y′ ₃ ) belongs to an area among R ₀ , R ₁ , R ₂ , and R ₃ , determine the pixel point (x′ ₃ , y′ ₃ ) and the fisheye image The coordinate mapping table M _k . According to the coordinate mapping table M ₅ , the bird's-eye view image pixel coordinate C _k (x' _k2 , y' _k2 ) corresponding to the pixel point (x′ ₃ , y′ ₃ ) can be determined, and the bird’s-eye view image can be determined according to the coordinate mapping table M _k The fisheye image pixel coordinates A _k (x′ _k0 , y′ _k0 ) corresponding to the pixel point C _k (x′ _k2 , y′ _k2 ), thus determining the pixel value of the final pixel point;

Step (4) If the pixel point (x′ ₃ , y′ ₃ ) belongs to an area among R ₄ , R ₅ , R ₆ , and R ₇ , determine the overlap between the pixel point (x′ ₃ , y′ ₃ ) and the fisheye Image coordinate mapping table M _s , M _t . According to the coordinate mapping table _{M 5} , the pixel coordinates C _s (x′ _s2 , y′ _s2 ), C _t (x _{′ t2} ,y′ _t2 ), according to the coordinate mapping table M _s , M _t can determine the fisheye image corresponding to the bird’s-eye image pixel C _s (x′ _s2 ,y′ _s2 ), C _t (x′ _t2 ,y′ _t2 ) Pixel coordinates A _s (x′ _s0 , y′ _s0 ), A _t (x′ _t0 , y′ _t0 ). The fusion factor of (x′ ₃ , y′ ₃ ) is determined according to the mapping table M ₆ , and the pixel is fused to obtain the final output pixel value.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. For those skilled in the art, there may be various modifications and changes in the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims (10)

1. The auxiliary parking method based on the panoramic stitching data mapping table is characterized in that it comprises: Set a calibration cloth on the ground around the vehicle body, and use the layout and size information of the checkerboard area and the public coverage area on the calibration cloth to roughly register the images in the coverage area of the adjacent camera installed on the vehicle body; Then use the object features in the common coverage area to fine-tune the images in the coverage areas of adjacent cameras, and use the dissimilarity of image features to minimize the sum of the dissimilarity of all image features collected by different cameras, find Optimal image registration position; The panorama stitching data mapping table is generated by using the panorama mosaic template image, and the panorama video image can be generated in real time according to the file containing the data mapping relationship table. 2. The assisted parking method based on the panorama splicing data mapping table as claimed in claim 1, wherein the four fisheye calibration images A ₀ (x ₀₀ , y ₀₀ ), A ₁ (x ₁₀ ,y ₁₀ ), A ₂ (x ₂₀ ,y ₂₀ ), A ₃ (x ₃₀ ,y ₃₀ ), perform distortion correction on these four fisheye calibration images, and get four distortion correction The final fisheye calibration images B ₀ (x ₀₁ , y ₀₁ ), B ₁ (x ₁₁ , y ₁₁ ), B ₂ (x ₂₁ , y ₂₁ ), B ₃ (x ₃₁ , y ₃₁ ). 3. The auxiliary parking method based on the panorama stitching data mapping table as claimed in claim 2, characterized in that B ₀ (x ₀₁ , y ₀₁ ), B ₁ (x ₁₁ , y ₁₁ ), B ₂ ( x ₂₁ , y ₂₁ ), B ₃ (x ₃₁ , y ₃₁ ) corner points in the checkerboard image, get the world coordinates and image coordinates of the checkerboard corner points in the image, according to the world coordinates and image coordinates of the checkerboard corner points Can calculate the homography transformation matrix H of B ₀ (x ₀₁ ,y ₀₁ ), B ₁ (x ₁₁ ,y ₁₁ ), B ₂ (x ₂₁ ,y ₂₁ ), B ₃ (x ₃₁ ,y ₃₁ ) images ₀ , H ₁ , H ₂ , H ₃ , according to the homography transformation matrix, for images B ₀ (x ₀₁ ,y ₀₁ ), B ₁ (x ₁₁ ,y ₁₁ ), B ₂ (x ₂₁ ,y ₂₁ ) , B ₃ (x ₃₁ ,y ₃₁ ) perform bird’s-eye view transformation respectively to obtain four bird’s-eye view calibration images C ₀ (x ₀₂ ,y ₀₂ ), C ₁ (x ₁₂ ,y ₁₂ ), C ₂ (x ₂₂ ,y ₂₂ ), C ₃ (x ₃₂ ,y ₃₂ ). 4. The assisted parking method based on the panoramic stitching data mapping table as claimed in claim 3, wherein the homography transformation needs to ensure that the image size of a single checkerboard grid involved in the four bird's-eye view calibration images is the same, and the bird's-eye view calibration The orientation of the checkerboard in the image is consistent with the orientation of the panorama stitching. 5. The auxiliary parking method based on the panorama stitching data mapping table as claimed in claim 1, wherein the generation mode of the panorama mosaic template image is: according to the actual size of the checkerboard area of the calibration cloth, the size of the checkerboard area The actual vertical spacing between them, the image size information of the checkerboard area, and the display distance of the panoramic bird's-eye view calibration image are used to generate a panoramic stitching template image. 6. The auxiliary parking method based on the panoramic stitching data mapping table according to claim 5, wherein the panoramic stitching template image consists of a central area R _cent , a front area R _A , a left area R _B , and a rear area R _C , the right area R _D consists of five blocks, the central area R _cent is the car model image, the front area R _A , the left area R _B , the rear area R _C , and the right area R _D are respectively related to the bird's-eye view calibration image in the corresponding direction C ₀ (x ₀₂ ,y ₀₂ ), C ₁ (x ₁₂ ,y ₁₂ ), C ₂ (x ₂₂ ,y ₂₂ ), and C ₃ (x ₃₂ ,y ₃₂ ) correspond to each other, and the corresponding The stitching gap adjacent to the bird's-eye view calibration image, the rectangular area corresponding to the stitching gap is the overlapping area of the adjacent bird's-eye view calibration image; According to the position of the stitching line and the image position of the checkerboard in the bird's-eye view image, the four bird's-eye view calibration images are stitched into a complete panoramic bird's-eye view calibration image D(x ₃ ,y ₃ ). 7. The assisted parking method based on the panoramic stitching data mapping table as claimed in claim 1, wherein said finding the optimal image registration position specifically comprises: determining the overlapping area of adjacent bird's-eye view calibration images, Feature extraction is performed on the overlapping area images of the adjacent bird's-eye view calibration images, and the positions of the two bird's-eye view calibration images are fine-tuned according to the image features of the two overlapping areas, and the best registration positions of the two adjacent bird's-eye view images are searched, and finally In the 4 overlapping areas, the sum of the dissimilarities of all image features collected by different cameras is minimized. 8. the assisted parking method based on panoramic stitching data mapping table as claimed in claim 1, is characterized in that, generates panoramic stitching mapping table file, and panoramic stitching mapping table is made up of following several parts: F0: Pixel coordinate mapping table M ₀ between the bird's-eye view calibration image C ₀ (x ₀₂ ,y ₀₂ ) and the front fish-eye calibration image A ₀ (x ₀₀ ,y ₀₀ ); F1: Pixel coordinate mapping table M ₁ between the bird's-eye view calibration image C ₁ (x ₁₂ ,y ₁₂ ) and the left fish-eye calibration image A ₁ (x ₁₀ ,y ₁₀ ); F2: Pixel coordinate mapping table M ₂ between the bird's-eye calibration image C ₂ (x ₂₂ , y ₂₂ ) and the rear fish-eye calibration image A ₂ (x ₂₀ , y ₂₀ ); F3: Pixel coordinate mapping table M ₃ between the bird's-eye calibration image C ₃ (x ₃₂ , y ₃₂ ) and the right fish-eye calibration image A ₃ (x ₃₀ , y ₃₀ ); F4: Mapping table M ₄ of the region between the panoramic bird's-eye view calibration image D(x ₃ ,y ₃ ) and the bird's-eye view calibration image in 4 directions. The mapping table M ₄ can be expressed as formula (1), where R ₀ , R ₁ , R ₂ , and R ₃ are the sub-images of the bird's-eye view calibration images C ₀ , C ₁ , C ₂ , and C ₃ respectively, and do not belong to the fusion area of adjacent bird's-eye view calibration images in the panoramic video image, and the fusion area of adjacent bird's-eye view calibration images is the sub-image of the overlapping area of adjacent bird's-eye view calibration images, R ₄ , R ₅ , R ₆ , and R ₇ belong to the fusion area of adjacent bird's-eye view calibration images, and the images in the overlapping areas are fused in the fusion area, where R ₄ is The sub-images of the overlap area of the bird's-eye calibration images C ₀ and C ₁ , R ₅ is the sub-image of the overlap area of the bird-eye calibration images C ₁ and C ₂ , R ₆ is the sub-image of the overlap area of the bird-eye calibration images C ₂ and C ₃ , R ₇ is the sub-image of the overlapping area of the bird's-eye view calibration image C ₀ , C ₃ ; ${\mathrm{<span\; class="notranslate">\; FM</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{c}\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}\\ \mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\\ \mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ \mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}\\ \mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}\\ \mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}\\ \mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 6</span>}}\\ \mathrm{<span\; class="notranslate">\; 7</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 7</span>}}\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$ F5: Panoramic bird's-eye view calibration image D(x ₃ ,y ₃ ) and 4-direction bird's-eye view calibration images C ₀ (x ₀₂ ,y ₀₂ ), C ₁ (x ₁₂ ,y ₁₂ ), C ₂ (x ₂₂ ,y ₂₂ ) , C ₃ (x ₃₂ , y ₃₂ ), the pixel point coordinate mapping table M ₅ , and for any pixel point (x′ ₃ , y′ ₃ ) in the adjacent bird’s-eye view calibration image fusion area, it needs to be explained (x′ ₃ ,y′ ₃ ) and the coordinate mapping relationship between the two bird's-eye view image pixels; F6: Determine the fusion factors h ₁ and h ₂ of pixels in the fusion regions R ₄ , R ₅ , R ₆ , and R ₇ of adjacent bird’s-eye view calibration images, where h ₁ and h ₂ are the corresponding coordinates of the overlapping parts of the bird’s-eye view images Lower the weight of the pixel, store the coordinates (x′ ₃ , y′ ₃ ) of any pixel involved in the fusion area R ₄ , R ₅ , R ₆ , R ₇ and the fusion factor of the pixel into the mapping table M ₆ . 9. the assisted parking method based on panoramic splicing data mapping table as claimed in claim 1, it is characterized in that, in real time, generate panoramic video image realization step: after obtaining 4 road fisheye real-time images around the vehicle body in real time, to R ₀ , R ₁ , R ₂ , and R ₃ areas, according to the data mapping table file generated by the calibration system, map to the coordinates of the original fisheye image to determine the output pixel value, for R ₄ , R ₅ , R ₆ , R _7. The pixels in the area are mapped to the coordinates of the original fisheye image to be calculated according to the data mapping table file, and the pixel points are fused according to the fusion factor determined in the mapping table to determine the output pixel value. 10. An auxiliary parking system based on a panoramic stitching data mapping table, characterized in that it includes: The image acquisition module uses 4-way fisheye ultra-wide-angle cameras installed in the front, left, rear and right of the vehicle body to acquire images and perform synchronous processing on the images; Judgment module, used to judge whether the image after synchronous processing needs to be calibrated, if necessary, the image after synchronous processing is transmitted to the system calibration module, otherwise, it is transferred to the panorama generation module; The system calibration module performs distortion correction on the synchronously processed front video image, rear video image, left video image, and right video image respectively, and then performs bird’s-eye view transformation respectively, performs panoramic stitching on the bird’s-eye transformed images and generates panoramic stitching data mapping surface; The panorama generation module is used to remap and transform the video image, and then process the transformed image and transmit it to the display module for image display.