CN117853329A - Image stitching method and system based on multi-view fusion of track cameras - Google Patents
Image stitching method and system based on multi-view fusion of track cameras Download PDFInfo
- Publication number
- CN117853329A CN117853329A CN202311788914.4A CN202311788914A CN117853329A CN 117853329 A CN117853329 A CN 117853329A CN 202311788914 A CN202311788914 A CN 202311788914A CN 117853329 A CN117853329 A CN 117853329A
- Authority
- CN
- China
- Prior art keywords
- image
- camera
- point
- fusion
- stitching
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 230000004927 fusion Effects 0.000 title claims abstract description 43
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 25
- 230000000007 visual effect Effects 0.000 claims abstract description 24
- 238000012544 monitoring process Methods 0.000 claims abstract description 13
- 238000012545 processing Methods 0.000 claims abstract description 12
- 239000011159 matrix material Substances 0.000 claims description 29
- 238000013519 translation Methods 0.000 claims description 20
- 230000008569 process Effects 0.000 claims description 19
- 238000005192 partition Methods 0.000 claims description 17
- 238000003708 edge detection Methods 0.000 claims description 16
- 238000013507 mapping Methods 0.000 claims description 12
- 238000005457 optimization Methods 0.000 claims description 10
- 238000001514 detection method Methods 0.000 claims description 8
- 230000033001 locomotion Effects 0.000 claims description 6
- 230000009466 transformation Effects 0.000 claims description 6
- 230000000694 effects Effects 0.000 claims description 5
- 238000003709 image segmentation Methods 0.000 claims description 5
- 238000007526 fusion splicing Methods 0.000 claims description 3
- 238000009499 grossing Methods 0.000 claims description 3
- 238000003384 imaging method Methods 0.000 claims description 3
- 230000000873 masking effect Effects 0.000 claims description 3
- 230000007246 mechanism Effects 0.000 claims description 3
- 230000003287 optical effect Effects 0.000 claims description 3
- 230000011218 segmentation Effects 0.000 claims description 3
- 238000011524 similarity measure Methods 0.000 claims description 3
- 239000000758 substrate Substances 0.000 claims description 3
- 238000007499 fusion processing Methods 0.000 claims description 2
- 238000009434 installation Methods 0.000 claims description 2
- 238000007781 pre-processing Methods 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 6
- 238000013135 deep learning Methods 0.000 abstract description 2
- 230000008901 benefit Effects 0.000 description 5
- 230000003190 augmentative effect Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012937 correction Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 238000012935 Averaging Methods 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000011426 transformation method Methods 0.000 description 1
Abstract
The invention discloses an image stitching method and system based on multi-view fusion of track cameras. The image stitching is an image processing technology based on OpenCV and deep learning, and can stitch a plurality of images with different visual angles, expand the visual field of a scene and improve the overall appearance and information quantity of the images. The image stitching algorithm and the system based on the multi-view fusion of the track camera can be widely applied to the fields of real-time monitoring, virtual reality, landscape photography and the like, and provide wider and more realistic vision and image experience for users.
Description
Technical Field
The invention relates to the technical field of image processing and computer vision algorithms, in particular to an image stitching method and system based on multi-view fusion of an orbit camera.
Background
An image stitching method and system based on multi-view fusion of track cameras is a complex system which relates to a plurality of technical fields. It combines camera motion control, image processing and computer vision algorithms to provide a high quality, panoramic image experience. The design and implementation of the system need to comprehensively consider factors in the aspects of track planning, camera control, visual angle switching, image registration, real-time requirements, system stability, user interaction and the like.
(1) Panoramic images have wide application in the fields of virtual reality, augmented reality, digital maps, video surveillance, and the like. Traditional panoramic image acquisition methods include using special equipment (such as panoramic cameras) or post-stitching after capturing multiple images. However, these methods generally have problems of high equipment cost, complex post-processing, and the like, and thus a more convenient and efficient panoramic image acquisition scheme is required.
(2) With the development of mobile camera technology, mobile cameras are widely used in fields of monitoring, photography and the like. The movable camera has the advantages of high freedom degree and adjustable visual angle, and can capture images with different visual angles. Therefore, the movable camera is combined with the image stitching technology, so that the panoramic image can be acquired and displayed.
(3) Conventional image stitching methods are generally based on a fixed viewing angle or a defined viewing angle range, and cannot fully utilize the viewing angle information of a plurality of mobile cameras. The invention aims to solve the problem of the image stitching algorithm and the system based on the multi-view fusion of the track cameras, and obtains images of a plurality of view angles through the cameras moving on a single track and fuses the images into a panoramic image, so that richer and more real visual experience is provided.
The image stitching method and the system aim to solve the following technical problems:
(1) Viewing angle matching problem: due to the fact that the positions of the cameras are the same, perspective difference exists between images with different angles. Therefore, it is necessary to solve the problem of how to accurately match images of different viewing angles in order to perform subsequent stitching processing.
(2) Image alignment problem: because of the difference of the visual angles and the difference of the shooting time between the images with different visual angles, the problem of how to accurately align the images needs to be solved, so that the spliced images have good continuity and consistency.
(3) Image fusion problem: because of the differences of illumination, colors and the like among images with different visual angles, the problem of how to perform image fusion needs to be solved, so that the spliced images have consistency in color and brightness, and a better impression effect is achieved.
(4) Real-time problem: for real-time application scenes such as a monitoring system, the algorithm needs to have higher real-time performance, and can rapidly process and splice a plurality of visual angle images so as to meet the requirement of real-time monitoring.
Disclosure of Invention
The invention aims to provide an image stitching method and system based on multi-view fusion of a track camera. By mounting two or more cameras on a single mobile camera robot and using their different perspectives for image stitching. By moving the camera, it is possible to acquire an image of a wider area and generate a panoramic image of high quality. The image stitching algorithm and the system based on the multi-view fusion of the track camera can be widely applied to the fields of real-time monitoring, virtual reality, landscape photography and the like, and provide wider and more realistic vision and image experience for users.
In order to achieve the above purpose, the technical scheme adopted by the invention is as follows: an image stitching method based on multi-view fusion of track cameras, which is characterized by comprising the following steps of installing two or more cameras on a single movable camera robot, acquiring images with different view angles through the movable cameras, and generating a high-quality panoramic image through an image stitching algorithm:
s1, camera installation and control: installing a plurality of cameras on the movable camera, wherein the cameras have different view angles and focal lengths, and the number and the positions of the cameras are designed and determined according to the needs; the movement and the gesture of the movable camera are controlled through a camera control system, and the movement and the shooting angle of the movable camera are controlled by using a remote controller, a computer or other equipment, so that the viewing angle and the position required by the panoramic image are ensured to be acquired;
s2, image acquisition: under different positions and angles of the movable cameras, acquiring image sequences of a plurality of cameras, and acquiring images through a preset time interval or trigger mechanism to ensure time consistency in the splicing process;
s3, calibrating the visual angle: performing visual angle calibration on each camera, including calibration of camera internal parameters and external parameters, and estimation of camera gestures, wherein the parameters are used for a subsequent image stitching process;
s4, view angle fusion: preprocessing the image of each camera, including removing distortion, adjusting brightness and contrast, and then, using a feature point matching or optical flow estimation algorithm to find the corresponding relation between images of different visual angles;
s5, image stitching: based on the correspondence, an image registration and stitching algorithm is used to stitch the images of multiple perspectives into a panoramic image, the stitching algorithm including but not limited to: feature-based methods, including SIFT, SURF, and direct image registration-based methods, including global transforms, local transforms;
s6, fusion optimization and display of the result: optimizing the spliced panoramic image, including removing discontinuity of a spliced area, color adjustment and edge smoothing, so as to improve the quality and the look and feel of the panoramic image; the final panoramic image is then displayed on a user interface or saved as an image file.
Preferably, the step S3 of the viewing angle calibration stage further includes: calibrating internal parameters and external parameters of a camera, and constructing a geometric mapping relation between pixel information on an image and physical information in a world coordinate system, wherein the internal parameters comprise focal length, principal points, aspect ratio and distortion parameters, and represent projection transformation of the pixel information on the image coordinate system and the physical information in the camera coordinate system; the external parameters comprise a rotation matrix and a translation vector, and represent the position relation between a camera coordinate system and a world coordinate system; assuming a mapping of one point P (x, y, z) in world coordinates to one point P (u, v) of a pixel coordinate system, the whole process is expressed by formula (1):
αp＝HP＝KPTP (1)
wherein K represents camera internal parameters, R represents a rotation matrix, T represents a translation vector, alpha is a scale factor, and KRT passes through H 3×4 Is shown here as H 3×4 The mapping matrix is called;
in a road scene, a monitoring camera meets a small hole imaging model and has the characteristics that a spin angle is zero and a main point is positioned in the center of an image, a reasonably simplified camera calibration model under a single scene is built according to the self attribute and the erection characteristic of the monitoring camera, and a boundary coordinate system origin is arranged on a road surface right below the camera c oZ c Plane and Y w oZ w The planes are all in the plane omega, and the included angle between the plane omega and the road direction is theta, X c Axis and X w The axes are parallel to each other and have the same direction;
according to a camera calibration model under a single scene, the internal parameter matrix is as follows:
wherein f is the focal length of the camera, the unit of the focal length is pixel, the position of the main point (Cx, cy) is a known parameter, and the matrix expression is shown as a formula (2);
the rotation matrix R is the counterclockwise rotation of the world coordinate system around the X-axisAs shown in formula (3);
the translation vector T is thus defined as the origin moving h distance along the Z axis of the world coordinate system to the camera origin, expressed as equation (4);
thus, the first and second substrates are bonded together,
in particular, for a three-dimensional point (x, y, 0) on the road surface, formula (1) may be substituted, and the corresponding image coordinates (u, v) are obtained:
wherein H is 11 、H 12 ......H 34 Is H 3×4 Mapping corresponding matrix elements in the matrix.
Preferably, the step S4 view fusion stage further includes: the most important task is to find out the overlapping area of two adjacent images, and then enter the next image stitching;
since the photo is taken by the horizontally moving camera, the side of the building or the overlapping area will generate perspective parallax in the photo, the side of the building in the photo will not be detected when the feature matching is performed, and the detected area containing a series of horizontal lines is the front of one building in the photo; when taking street view photographs, perspective parallax always exists on a face that is not parallel to the camera translation direction, such as the side of a building; if the overlapping area is on a plane parallel to the camera translation direction, such as the front of a building, the problem of perspective parallax is not considered; the front of a building is usually provided with a plurality of windows and balconies, if the pictures are subjected to image processing, the pictures are composed of horizontal lines and vertical lines, and conversely, when the pictures are positioned on the side of the building, the horizontal lines become inclined, so that perspective parallax is generated, and the horizontal lines and the vertical lines are detected without detecting the pictures, so that the interference of the perspective parallax on image splicing is avoided.
Preferably, the step S5 image stitching stage further includes: in the image stitching stage, the purpose of edge detection is to extract information from an image, and the resulting image obtained by the method contains much less information than the original image, but the information contained in the image is more convenient to compare in the process of generating a panoramic image and is used for image matching; edge detection is performed using a laplace template, in order to remove noise from the original image, the image is median filtered prior to edge detection,
is provided with a template->Let I be the original image, I mn The pixel is at point (m, n). />Edge detection will produce an image I, the pixels being denoted as I mn ；
i mn ＝i m-dn-d ×l 11 +i m-dn ×l 12 +i m-dn+d ×l 13 +i mn-d ×l 21 +i mn ×l 22
+i mn+d ×l 23 +i m+dn-d ×l 31 +i m+dn ×l 32 +i m+dn+d
×l 33 (8)
In the above formula, d is the distance between the point (m, n) and the adjacent point associated with it, in the edge detection, d is set to 1, if i mn >0, then the point (m, n) is on the boundary of the object;
then, the horizontal line and the vertical line are extracted by the following method:
let x be the parameter that determines whether the segments are combined, for a particular point i mn If the following formula is satisfied:
i m n-x ＝i m n ＝i m n+x (9)
the points [ (m, n-x), (m, n-x+1), (m, n-1), (m, n), (m, n+1), (m, n+x-1), (m, n+x) ] will constitute a horizontal straight line;
similarly, when detecting a vertical line, a parameter y is set if the following equation is satisfied:
i m-y n ＝i m n ＝i m+y n (10)
the points [ (m-y, n), (m-y+1, n) ] will constitute a vertical line;
the detection of horizontal lines and vertical lines produces an output image, image division is performed by the detection results of the horizontal lines and vertical lines, and a division region R i Comprising a set of horizontal and vertical lines, where all partitions take the same height, since the camera used is a continuous horizontally moving image acquisition and the purpose of image segmentation is only to identify the corresponding areas of adjacent images, where the width is determined according to specific requirements, any vertical line of x-abscissa in the partition, with which at least one horizontal line intersects;
in the dividing region R i When compared to another partition, it is described with 3 characteristic parameters: HL (HL) i m i ，E v Sum mu v Wherein HL is 1 m 1 Is the value of the ordinate of the horizontal line at the top of the partition, j is the total value of the partition horizontal line. E (E) v Mu, the average value of the ordinate of the horizontal line in the partitioned area v The distribution condition of the horizontal line is represented, and the specific relation is as follows:
partition R of two images p And R is q Is the similarity measure function of:
and finally, pattern matching is carried out by finding out similar segmentation areas, a stitching algorithm is carried out, and images of a plurality of visual angles are fused into a panoramic image.
Preferably, the step S6 fuses the optimization stage, and further includes: in the fusion process, the photographed pictures are obtained through camera translation, so that the pixels which tend to be in the middle are more representative in the splicing process, namely the middle part has higher weight ratio in the process of image fusion, and the optimization can be made according to the characteristics as follows:
let I be the original image, the center pixel point is denoted as I at (x, y) xy The upper left corner vertex (1, 1) pixel is denoted as i 11 The lower right corner pixel is denoted as i mn 。
Masking maskMore haveCharacteristic that the pixel point in the middle part has a higher weight ratio, the center point (x, y) mask is set to M xy ＝1，M 11 ＝M 1n ＝M m1 ＝M mn =α, then point (p, q)
The obtained Mask is multiplied with the original image I, and the obtained results are fused, so that the intermediate region of the original image has higher confidence coefficient, and the picture fusion splicing effect is better.
Compared with the prior art, the invention has the beneficial effects that: the image stitching method and system based on the multi-view fusion of the track cameras is a technology for fusing images with different view angles captured by a single track movable single camera into a panoramic image through an algorithm. The system of the algorithm has wide application range and can be applied to the fields of monitoring systems, virtual reality, augmented reality and the like. It can provide a more comprehensive, more realistic visual experience for the user, while providing more information and details. The main technical innovation points are as follows:
(1) Viewing angle matching: view matching is one of the core steps of the method. The key point is how to accurately extract and match feature points in images of different perspectives to determine correspondence between them.
(2) Image alignment: image alignment is a critical step in ensuring consistent position and orientation of images from different perspectives in space. Geometric transformation methods such as affine transformation, perspective transformation, etc. can be used to achieve accurate alignment of the images.
(3) And (3) image fusion: the image fusion is to process the aligned images so that the spliced images have consistency in terms of color, brightness and the like. Common image fusion methods include weighted averaging, multi-resolution fusion, and the like.
(4) Real-time optimization: for real-time application scenarios, the real-time nature of the algorithm is an important consideration. The processing speed of the algorithm can be optimized by adopting the technical means of parallel computing, hardware acceleration and the like so as to meet the requirements of applications such as real-time monitoring and the like.
Drawings
The invention will be described in further detail with reference to the drawings and the detailed description.
Fig. 1 is an overall flowchart of an image stitching method based on multi-view fusion of an orbital camera according to the invention.
Fig. 2 is a schematic design of the track camera.
Fig. 3 is a plurality of pictures taken in a continuous pan provided by the embodiment.
Fig. 4 is a complete translational length diagram obtained by splicing the pictures in fig. 3 by the image splicing method according to the present invention.
Detailed Description
Other advantages and effects of the present invention will become apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present invention with reference to specific examples. The invention may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present invention.
The image splicing method and system based on the multi-view fusion of the track camera provided by the invention are characterized in that two or more cameras are arranged on a single movable camera robot, and the different view angles of the cameras are utilized for image splicing. By moving the camera, an image of a wider area is acquired, and a high-quality panoramic image is generated. Image stitching is an important technology in the field of image processing, and is an image processing technology based on OpenCV and deep learning. The method can splice a plurality of images with different visual angles, can expand the visual field of a scene and improve the overall appearance and information quantity of the images.
Specifically, as shown in fig. 1, the image stitching method based on the multi-view fusion of the track camera comprises the following steps:
step1: camera mounting and control: a plurality of cameras are mounted on the movable camera, and the cameras can have different view angles and focal lengths. The number and the positions of the cameras are designed and determined according to the needs; the motion and pose of the moveable camera is controlled by a camera control system. A remote control, computer or other device may be used to control to ensure the viewing angle and position required to obtain the panoramic image.
Step2: and (3) image acquisition: image sequences of multiple cameras are acquired at different positions and angles of the movable camera. The image acquisition can be performed by a preset time interval or trigger mechanism, so that the time consistency in the splicing process is ensured.
Step3: viewing angle calibration: and performing visual angle calibration on each camera, including calibration of camera internal parameters and external parameters and estimation of camera gestures. These parameters will be used in the subsequent image stitching process.
Step4: view angle fusion: the image of each camera is preprocessed, such as removing distortion, adjusting brightness and contrast, etc. Then, the corresponding relation between images of different visual angles is found by utilizing methods such as feature point matching or optical flow estimation.
Step5: and (3) image stitching: based on the corresponding relation, an image registration and stitching algorithm is adopted to fuse the images of the multiple view angles into a panoramic image. Common stitching algorithms include feature-based methods (e.g., SIFT, SURF, etc.) and direct image registration-based methods (e.g., global transforms, local transforms, etc.).
Step6: fusion optimization and display of the results: optimizing the spliced panoramic image, including removing discontinuity, color adjustment, edge smoothing and the like of a spliced area so as to improve the quality and the look and feel of the panoramic image; the final panoramic image is then displayed on a user interface or saved as an image file.
In Step3, the most important Step is to calibrate the internal parameters and external parameters of the camera, which essentially aims to construct the geometric mapping relationship between the pixel information on the image and the physical information in the world coordinate system. The internal parameters include focal length, principal point, aspect ratio, distortion parameters, etc., and represent projection transformation of pixel information on an image coordinate system and physical information in a camera coordinate system, and the external parameters mainly include rotation matrix and translation vector, and represent the positional relationship between the camera coordinate system and a world coordinate system. The whole process (assuming a mapping of one point P (x, y, z) in world coordinates to one point P (u, v) of a pixel coordinate system) can be represented by the formula (1):
αp＝HP＝KPTP (1)
wherein K represents a camera internal parameter, R represents a rotation matrix, T represents a translation vector, and α is a scale factor. KRT available through H 3×4 Is shown here as H 3×4 The mapping matrix is called.
For example, in a road scene, the monitoring camera meets the small hole imaging model, has the characteristics of zero spin angle, main point being positioned in the center of an image and the like, establishes a reasonably simplified camera calibration model under a single scene according to the self attribute and the erection characteristic of the monitoring camera, and sets the origin of a boundary coordinate system on the road surface right below the camera, Y c oZ c Plane and Y w oZ w The planes are all in the plane omega, and the included angle between the plane omega and the road direction is theta, X c Axis and X w The axes are parallel to each other and are aligned.
According to a camera calibration model under a single scene, the internal parameter matrix is as follows:
wherein f is the focal length of the camera, the unit is pixel, the main point position (Cx, cy) is a known parameter, and the matrix expression is shown in formula (2).
The rotation matrix R is the counterclockwise rotation of the world coordinate system around the X-axisAs shown in formula (3). The translation vector T is thus defined as the origin moving h distance along the Z axis of the world coordinate system to the camera origin,can be represented by formula (4).
Thus, the first and second substrates are bonded together,
in particular, for three-dimensional points (x, y, 0) on the road surface, formula (1) can be substituted, and corresponding image coordinates (u, v) can be obtained:
wherein H is 11 、H 12 ......H 34 Is H 3×4 Mapping corresponding matrix elements in the matrix.
In Step4 view fusion phase, the most important task is to find the overlapping area of two adjacent images. Then the next image stitching step can be entered.
Since the photographs are taken by horizontally moving cameras, the sides of the building or overlapping areas will produce perspective parallax in the photographs, the sides of the building in the photographs will not be detected when the feature matching is performed, and the detected area containing a series of horizontal lines will be the front of one of the buildings in the photographs. When taking street view photographs, perspective parallax always exists on a face that is not parallel to the camera translation direction, such as the side of a building; if the overlapping area is on a plane parallel to the camera translation direction, for example the front of a building, the problem of perspective parallax is not considered. A building is typically lined up with a number of windows, balconies, etc. If we image process these photos, they consist of horizontal and vertical lines. In contrast, when they are located at the sides of a building, the horizontal lines become inclined, which creates perspective parallax, which is not detected by the horizontal line and vertical line detection, thus avoiding interference of the perspective parallax with image stitching.
In Step5 image stitching stage, the purpose of edge detection is to extract information from one image, and the resulting image obtained by this method contains much less information than the original image, but it contains more information that is more convenient to compare in the process of generating panoramic images. Is used for image matching. Common edge detection operators are Roberts gradients, laplace templates, etc. Here edge detection is performed using a laplace template. In order to remove noise from the original image, the image is median filtered before edge detection.
We set up template->Let I be the original image, I mn The pixel is at point (m, n). />Edge detection will produce an image I, the pixels being denoted as I mn 。
i mn ＝i m-dn-d ×l 11 +i m-dn ×l 12 +i m-dn+d ×l 13 +i mn-d ×l 21 +i mn ×l 22
+i mn+d ×l 23 +i m+dn-d ×l 31 +i m+dn ×l 32 +i m+dn+
×l 33 (8)
In the above formula, d is the distance between the point (m, n) and the adjacent point associated with it, and in the edge detection, d is set to 1. If i mn >0, then point (m, n) is on the boundary of the object.
Then, the horizontal line and the vertical line are extracted by the following method:
let x be the number of line segments to be groupedParameters of the combination. For a specific point i mn If the following formula is satisfied:
i m n-x ＝i m n ＝i m n+x (9)
the points [ (m, n-x), (m, n-x+1), (m, n-1), (m, n), (m, n+1), (m, n+x-1), (m, n+x) ] will constitute one horizontal straight line.
Similarly, when detecting a vertical line, a parameter y is set if the following equation is satisfied:
i m-y n ＝i m n ＝i m+y n (10)
the points [ (m-y, n), (m-y+1, n) ] will constitute a vertical line.
The detection of the horizontal line and the vertical line generates an output image, and the image segmentation is performed by the detection results of the horizontal line and the vertical line. A dividing region R i Comprising a set of horizontal and vertical lines. All the partitions here take the same height, since the camera we use is one continuous horizontally moving image acquisition and the purpose of image segmentation is only to identify the corresponding areas of the adjacent images. The width is here determined according to specific requirements, and any vertical line of abscissa x in the partition has at least one horizontal line intersecting it.
In the dividing region R i When compared to another partition we describe it with 3 characteristic parameters: HL (HL) i m i ，E v Sum mu v . Wherein HL is 1 m 1 Is the value of the ordinate of the horizontal line at the top of the partition, j is the total value of the partition horizontal line. E (E) v Mu, the average value of the ordinate of the horizontal line in the partitioned area v Representing the distribution of horizontal lines. The specific relation is as follows:
partition R of two images p And R is q Is the similarity measure function of:
and finally, carrying out pattern matching and a splicing algorithm by finding similar segmentation areas. The images from the multiple perspectives are fused into a panoramic image.
In the Step6 fusion optimization stage, during fusion, as the photographed pictures are obtained by camera translation, the more middle pixels tend to be representative during splicing, namely the middle part has higher weight ratio during image fusion, so that optimization can be performed according to the characteristics:
let I be the original image, the center pixel point is denoted as I at (x, y) xy The upper left corner vertex (1, 1) pixel is denoted as i 11 The lower right corner pixel is denoted as i mn 。
Masking maskMore characteristic, if the pixel point in the middle part has higher weight ratio, the center point (x, y) mask is set as M xy ＝1，M 11 ＝M 1n ＝M m1 ＝M mn =α. Then point (p, q)
The obtained Mask is multiplied with the original image I, and the obtained results are fused, so that the intermediate region of the original image has higher confidence coefficient, and the picture fusion splicing effect is better.
In addition to image stitching algorithms and systems based on single track mobile camera multi-view fusion, there are other alternatives to achieve the same goal, such as:
(1) Panoramic camera: panoramic images can be directly captured by using a specially designed panoramic camera without post stitching. Panoramic cameras typically have multiple lenses or fisheye lenses that can capture images from multiple perspectives simultaneously and fuse them into a panoramic image through hardware internal processing. The method has the advantages of simple operation and good real-time performance, but has the disadvantages of higher equipment cost and limited functions by camera hardware.
(2) Multiple camera array: the panoramic image can be obtained by arranging a plurality of cameras in an array, photographing with a certain overlapping area, and then stitching their images. This method is often used in the fields of virtual reality, augmented reality, and the like. This has the advantage that a panoramic image with high resolution can be obtained, but requires accurate camera synchronization and hardware layout, and the system complexity is high for large-scale camera arrays.
(3) Multi-image stitching software: there are many multi-image stitching software, such as Photoshop, PTGui, etc., based on computer vision and image processing techniques. The software can perform feature point matching, geometric transformation and fusion on a plurality of images in a manual or semi-automatic mode to generate a panoramic image. While these software often require some manipulation and adjustment by the user, they have flexibility and wide applicability.
These alternatives can choose the most suitable method according to the specific situation under different application scenarios and requirements. Image stitching algorithms and systems based on single track mobile camera multi-view fusion may have unique advantages in some specific scenarios, but also need to be selected in consideration of factors such as cost, instantaneity, system complexity, and user requirements.
Examples
Fig. 2 is a schematic design of a track camera: hanging the camera on a track, a plurality of translational pictures can be obtained by controlling the camera to move and shoot, as shown in fig. 2, P1, P2, P3.
Firstly, performing correction processing on internal parameters of a camera, and taking the upper left corner of an image shot by the camera at the extreme edge P1 position as a three-dimensional coordinate origin.
The data matrix formed by the images obtained after the correction of the photographs taken at the P1 point is A1, the data matrix formed by the images taken at the P2 point is A2, and so on, and the image matrix of the PN point is AN.
In the visual angle fusion stage, the detection calculation of the horizontal vertical lines is carried out through the obtained matrix, the image segmentation is carried out according to the obtained matrix, and then the image matching is carried out on the segmented area.
In the image stitching stage, mask values can be calculated to emphasize the importance of the center of the image, and the center point (x, y) Mask is set to M xy Mask of four corner vertices is set to m=1 11 ＝m 1n ＝M m1 ＝M mn ＝0.5。
The pixel size of the image is 1180 x 880, which is calculated by:
fig. 3 shows a plurality of pictures taken in a continuous translation.
By the method, a complete translation length chart shown in fig. 4 is obtained after the splicing.
The above embodiments are merely illustrative of the principles of the present invention and its effectiveness, and are not intended to limit the invention. Modifications and variations may be made to the above-described embodiments by those skilled in the art without departing from the spirit and scope of the invention. Accordingly, it is intended that all equivalent modifications and variations of the invention be covered by the claims, which are within the ordinary skill of the art, be within the spirit and scope of the present disclosure.
Claims (5)
1. An image stitching method based on multi-view fusion of track cameras, which is characterized by comprising the following steps of installing two or more cameras on a single movable camera robot, acquiring images with different view angles through the movable cameras, and generating a high-quality panoramic image through an image stitching algorithm:
s1, camera installation and control: installing a plurality of cameras on the movable camera, wherein the cameras have different view angles and focal lengths, and the number and the positions of the cameras are designed and determined according to the needs; the movement and the gesture of the movable camera are controlled through a camera control system, and the movement and the shooting angle of the movable camera are controlled by using a remote controller, a computer or other equipment, so that the viewing angle and the position required by the panoramic image are ensured to be acquired;
s2, image acquisition: under different positions and angles of the movable cameras, acquiring image sequences of a plurality of cameras, and acquiring images through a preset time interval or trigger mechanism to ensure time consistency in the splicing process;
s3, calibrating the visual angle: performing visual angle calibration on each camera, including calibration of camera internal parameters and external parameters, and estimation of camera gestures, wherein the parameters are used for a subsequent image stitching process;
s4, view angle fusion: preprocessing the image of each camera, including removing distortion, adjusting brightness and contrast, and then, using a feature point matching or optical flow estimation algorithm to find the corresponding relation between images of different visual angles;
s5, image stitching: based on the correspondence, an image registration and stitching algorithm is used to stitch the images of multiple perspectives into a panoramic image, the stitching algorithm including but not limited to: feature-based methods, including SIFT, SURF, and direct image registration-based methods, including global transforms, local transforms;
s6, fusion optimization and display of the result: optimizing the spliced panoramic image, including removing discontinuity of a spliced area, color adjustment and edge smoothing, so as to improve the quality and the look and feel of the panoramic image; the final panoramic image is then displayed on a user interface or saved as an image file.
2. The method for image stitching based on multi-view fusion of an orbital camera according to claim 1, wherein the step S3 of view calibration stage further comprises: calibrating internal parameters and external parameters of a camera, and constructing a geometric mapping relation between pixel information on an image and physical information in a world coordinate system, wherein the internal parameters comprise focal length, principal points, aspect ratio and distortion parameters, and represent projection transformation of the pixel information on the image coordinate system and the physical information in the camera coordinate system; the external parameters comprise a rotation matrix and a translation vector, and represent the position relation between a camera coordinate system and a world coordinate system; assuming a mapping of one point P (x, y, z) in world coordinates to one point P (u, v) of a pixel coordinate system, the whole process is expressed by formula (1):
αp＝HP＝KPTP (1)
wherein K represents camera internal parameters, R represents a rotation matrix, T represents a translation vector, alpha is a scale factor, and KRT passes through H 3×4 Is shown here as H 3×4 The mapping matrix is called;
in a road scene, a monitoring camera meets a small hole imaging model and has the characteristics that a spin angle is zero and a main point is positioned in the center of an image, a reasonably simplified camera calibration model under a single scene is built according to the self attribute and the erection characteristic of the monitoring camera, and a boundary coordinate system origin is arranged on a road surface right below the camera c oZ c Plane and Y w oZ w The planes are all in the plane omega, and the included angle between the plane omega and the road direction is theta, X c Axis and X w The axes are parallel to each other and have the same direction;
according to a camera calibration model under a single scene, the internal parameter matrix is as follows:
wherein f is the focal length of the camera, the unit of the focal length is pixel, the position of the main point (Cx, cy) is a known parameter, and the matrix expression is shown as a formula (2);
the rotation matrix R is the counterclockwise rotation of the world coordinate system around the X-axisAs shown in formula (3);
the translation vector T is thus defined as the origin moving h distance along the Z axis of the world coordinate system to the camera origin, expressed as equation (4);
thus, the first and second substrates are bonded together,
in particular, for a three-dimensional point (x, y, 0) on the road surface, formula (1) may be substituted, and the corresponding image coordinates (u, v) are obtained:
wherein H is 11 、H 12 ......H 34 Is H 3×4 Mapping corresponding matrix elements in the matrix.
3. The method for image stitching based on multi-view fusion of an orbital camera according to claim 2, wherein the step S4 view fusion stage further comprises: the most important task is to find out the overlapping area of two adjacent images, and then enter the next image stitching;
since the photo is taken by the horizontally moving camera, the side of the building or the overlapping area will generate perspective parallax in the photo, the side of the building in the photo will not be detected when the feature matching is performed, and the detected area containing a series of horizontal lines is the front of one building in the photo; when taking street view photographs, perspective parallax always exists on a face that is not parallel to the camera translation direction, such as the side of a building; if the overlapping area is on a plane parallel to the camera translation direction, such as the front of a building, the problem of perspective parallax is not considered; the front of a building is usually provided with a plurality of windows and balconies, if the pictures are subjected to image processing, the pictures are composed of horizontal lines and vertical lines, and conversely, when the pictures are positioned on the side of the building, the horizontal lines become inclined, so that perspective parallax is generated, and the horizontal lines and the vertical lines are detected without detecting the pictures, so that the interference of the perspective parallax on image splicing is avoided.
4. The image stitching method based on multi-view fusion of track cameras as claimed in claim 3, wherein the step S5 image stitching stage further comprises: in the image stitching stage, the purpose of edge detection is to extract information from an image, and the resulting image obtained by the method contains much less information than the original image, but the information contained in the image is more convenient to compare in the process of generating a panoramic image and is used for image matching; edge detection is performed using a laplace template, in order to remove noise from the original image, the image is median filtered prior to edge detection,
is provided with a template->Let I be the original image, I mn The pixel is at the point (m, n), where>Edge detection will produce an image I, the pixels being denoted as I mn ；
i mn ＝i m-d n-d ×l 11 +i m-d n ×l 12 +i m-d n+d ×l 13 +i m n-d ×l 21 +i m n ×l 22 +i m n+d ×l 23 +i m+d n-d ×l 31 +i m+d n ×l 32 +i m+d n+d ×l 33 (8)
In the above formula, d is the distance between the point (m, n) and the adjacent point associated with it, in the edge detection, d is set to 1, if i mn > 0, then the point (m, n) is on the boundary of the object;
then, the horizontal line and the vertical line are extracted by the following method:
let x be the parameter that determines whether the segments are combined, for a particular point i mn If the following formula is satisfied:
i m n-x ＝i m n ＝i m n+x (9)
the points [ (m, n-x), (m, n-x+1), (m, n-1), (m, n), (m, n+1), (m, n+x-1), (m, n+x) ] will constitute a horizontal straight line;
similarly, when detecting a vertical line, a parameter y is set if the following equation is satisfied:
i m-y n ＝i m n ＝i m+y n (10)
the points [ (m-y, n), (m-y+1, n) ] will constitute a vertical line;
the detection of horizontal lines and vertical lines produces an output image, image division is performed by the detection results of the horizontal lines and vertical lines, and a division region R i Comprising a set of horizontal and vertical lines, where all partitions take the same height, due to the use ofThe camera of (2) is a continuous horizontally moving image acquisition and the purpose of image segmentation is only to identify the corresponding areas of adjacent images, where the width is determined according to specific requirements, and any vertical line with x-axis in the segmented area has at least one horizontal line intersecting it;
in the dividing region R i When compared to another partition, it is described with 3 characteristic parameters: HL (HL) i m i ，E v Sum mu v Wherein HL is 1 m 1 Is the value of the ordinate of the horizontal line at the top of the partition, j is the total value of the partition horizontal line, E v Mu, the average value of the ordinate of the horizontal line in the partitioned area v The distribution condition of the horizontal line is represented, and the specific relation is as follows:
partition R of two images p And R is q Is the similarity measure function of:
and finally, pattern matching is carried out by finding out similar segmentation areas, a stitching algorithm is carried out, and images of a plurality of visual angles are fused into a panoramic image.
5. The method for image stitching based on multi-view fusion of an orbital camera according to claim 4, wherein the step S6 of fusion optimization further comprises: in the fusion process, the photographed pictures are obtained through camera translation, so that the pixels which tend to be in the middle are more representative in the splicing process, namely the middle part has higher weight ratio in the process of image fusion, and the optimization can be made according to the characteristics as follows:
let I be the original image, the center pixel point is denoted as I at (x, y) xy The upper left corner vertex (1, 1) pixel is denoted as i 11 The lower right corner pixel is denoted as i mn ，
Masking maskMore characteristic, if the pixel point in the middle part has higher weight ratio, the center point (x, y) mask is set as M xy ＝1，M 11 ＝M 1n ＝M m1 ＝M mn =α, then point (p, q)
The obtained Mask is multiplied with the original image I, and the obtained results are fused, so that the intermediate region of the original image has higher confidence coefficient, and the picture fusion splicing effect is better.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311788914.4A CN117853329A (en) | 2023-12-25 | 2023-12-25 | Image stitching method and system based on multi-view fusion of track cameras |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311788914.4A CN117853329A (en) | 2023-12-25 | 2023-12-25 | Image stitching method and system based on multi-view fusion of track cameras |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117853329A true CN117853329A (en) | 2024-04-09 |
Family
ID=90541205
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311788914.4A Pending CN117853329A (en) | 2023-12-25 | 2023-12-25 | Image stitching method and system based on multi-view fusion of track cameras |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117853329A (en) |
-
2023
- 2023-12-25 CN CN202311788914.4A patent/CN117853329A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021120407A1 (en) | Parallax image stitching and visualization method based on multiple pairs of binocular cameras | |
| CN106462944B (en) | High-resolution panorama VR generator and method | |
| Huang et al. | Panoramic stereo imaging system with automatic disparity warping and seaming | |
| EP3163535B1 (en) | Wide-area image acquisition method and device | |
| Shum et al. | Panoramic image mosaics | |
| CN110782394A (en) | Panoramic video rapid splicing method and system | |
| WO2014023231A1 (en) | Wide-view-field ultrahigh-resolution optical imaging system and method | |
| CN111583116A (en) | Video panorama stitching and fusing method and system based on multi-camera cross photography | |
| CN108629829B (en) | Three-dimensional modeling method and system of the one bulb curtain camera in conjunction with depth camera | |
| JPH11509946A (en) | Method and system for image combination using parallax dependent technique | |
| CN106534670B (en) | It is a kind of based on the panoramic video generation method for connecting firmly fish eye lens video camera group | |
| CN108830925B (en) | Three-dimensional digital modeling method based on spherical screen video stream | |
| JP7196421B2 (en) | Information processing device, information processing system, information processing method and program | |
| CN110689476A (en) | Panoramic image splicing method and device, readable storage medium and electronic equipment | |
| CN111866523B (en) | Panoramic video synthesis method and device, electronic equipment and computer storage medium | |
| KR100614004B1 (en) | An automated method for creating 360 degrees panoramic image | |
| CN113763544A (en) | Image determination method, image determination device, electronic equipment and computer-readable storage medium | |
| CN108765292A (en) | Image split-joint method based on the fitting of space triangular dough sheet | |
| CN108898550A (en) | Image split-joint method based on the fitting of space triangular dough sheet | |
| CN109272445B (en) | Panoramic video stitching method based on spherical model | |
| Limonov et al. | Stereoscopic realtime 360-degree video stitching | |
| CN107146195A (en) | Sphere image split-joint method and device | |
| CN117853329A (en) | Image stitching method and system based on multi-view fusion of track cameras | |
| Zhu et al. | Pedestrian-aware panoramic video stitching based on a structured camera array | |
| CN114022562A (en) | Panoramic video stitching method and device capable of keeping integrity of pedestrians |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |