CN103517041B - Based on real time panoramic method for supervising and the device of polyphaser rotation sweep - Google Patents

Abstract

The present invention proposes a real-time panorama monitoring method and device based on multi-camera rotary scanning, which completes panorama presentation of a 360-degree surrounding environment and realizes continuous updating of panorama images. Step 1, initialization; step 2, each camera device collects a frame of image respectively, and performs pixel intensity compensation, the sampling results of each device are stored in their respective sampling image sequences, and at the same time record the position of the rotating pan/tilt at the time of current sampling Angle information; display the initial image of each image sequence, and determine the position of the initial frame in each group on the 360-degree panoramic image according to the relative positions installed between the cameras; step 3, select an image sequence, extract the previous frame A and current Frame B, detect and describe the feature points in the ROI area, then match the feature points extracted in the two frames of A and B, and adjust the non-maximum value suppression threshold in feature detection according to the matching results; step 4, according to the matching results Compute the homography matrix.

Description

Real-time panorama monitoring method and device based on multi-camera rotation scanning

technical field

The invention belongs to the field of image processing and the technical field of video monitoring, and in particular relates to a multi-camera rotary scanning panorama monitoring system.

Background technique

Existing panoramic imaging equipment is mainly divided into three categories, one is to rely on special imaging equipment such as fisheye lens to obtain images with a large viewing angle; Scene images; the third category is to rely on a constantly rotating camera to capture image sequences of the surrounding environment for splicing and fusion to complete panoramic imaging. However, the image taken by special imaging equipment such as fisheye lens has serious distortion (CN102222337A); the second type of multi-camera scene fusion method needs a lot of imaging equipment to cover the 360-degree range, the cost is high, and the system is complicated (CN102117008A; CN101866482B); The rotating equipment of last class single camera is subjected to the limitation real-time property of rotation speed and splicing method, and it takes a long time to update the 360-degree scene once (CN101221351B). For 360-degree panoramic monitoring tasks, not only a clear and complete view is required, but also factors such as real-time performance and system cost must be considered.

Contents of the invention

The present invention aims to overcome the above drawbacks, and proposes a real-time panoramic monitoring method and device based on multi-camera rotary scanning, which completes the panoramic presentation of the 360-degree surrounding environment and realizes continuous updating of panoramic images. On the basis of providing 360-degree panoramic images, distortion is avoided, the number of image acquisition devices required is small, and there are no special requirements such as wide-angle and fisheye. The system is simple and achieves high real-time performance. The calibrated multi-imaging devices have consistent images High system stability.

A real-time panoramic monitoring method based on multi-camera rotation scanning, comprising the following steps:

Step 1. Determine the sampling interval according to the rotation speed of the turntable and the field of view of the camera; determine the region of interest according to the rotation speed, direction of rotation, and field of view of the turntable; The initial frame of imaging equipment sampling is displayed on the corresponding initial position of the 360-degree panoramic image according to the installation position of the imaging equipment and the horizontal calibration offset, and the angle information of the rotating pan/tilt at the time of current sampling is recorded at the same time;

Step 2. Each camera device collects a frame of image respectively, and performs pixel intensity compensation. The sampling results of each device are stored in their respective sampling image sequences, and at the same time record the angle information of the rotating pan/tilt at the current sampling time; display each The initial image of an image sequence, according to the relative position of the camera installation, determine the position of the initial frame in each group on the 360-degree panoramic image;

Step 3: Select an image sequence, extract the previous frame A and the current frame B, detect and describe the feature points in the ROI area, then match the feature points extracted in the two frames A and B, and adjust the feature detection according to the matching results The non-maximum suppression threshold in ;

Step 4. Calculate the homography matrix according to the matching result, and identify whether the splicing condition is met according to the parameters in the homography matrix; if the splicing condition is met, obtain the offset of the B frame relative to the A frame according to the homography matrix, and use The method of fading in and fading out eliminates the splicing gap, splicing frame B to frame A according to the offset and displaying it on the panoramic image; if it does not meet the splicing conditions, receive the current angle information, and display frame B on the basis of this The corresponding position of the panorama image.

Step five, check whether there is a next frame, if yes, go to step one, otherwise end.

The parameter calibration described in step 1 includes: sampling one frame each when each imaging device rotates past the 0 position of the pan/tilt, and performing parameter calibration of all imaging devices through image matching, including vertical offset calibration and horizontal offset calibration , Brightness gain calibration.

In step 3, adjust the non-maximum value suppression threshold in the feature detection and use the feature point detection method in the surf (Speeded-UpRobustFeatures) algorithm to extract feature points; then use the ORB (Comparative Evaluation of Binary Features) feature algorithm to describe the detected feature points ; Then perform feature matching according to the feature points in the two frames of A and B, and filter the matching results; modify the relevant threshold in feature detection to keep the number of feature points at a value as small as possible to meet the matching requirements.

A real-time panoramic monitoring device based on multi-camera rotation scanning, comprising:

The input module is fixed on the turntable by one or more image acquisition devices. The imaging devices shoot at different angles, and the relative positions are fixed. Every time a period of time is passed, a sample is built, and the sampled images of each imaging device form an image sequence. , it is required that the adjacent frames in each sequence need to have enough overlapping areas to provide the buffer storage area of the latest two adjacent frames;

The calibration module is used to calibrate the parameters between different imaging devices, including vertical offset calibration, horizontal offset calibration, and brightness gain calibration;

Stitching module, in each image sequence, the current frame is matched with the previous frame, the offset is calculated, and the stitching seam is eliminated at the same time, so as to determine the display position of the current frame;

Stitching failure response module, when linear splicing failure is detected in the image processing module, the position of the current frame in the panoramic image is estimated by using the position of the pan/tilt responding when the current frame is captured and the relative position of the image capture device corresponding to the image sequence on the turntable display location;

The display module provides a buffer storage area for the image to be displayed and position information, and updates the corresponding position in the panoramic image with the current frame according to the information provided by the image processing or splicing failure processing module to complete the update of the panoramic image.

Beneficial effects of the present invention:

The real-time panorama monitoring method and device based on multi-camera rotary scanning disclosed by the present invention adopts the method of multiple image acquisition devices to rotate and capture image sequences to obtain the information of the surrounding environment, which is doubled compared with the previous single-camera rotary splicing device. The update rate of the panoramic image is improved, and the number of imaging devices required is small, and ordinary cameras can meet the requirements, so the hardware system overhead required is kept low; the multi-scale feature point detection method and ORB feature description method based on Hessian determinant make this Compared with the previous feature matching stitching algorithm, the invention has higher scale invariance and illumination invariance, making the image stitching more stable and accurate; at the same time, using the angle information returned by the gimbal, it can assist in dealing with the failure of image stitching, so that The reliability of the system is improved.

Description of drawings

Fig. 1 is the two-camera turntable device of the embodiment of the present invention;

Fig. 2 is the flowchart of panoramic image generation algorithm of the embodiment of the present invention;

FIG. 3 is a schematic diagram of 360-degree panorama mosaic according to an embodiment of the present invention.

Specific implementation method

The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, not all of them. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present invention belong to the protection scope of the present invention.

As shown in Figure 1, N camera devices are used in this embodiment (N=3 in this embodiment), fixed at intervals of 360°/N degrees on the turntable, and the field of view of the camera is v _fa in this embodiment. _fa = 20°, the sampling image resolution is L _width × L _height ; the minimum overlapping degree of adjacent frames is set to α _min (0<α _min <1), where α _mi is the empirical value of this method and generally takes 0.3, namely The reference frame (left figure) ROI is the horizontal axis L _width × α pixel to the L _width pixel area, and the current frame (right figure) ROI is the horizontal axis 0 pixel to the L _width × (1-α) pixel area; in addition, the present embodiment The turntable in the top view rotates clockwise. The width of the panoramic display area is W and the height is H.

The present invention provides a panoramic monitoring method based on multi-camera rotation shooting, as shown in Figure 2, specifically comprising the following steps:

Step 1: Initialize the turntable, start from position 0 of the turntable to rotate clockwise at a constant speed, and the speed is w. The initial frame is sampled and displayed at the initial position of the 360-degree panoramic image, and the angle information of the rotating gimbal at the time of current sampling is recorded at the same time.

Among them, the sampling interval needs to be determined according to the rotation speed of the turntable and the field of view of the camera. According to the camera field of view v _fa , the overlapping degree α of adjacent frames, and the rotation speed w, the sampling interval t can be calculated:

$\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; w</span>}}<span\; class="notranslate">\; <</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; min</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; w</span>}}$

Since all imaging devices are fixed on the turntable, they all maintain the same rotational angular velocity w, and the sampling interval is t.

Calibration consists of the following steps:

1. Select the image frame collected by camera 1 as frame A, and randomly select another camera sampling frame as frame B, detect the feature points in the two frames respectively, and then match the feature points extracted in the two frames A and B, Adjust the non-maximum suppression threshold in the feature detection according to the matching result; Specifically, the steps include: adopting the feature point detection method in the surf (Speeded-UpRobustFeatures) algorithm to extract the feature point; then using the ORB (ComparativeEvaluationofBinaryFeatures) feature The algorithm describes the detected feature points; then performs feature matching according to the feature points in A and B frames, and filters the matching results. See Step 3 for details.

2. According to the matching result, calculate the homography matrix, and identify whether the splicing condition is met according to the parameters in the homography matrix. If the splicing condition is not met, let the camera corresponding to frame B capture another frame of image at position 0 of the gimbal, and repeat step 3-1 until the splicing condition is met.

3. Horizontal offset calibration value x′ _i =h ₁₃ , vertical offset calibration value y′ _i =h ₂₃ , count the pixel intensity sum of the overlap area of two frames A and B where (x,y) and (x′,y′) are the pixel coordinates aligned with each other in frame A and frame B respectively, Ω and Ω′ are the corresponding overlapping regions in frame A and frame B, and i is the corresponding camera number, ε ₁ =0. Repeat 1 to 3 until all cameras are calibrated.

Step 2, image sequence initialization. Each camera device collects a frame of image respectively, and performs pixel intensity compensation, that is, for all pixels in the current frame, I(x,y)=I(x,y)+ε _i , i is the imaging device corresponding to the frame Numbering. The sampling results of each device are saved in their respective sampling image sequences, and the angle information of the rotating pan/tilt at the time of current sampling is recorded at the same time.

The position of the initial frame in each group on the 360-degree panoramic image is determined according to the relative positions of the camera installations. The starting position of the sequence 1 image displayed in the panoramic image is (X ₁ , Y ₁ ), then the corresponding position of the sequence 2 image is (X ₂ , Y ₂ ), and the corresponding position of the sequence 3 image is (X ₃ , Y ₃ ), where X ₁ =0, $x_3=x_1+W\&times;\frac{2}{3}+{\mathrm{\&epsiv;}}_3,Y_1=0,Y_2=Y_3=0.$

Step 3: Select an image sequence, extract the previous frame A and the current frame B, detect and describe the feature points in the ROI area, then match the feature points extracted in the two frames A and B, and adjust the feature detection according to the matching results Relevant threshold in; specifically, described step 3 comprises: adopt the feature point detection method in surf algorithm to carry out the extraction of feature point; Adopt ORB feature algorithm to describe the feature point that detects again; Then according to A, B two frames Feature matching is performed on the feature points in , and the matching results are screened; the relevant thresholds in feature detection are modified to keep the number of feature points at a value as small as possible to meet the matching requirements, so as to save algorithm time.

The selection of image feature points should consider the brightness changes caused by the environment and the scale changes caused by the movement of the monitoring system. At the same time, attention should be paid to the time overhead of the algorithm to ensure that all image processing can be completed within a sampling interval t. The feature point detection is based on the multi-scale feature point detection method of the Hessian determinant in the surf algorithm. Specifically: first establish the scale space, and calculate the approximate value of the Hessian matrix determinant of each pixel in the image. The Hessian matrix is:

$\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left[\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; )</span>& {\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; )</span>\\ {\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; )</span>& {\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; )</span>\end{array}\right]$

where the σ scale is L _xx is the second derivative of the Gaussian filter The result of convolution with the input image, the calculation of L _xy and L _yy is similar, For the convenience of calculation, the Surf algorithm uses the convolution D _xx , D _xy , D _yy of the box filter template and the input image to replace L _xx , L _xy , and L _yy . The Hessian determinant can be expressed as:

det(H _approx )＝D _xx D _yy -(0.9D _xy ) ²

Then set a threshold θ _f for non-maximum value suppression to detect feature points, and then interpolate in the scale space to obtain stable feature point positions and scale values. In this embodiment, the initial value of θ _f is set to 400, and it is adjusted adaptively according to the splicing effect to balance the detection effect and time overhead. specific,

${\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&Delta;\&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}& {\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; g</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; ></span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; x</span>}}\\ {\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&Delta;\&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}& {\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; g</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; \&le;</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; min</span>}}\end{array}\right.$

Wherein, M _good is the number of feature point pairs after screening, M _max and M _min are set adjustment limits respectively, in this embodiment, M _max =40, M _min =15.

After the feature points are obtained, the features are described to prepare for feature matching. Specifically, this paper uses the feature description method in the ORB algorithm to encode the binary feature through random point pairs, and then adds the rBRIEF feature to describe the feature direction.

After obtaining the description of the feature points, the BF (BruteForce) algorithm is used for feature matching, and the matching results are initially screened, and the feature point pairs whose distance between two points is greater than 50% of the maximum distance are eliminated. The reserved feature point pairs are used as the data for calculating the homography matrix.

Step 4: Calculate the homography matrix according to the matching result, and identify whether the splicing condition is met according to the parameters in the homography matrix.

The homography matrix H is the transformation matrix between the source image coordinates and the target image coordinates.

$\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; =</span>\left[\begin{array}{ccc}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 12</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 13</span>}}\\ {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; two</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; three</span>}}\\ {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 31</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 32</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 33</span>}}\end{array}\right]$

Let (x ₁ ,y ₁ ) and (x ₂ ,y ₂ ) be the pixel coordinates in the original image and the pixel coordinates in the target image respectively, then

$\left(\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\end{array}\right)<span\; class="notranslate">\; =</span>\left(\begin{array}{c}\frac{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 12</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 13</span>}}}{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 31</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 32</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 33</span>}}}\\ \frac{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; two</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; three</span>}}}{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 31</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 32</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 33</span>}}}\end{array}\right)$

All cameras in this system are fixed on the turntable and are on the same horizontal plane, so there is no rotation transformation and projection transformation during normal splicing. The elements in the homography matrix H should satisfy: h ₁₁ and h ₂₂ are approximately 1, h ₁₂ , h ₂₁ , h ₃₁ , and h ₃₂ are approximately 0. Therefore, the condition ξ=h ₁₂ +h ₂₁ +h ₃₁ +h ₃₂ is set. When ξ<ξ _min , the splicing is considered successful, otherwise it is considered as a splicing failure. ξ _min is determined based on empirical values. In this embodiment, ξ _min = 0.1.

Step 5, if the splicing condition is met, according to the homography matrix, get the offset of the B frame relative to the A frame, use the gradual in and out method to eliminate the splicing gap, splicing the B frame to the A frame according to the offset and Display on the panoramic image; if it does not meet the stitching conditions, receive the current angle information, based on this, display frame B at the corresponding position of the panoramic image.

In step 4, if the splicing result is successful splicing, h ₁₃ and h ₂₃ in the homography matrix are the horizontal offset x _offset and the vertical offset y _offset respectively. Since there may be differences in the sampling effect between the two frames, there are obvious traces at the stitching seam of the two pictures at this time. Here, the gradual in and gradual out method is used to process the overlapping area to eliminate the seam. The specific method is as follows:

Let I _A (x, y), I _B (x, y) be the image intensity at pixel (x, y) in frame A and frame B, w _seam be the width of the overlapping area k be the distance from the current pixel to the left of the overlapping area The number of pixels of the border. but

${\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; B</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; k</span>}}{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; m</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; k</span>}}{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; m</span>}}}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; B</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span>$

In step 4, if the stitching result fails, read the turntable angle information u of the current frame. For sequence 1, the horizontal offset of the current frame is For sequences 2 and 3, the horizontal offset of the current frame is

$\mathrm{<span\; class="notranslate">\; W</span>}<span\; class="notranslate">\; \&times;</span>\frac{<span\; class="notranslate">\; (</span>\frac{\mathrm{<span\; class="notranslate">\; 360</span>}}{\mathrm{<span\; class="notranslate">\; N</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; mod</span>}\mathrm{<span\; class="notranslate">\; 360</span>}}{\mathrm{<span\; class="notranslate">\; 360</span>}}<span\; class="notranslate">\; ,</span>$

Wherein N is the number of sequences, n is the number of the current sequence, 0≤n≤N, in this embodiment N=3, the vertical offset is equal to the vertical offset of the previous frame of each sequence. The final offset should be added with the offset calibration value of the corresponding camera, x _offset = x _offset + x′ _i , y _offset = y _offset + y′ _i .

As shown in FIG. 3 , finally, the current frame in each image sequence is displayed at a corresponding position in the panoramic image according to the horizontal offset and the vertical offset.

Step six, check whether there is a next frame, and if so, go to step two.

The invention discloses a real-time panoramic monitoring device based on multi-camera rotation scanning, which includes the following modules:

The input module is fixed on the turntable by one or more image acquisition devices, and the imaging devices shoot at different angles, and the relative positions are fixed. Sampling is carried out every time a period of time is established, and the sampled images of each imaging device form an image sequence, requiring that the adjacent frames in each sequence need to have enough overlapping areas to provide a buffer storage area for the latest two adjacent frames. For example, in this embodiment, an image acquisition system composed of two lenses is adopted, as shown in FIG. 1 .

The calibration module is used to calibrate parameters between different imaging devices, including vertical offset calibration, horizontal offset calibration, and brightness gain calibration.

In the splicing module, in each image sequence, the current frame is matched with the previous frame, the offset is calculated, and the splicing seam is eliminated at the same time, so as to determine the display position of the current frame.

Splicing failure response module, when linear splicing failure is detected in the image processing module, this module uses the position of the pan/tilt responding when the current frame is captured and the relative position of the image capture device corresponding to the image sequence on the turntable to estimate the current frame in the panoramic image. display position in .

The display module provides a buffer storage area for the image to be displayed and position information, and updates the corresponding position in the panoramic image with the current frame according to the information provided by the image processing or splicing failure processing module to complete the update of the panoramic image.

The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and scope of application . In summary, the contents of this specification should not be construed as limiting the present invention.

Claims (4)

1., based on a real time panoramic method for supervising for polyphaser rotation sweep, it is characterized in that, comprise the following steps:

Step one, determine the sampling interval according to gantry rotation velocity and viewing field of camera angle size; According to gantry rotation velocity, direction of rotation and angle of visual field size determination area-of-interest; The parametric calibration of multiple image capture device, at synchronization to each imaging device sampling initial frame, according to imaging device installation site and horizontal alignment side-play amount, be presented on the corresponding initial position of 360 degree of panoramic pictures, the angle information simultaneously during record present sample residing for rotary head;

Step 2, every platform picture pick-up device gather a two field picture respectively, and carry out pixel intensity compensation, and the sampled result of every platform equipment is kept in respective sequence of subsampled images respectively, the angle information simultaneously during record present sample residing for rotary head; Show the initial pictures of each image sequence, according to the relative position installed between camera, determine the often position of initial frame on 360 degree of panoramic pictures in group;

Step 3, choose an image sequence, extract previous frame A and present frame B, detect and describe the characteristic point in ROI region, then the characteristic point that A, B two extracts in frame is mated, adjust the non-maxima suppression threshold value in feature detection according to matching result;

Step 4, according to matching result calculate homography matrix, identify whether to meet splicing condition according to the parameter in homography matrix; Meet splicing condition, according to homography matrix, obtain the side-play amount of B frame relative to A frame, adopt the method being fade-in gradually to go out to eliminate splicing gap, according to side-play amount, the splicing of B frame is shown to A frame on panoramic picture; If do not meet splicing condition, receive current angular information, on this basis, frame B is presented at the relevant position of panoramic picture;

Whether step 5, detection have next frame, if had, go to step one, otherwise terminate.

2. a kind of real time panoramic method for supervising based on polyphaser rotation sweep as claimed in claim 1, it is characterized in that, parametric calibration described in step one comprises: a frame of sampling when every platform imaging device rotates past The Cloud Terrace 0, pass through images match, carry out the parametric calibration of whole imaging device, comprise vertical shift calibration, horizontal-shift calibration, luminance gain calibration.

3. a kind of real time panoramic method for supervising based on polyphaser rotation sweep as claimed in claim 1 or 2, it is characterized in that, the non-maxima suppression threshold value adjusted in step 3 in feature detection adopts the feature point detecting method in surf (Speeded-UpRobustFeatures) algorithm to carry out the extraction of characteristic point; ORB (ComparativeEvaluationofBinaryFeatures) characteristics algorithm is adopted to be described the characteristic point detected again; Carry out characteristic matching according to the characteristic point of A, B two in frame afterwards, and matching result is screened; Dependent thresholds in amendment feature detection, makes characteristic point quantity remain on the least possible numerical value of satisfied coupling demand.

4., based on a real time panoramic supervising device for polyphaser rotation sweep, it is characterized in that, comprising:

Input module, be fixed on turntable by multiple image capture device, different angles taken by imaging device, and relative position is fixed, often once to sample through to build after a while, the sampled images of each imaging device forms an image sequence respectively, requires that the consecutive frame in each sequence needs enough overlapping regions, provides the buffer storage of up-to-date adjacent two frames;

Calibration module, calibrates parameter between different imaging device, comprises vertical shift calibration, horizontal-shift calibration, luminance gain calibration;

Concatenation module, in each image sequence, present frame all carries out characteristic matching with previous frame, calculates side-play amount, eliminates splicing seams simultaneously, determine present frame display position;

Splicing failure response module, when linear mosaic failure being detected in image processing module, the The Cloud Terrace position of response when utilizing present frame to gather and the relative position of image capture device on turntable corresponding to this image sequence, estimate the display position of present frame in panoramic picture;

Display module, provides the buffer storage of image to be displayed and positional information, according to the information that image procossing or splicing failure handling module provide, upgrades relevant position, complete the renewal of panoramic picture in panoramic picture with present frame.