CN115035162A - Monitoring video personnel positioning and tracking method and system based on visual slam - Google Patents

Abstract

The invention discloses a method and a system for positioning and tracking monitoring video personnel based on visual slam.A depth camera is used for carrying out three-dimensional reconstruction on an environment to obtain a point cloud map of the environment; combining external reference calibration with a scene of a monitoring camera, and performing external reference calibration on the monitoring camera through a calibration plate to obtain position and posture information of the monitoring camera; the method comprises the steps of tracking personnel through a monitoring camera, identifying people in a monitored image by utilizing a deep neural network, calculating the three-dimensional positions of the people according to the ground priors of the pedestrians appearing in the monitoring by adopting the principle of inverse perspective transformation based on the previously calibrated positions and postures, drawing the tracks of the personnel in a constructed point cloud map, and enabling the tracks to be presented in the point cloud map.

Description

Method and system for location and tracking of surveillance video personnel based on visual slam

technical field

The invention relates to the technical field of machine vision, in particular to a method and system for positioning and tracking personnel in surveillance video based on visual slam.

Background technique

In recent years, the video surveillance market at home and abroad has grown explosively, and surveillance development has been trending towards high-definition and intelligence. However, traditional two-dimensional surveillance systems require frequent switching of surveillance images, which has many drawbacks. With the development of artificial intelligence, more and more tasks will be completed automatically by machines, which frees people from boring work and improves work efficiency.

In the prior art, the location of persons in a scene is generally determined through Wi-Fi and the like. Wi-Fi positioning generally uses the "nearest neighbor method" to determine which hotspot or base station is closest to, that is, where it is considered to be. If there are multiple sources nearby, cross positioning (triangulation positioning) can be used to improve positioning accuracy. When a user has turned on Wi-Fi or a mobile cellular network when using a smartphone, it may become a data source. It is necessary to record the signal strength of a huge number of determined location points in advance, and compare the signal strength of the newly added device with the signal strength of the huge amount of data. database to determine the location.

Compared with the Wi-Fi positioning method, the monitored person needs to actively carry a handheld device to receive wireless signals. The existing calibration methods all require moving the camera. However, in the actual environment, most of the monitoring cameras have been fixed to a certain position and cannot be used. The calibration method of the mobile camera commonly used in visual SLAM is used to calibrate it, that is, there is no way to accurately obtain the specific representation of its position in the environment in the computer. Based on this problem, the present invention adopts the visual positioning method relying on the monitoring system, and combines the external parameter calibration with the monitoring camera scene. Only through the video image information captured by the camera, automatic processing is performed to accurately locate the position of the person.

SUMMARY OF THE INVENTION

Aiming at the problems in the prior art that two-dimensional monitoring operations are cumbersome, unintuitive, low in efficiency, and relying on handheld devices for personnel positioning and tracking, the present invention proposes a method and system for monitoring video personnel positioning and tracking based on visual slam.

In order to achieve the above object, the present invention provides the following technical solutions:

On the one hand, the present invention provides a monitoring video personnel positioning and tracking method based on visual slam, which uses a depth camera to reconstruct the environment three-dimensionally to obtain a point cloud map of the environment; The board calibrates the external parameters of the surveillance camera to obtain the position and attitude information of the surveillance camera; tracks the person through the surveillance camera, uses the deep neural network to identify the person in the surveillance image, and uses inverse perspective based on the previously calibrated position and attitude. The principle of transformation is to calculate the three-dimensional position of the person on the ground according to the a priori of the pedestrians appearing in the monitoring, so as to realize the drawing of the person's trajectory in the constructed point cloud map, and make it appear in the point cloud map.

Further, the above-mentioned monitoring video personnel positioning and tracking method based on visual slam is characterized in that, comprises the following steps:

S1. 3D reconstruction based on visual slam: construct a 3D point cloud map of the scene and record the data required for external parameter calibration, and use the RGBD image stream and inertial sensor data provided by the depth camera to construct a 3D point cloud map of the scene based on visual odometry And save it to the file; in the process of 3D reconstruction, shoot the checkerboard calibration board and record the position and attitude of the camera at this time;

S2, calibration: the surveillance camera takes a picture of the surveillance image of the checkerboard calibration board, and combines the calibration data provided during the 3D reconstruction in step S1 to calculate the location and attitude of the surveillance camera in the 3D point cloud map;

S3. Position tracking and calculation of pedestrians in the surveillance camera: track the person, and identify the position of the person in the image based on the position and attitude data obtained in step S2 and the surveillance video stream provided by the surveillance camera. the location in the point cloud map;

The process of tracking and calculating the pedestrian's position in the surveillance camera is as follows:

S31, according to the type of self-monitoring camera and whether the movement has occurred, select whether to enter the posture correction process, if yes, enter step S32, otherwise enter step S33;

S32, if entering the attitude correction process, extract the vanishing point in the image and compare the position of the vanishing point, determine whether the monitoring screen rotates according to whether the vanishing point moves, and update the rotation;

S33, enter the positioning process, first take a frame of surveillance video image;

S34, perform target detection on the pedestrian in the image, and obtain the coordinates of the target frame;

S35, target tracking is performed on the detected target frame, and the corresponding personnel position coordinates are given;

S36. Calculate the spatial position coordinates of the personnel according to the calibration parameters of the surveillance camera;

S37, if the positioning is not terminated, acquire the next frame of image and return to step S33, otherwise end.

Further, the construction process of the three-dimensional point cloud map of the scene in step S1 adds a point cloud map processing thread, which is used to accept the incoming pose information and RGBD image frames of each frame of the camera, and output an accurate point cloud map. The specific process is as follows :

S141. Screen the incoming pose information and RGBD image frame of each frame of the depth camera. When the camera angle change between the current frame and the last selected frame is greater than 10° and the displacement change is greater than 2 meters, select the current frame , and perform subsequent point cloud map generation operations;

S142, calculate the point cloud block of the current frame, and rotate it to a unified world coordinate system;

S143, splicing and merging the point cloud blocks generated by all frames to obtain an overall point cloud map, filtering and removing outliers on the point cloud map, so as to compress the data volume of the point cloud map and optimize the visual perception of the map at the same time ;

S144. When a loopback occurs during the mapping process, ORB-SLAM3 re-optimizes the pose of the selected frame, re-splices the point cloud, and re-does the point cloud processing operation according to step S143.

Further, the external parameter calibration method of step S2 is:

S21. The surveillance camera shoots the checkerboard calibration board: select the origin position of the world coordinate system, and the surveillance camera slowly moves from the origin position to the checkerboard calibration board. In this process, ORB_SLAM3 is used to estimate the pose of the surveillance camera in real time. , when the surveillance camera moves to the front of the checkerboard calibration board, close the program and save the photos taken by the current surveillance camera and the pose of the camera;

S22. Surveillance camera internal parameter calibration: place the checkerboard calibration board within the range of the surveillance camera, move the checkerboard calibration board from multiple angles, record a video, extract frames from the video, identify the checkerboard, and use Zhang Zhengyou's calibration method to calibrate the surveillance camera internal reference and distortion;

S23. Surveillance camera external parameter calibration: According to the actual three-dimensional position information of the target feature point and the two-dimensional position of the target feature point in the image, the direct linear transformation method is used to solve the camera coordinate system and the target coordinate system, so as to realize the monitoring camera and the three-dimensional Calculation of relative positional relationship between point cloud maps.

Further, the calculation method of the spatial position coordinates of the personnel in step S36 is:

According to the pose transformation matrix of the surveillance camera, the camera model optical center P _ow = (X _ow , Y _ow , Z _ow ) of the surveillance camera is obtained:

P _ow =-RT ^t #(21)

Take the midpoint M(u, v) of the lower bottom edge of the target frame of the person to be positioned, calculate the spatial position of point M on the normalized plane according to the projection equation, set the depth d to be 1 meter, and solve it according to formula (22) The spatial coordinates of the point M on the normalized plane P _m =(X _m , Y _m , Z _m );

According to the height h of the ground in the world coordinate system, the equation of the plane where the ground is located is given as shown in equation (24), and the plane where the ground is located is converted into a point and a normal vector on the plane to represent:

z=h#(23)

p ₀ = (0, 0, h),

写作参数方程如(25)所示，其中

为射线的方向向量，

t为参数t∈[0，∞)，will ray

The writing parameter equation is shown in (25), where

is the direction vector of the ray,

t is the parameter t∈[0, ∞),

与地面所在平面交点为P_g，则有：set ray

The intersection point with the plane where the ground is located is P _g , then:

After finishing formula (26), we get:

From the distributive law of vector dot product:

交点P_g的坐标即为行人在世界坐标系下的三维坐标。Solve for the intersection

The coordinates of the intersection point _Pg are the three-dimensional coordinates of the pedestrian in the world coordinate system.

Further, the above-mentioned monitoring video personnel positioning and tracking method based on visual slam also includes step S4 visual display: displaying the three-dimensional point cloud map of the scene in step S1 and the position of the person in the three-dimensional point cloud map in step S3, and providing GUI. The interface is for user interaction and supervision. The specific content displayed includes multiple 3D point clouds and their corresponding cameras and trajectories. When a certain point cloud is selected to be displayed, other point clouds and corresponding cameras and trajectories are hidden.

On the other hand, the present invention also provides a visual slam-based surveillance video personnel positioning and tracking system, comprising the following modules to realize the above method steps:

The 3D reconstruction module is used to construct the 3D point cloud map of the scene and record the data required for external parameter calibration. It runs only once in the entire use cycle, and uses the RGBD image stream and inertial sensor data provided by the depth camera to construct the scene's The 3D point cloud map is saved to the file for display by the visual display module; in the process of 3D reconstruction, the checkerboard calibration board and the position and attitude of the camera are recorded for the calibration module to use;

The calibration module is used to measure the position and attitude of each surveillance camera in the 3D point cloud map. It runs only once in the entire use cycle. The surveillance camera takes a picture of the surveillance image of the checkerboard calibration board, combined with the calibration data provided during 3D reconstruction. Calculate the position and attitude of the surveillance camera in the 3D point cloud map for use by the position calculation module;

The position calculation module is used to identify the position of the character in the image and give the position of the character in the 3D point cloud map based on the position and attitude data obtained by the calibration module and the monitoring video stream provided by the monitoring camera for the visual display module to display. ;

The visual display module is used to display the 3D point cloud map of the scene and the position of the characters in the 3D point cloud map, and provide a GUI interface for user interaction and supervision.

Compared with the prior art, the beneficial effects of the present invention are:

The monitoring video personnel positioning and tracking method based on the visual slam of the present invention has three modules of mapping, calibration and monitoring, and realizes the capability expansion from 2D monitoring to 3D monitoring. At the same time, a display system suitable for this monitoring scene is designed, and optimized in the system point cloud display and interface video and trajectory output.

1. The present invention proposes a three-dimensional map construction method based on visual SLAM and inertial odometry. Through the three main processes of tracking, local mapping and loopback optimization, the reconstructed three-dimensional point cloud map is optimized to achieve the The scene in the building is reconstructed with better effect, so that the monitoring personnel can quickly understand and accurately locate the map.

2. In order to solve the problem that most surveillance cameras have been fixed to a certain position in the actual environment, the calibration method of mobile cameras commonly used in visual SLAM cannot be used to calibrate them, that is, there is no way to accurately obtain their position in the environment. Concrete representation in a computer. The invention provides a method for calibrating external parameters of a general surveillance camera and a three-dimensional point cloud of the environment. By using a checkerboard calibration board, it can adapt to the needs of calibrating external parameters of cameras in various environments.

3. Through the depth camera parameter calibration method of the present invention, the position information related to the depth camera is obtained, and the trajectory of the character can be accurately calculated only by processing the video captured by the ordinary general camera. At the same time, according to each person's tracking and drawing in the generated point cloud map, the corresponding information of the track is saved and displayed on the interface, and the target tracking track timeout can be automatically eliminated.

4. The present invention proposes a character position calculation tracking algorithm based on a three-dimensional visual algorithm by calculating the position of the personnel and optimizing the trajectory. The algorithm first uses a deep neural network to identify the character in the monitoring image, and then based on the previously demarcated positional relationship, Based on the principle of inverse perspective transformation, the three-dimensional position of the person is calculated according to the priori that the pedestrians appearing in the monitoring are all on the ground.

5. Based on the above three-dimensional point cloud map and graphical interface, the present invention also provides functions for expansion, which can identify the flames or smoke appearing in the video image, adapt to the real scene monitoring requirements in the building, and ensure that the In abnormal situations, the system can automatically detect danger and give an alarm in time.

6. The present invention improves the trajectory drawing method in the 3D point cloud map by designing the data transmission and display method, formats and transmits the background point cloud file to the system, and establishes a system that can display the 3D point cloud, camera model and trajectory. The 3D display interface of the Three.JS graphics library enables monitoring personnel to understand the output results easily and clearly, and it is convenient to take corresponding measures. The designed interface is versatile and can be displayed on a variety of devices.

7. The present invention combines the advantages of visual SLAM in map reconstruction to design and develop a 3D intelligent monitoring system, which can comprehensively solve the pain points of traditional two-dimensional monitoring, and has the advantages of being intuitive, efficient, low-cost, and easy to deploy. It can show 3 The 3D point cloud map can monitor the three-dimensional layout of the entire building, realize the function of target tracking with one key, and automatically calculate and draw the trajectory of people entering the building in the point cloud map to achieve a clear and accurate three-dimensional display and trajectory of the real-time trajectory. You can also query the relevant information in the list, and you can also switch to view the monitoring screen of each floor with one key. You can also add fire warning, smoke recognition, behavior recognition and other functions.

8. The system designed by the present invention simultaneously supports the calculation and display of the relative poses of fixed surveillance cameras, which is convenient to apply to existing surveillance scenarios, and facilitates the new generation of information technology and surveillance systems such as big data and cloud computing under the Internet + background. Deep integration.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only described in the present invention. For some of the embodiments, those of ordinary skill in the art can also obtain other drawings according to these drawings.

1 is a visual slam-based monitoring video personnel positioning and tracking system provided by an embodiment of the present invention;

FIG. 2 is a flowchart of a three-dimensional reconstruction module provided by an embodiment of the present invention;

3 is a flowchart of point cloud map calibration provided by an embodiment of the present invention;

Fig. 4 is the corresponding relationship between the calibration plate image captured by the RGBD camera and the surveillance camera provided by the embodiment of the present invention and the extracted feature points;

FIG. 5 is a flow chart of target detection tracking and position calculation provided by an embodiment of the present invention;

6 is a vanishing point extraction diagram provided by an embodiment of the present invention;

7 is a schematic diagram of a typed array provided by an embodiment of the present invention;

FIG. 8 is a sample diagram of a 3D point cloud loading provided by an embodiment of the present invention;

FIG. 9 is a sample diagram of a camera model provided by an embodiment of the present invention;

10 is a flow chart of trajectory update drawing provided by an embodiment of the present invention;

11 is a system data transmission diagram of an embodiment of the present invention;

12 is a display of functions related to the trajectory of a video surveillance system based on 3D vision visualization according to an embodiment of the present invention;

13 is an overall interface diagram of a three-dimensional vision-based visualization video surveillance system according to an embodiment of the present invention, and display of switch functions and floor switching functions;

FIG. 14 is a display of fire and smoke alarm functions of a three-dimensional vision-based visualization video surveillance system according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present invention.

An embodiment of the present invention provides a visual slam-based surveillance video personnel location tracking system. The hardware device adopts a Kinect for Azure depth camera, a checkerboard calibration board, a surveillance camera, a personal computer or a notebook computer. The system includes the following modules (as shown in FIG. 1 ) shown):

The 3D reconstruction module is used to construct the 3D point cloud map of the scene and record the data required for external parameter calibration. It runs only once in the entire use cycle, and uses the RGBD image stream and inertial sensor data provided by the depth camera to construct the scene's The 3D point cloud map is saved to the file for display by the visual display module; in the process of 3D reconstruction, the checkerboard calibration board and the position and attitude of the camera are recorded for the calibration module to use;

The calibration module is used to measure the position and attitude of each surveillance camera in the 3D point cloud map. It runs only once in the entire use cycle. The surveillance camera takes a picture of the surveillance image of the checkerboard calibration board, combined with the calibration data provided during 3D reconstruction. Calculate the position and attitude of the surveillance camera in the 3D point cloud map for use by the position calculation module;

The position calculation module is used to identify the position of the character in the image and give the position of the character in the 3D point cloud map based on the position and attitude data obtained by the calibration module and the monitoring video stream provided by the monitoring camera for the visual display module to display. ;

The visual display module is used to display the 3D point cloud map of the scene and the position of the characters in the 3D point cloud map, and provide a GUI interface for user interaction and supervision.

The system consists of the above four modules, of which the 3D reconstruction module, the calibration module and the position calculation module are innovative at the technical level, and the visual display module is innovative at the product level.

Based on the above system, an embodiment of the present invention proposes a method for locating and tracking personnel in surveillance video based on visual slam, and the specific steps are as follows:

S1. 3D reconstruction based on visual slam: construct a 3D point cloud map of the scene and record the data required for external parameter calibration, and use the RGBD image stream and inertial sensor data provided by the depth camera to construct a 3D point cloud map of the scene based on visual odometry And save it to the file; in the process of 3D reconstruction, shoot the checkerboard calibration board and record the position and attitude of the camera at this time.

In the 3D reconstruction process, the device parameter calibration uses the IMU_utils tool, Kalibr toolbox and QR code calibration board to measure the parameters of the depth camera, including: depth camera internal parameters, IMU parameters, time offset between IMU and depth camera, and IMU and the pose transformation relationship between the depth camera coordinate system.

The 3D reconstruction module is based on the ORB_SLAM3 open source framework and adapts the Kinect for Azure depth camera so that it can be used normally under actual working conditions. Step S1 The construction process of the 3D point cloud map of the scene is shown in Figure 2, and the modified content is as follows:

S11. Downsample the RGBD image stream provided by the depth camera, and the resolution is reduced from 1280×720 to 960×540;

S12, align the timestamps for the incoming data stream according to the calibration parameters saved in the file;

S13. The distance estimation in the Z-axis direction of the feature points is changed from triangulation to depth map reading;

S14 , adding a point cloud map processing thread to reconstruct the dense point cloud of the scene terrain.

Visual odometry only provides depth camera positions and poses, thus requiring a dense point cloud reconstruction of the scene terrain. It is divided into two steps. First, the point cloud is spliced according to the depth camera pose and the depth camera model, and then the point cloud is processed to remove duplicate points and redundant points. In this system, the above functions are realized through the point cloud map processing thread, and the flow chart is shown in Figure 2. The function of the point cloud map processing thread is to accept the incoming pose information and RGBD image frame (including color map and depth map) of each frame of the depth camera, and output an accurate point cloud map. The step S14 point cloud map processing thread includes:

S141. Screen the incoming pose information and RGBD image frame of each frame of the depth camera. When the camera angle change between the current frame and the last selected frame is greater than 10° and the displacement change is greater than 2 meters, select the current frame , and perform subsequent point cloud map generation operations;

S142, calculate the point cloud block of the current frame according to the depth camera model formula, and rotate it to a unified world coordinate system;

S143, splicing and merging the point cloud blocks generated by all frames to obtain an overall point cloud map, filtering and removing outliers on the point cloud map, so as to compress the data volume of the point cloud map and optimize the visual perception of the map at the same time ;

S144. When a loopback occurs during the mapping process, ORB-SLAM3 re-optimizes the pose of the selected frame, re-splices the point cloud, and re-does the point cloud processing operation according to step S143.

The detailed process of converting the RGBD image frames obtained by the depth camera into point cloud blocks and stitching them into the world coordinate system is as follows:

For a pixel p in the image, let its coordinates be (u, v), and the coordinates of the three-dimensional point P projected by p in the point cloud are (X _c , Y _c , Z _c ), then the point in the point cloud is The generation formula is as follows:

where f _x , f _y , c _x , and _cy represent the internal parameters of the camera, which are obtained by the device parameter calibration method given above, and d represents the depth of point p read from the depth map.

Through ORB-SLAM3, the rotation and translation matrices R _cw and t _cw of the camera in the world coordinate system are obtained, and they are merged together:

Then the coordinates X _w , Y _w , and Z _w of the point P in the world coordinate system can be obtained as:

Based on the above method, all pixels in the image can be converted into 3D points in the world coordinate system. The advantage of this approach is that after the loop detection thread of ORB-SLAM corrects the key frames, the correction results can be used. on dense maps. Compared to stitching point clouds directly, this approach can eliminate accumulated errors.

In the ORB_SLAM3 framework, since the world coordinate system is based on the position of the camera during initialization, the generated point cloud map will inevitably be skewed, which will bring about visual display of subsequent maps and subsequent external parameter calibration and other work. Negative effects, so the coordinate system calibration of the point cloud map is required. As shown in Figure 3, the calibration process of the 3D point cloud map is as follows:

1) Calculate the plane equation of the point cloud ground:

Using the plane detection method based on RANSAC in the PCL library, find the plane equation of the ground ax+by+cz+d=0, where a, b, c, d are the four parameters of the plane equation, and the ground plane equation can be obtained through the plane equation. The normal vector is (a,b,c).

2) Calculate the rotation matrix of the point cloud ground and the horizontal plane of the coordinate system:

Calculate the rotation matrix of the point cloud ground and the horizontal plane of the coordinate system, through which the rotation transformation of the point cloud map can be realized, thereby calibrating the point cloud map. The calculation method is as follows:

(1) Set the normal vector of the point cloud ground as v1=(a,b,c), the normal vector of the horizontal plane of the coordinate system as v2=(0,0,1), and calculate the rotation axis n of the rotation transformation between the two vectors With the rotation angle θ, the calculation formula is as follows:

(2) The rotation matrix R is obtained from the rotation axis n and the rotation angle θ, and the calculation formula is as follows:

R=cosθI+(1-cosθ)nn ^T +sinθn^

The ^ symbol is the conversion symbol from vector to antisymmetric matrix. If vector a=(a1, a2, a3), the specific conversion formula is as follows:

3) Calibrate the point cloud map using the rotation matrix:

Assuming that a certain point in the point cloud map to be calibrated is p ₀ =(x ₀ , y ₀ , z ₀ ), and the coordinates of the point after calibration are p ₁ =(x ₁ , y ₁ , z ₁ ), the conversion formula is as follows:

p ₁ =Rp ₀

Applying this formula to all the points in the point cloud map to be calibrated can realize the calibration of the entire point cloud map, and finally realize the alignment between the ground and the horizontal plane of the coordinate system.

S2. Calibration: Measure the position and attitude of each surveillance camera in the 3D point cloud map, and the surveillance camera takes a picture of the surveillance image of the checkerboard calibration board. Combined with the calibration data provided during the 3D reconstruction in step S1, the surveillance camera is calculated in 3D. Position and pose in the point cloud map.

This step mainly solves the following problems: for the surveillance cameras already installed in the building, how to obtain the coordinates of these cameras in the world coordinate system, the calibration process of step S2 is as follows:

S21. The surveillance camera shoots the standard checkerboard calibration board:

After 3D reconstruction of the environment using an RGBD camera, a point cloud map of the environment is obtained. In order to use the surveillance camera to complete the personnel positioning work, it is also necessary to measure the relative positional relationship between the surveillance camera and the point cloud map, that is, the external parameters. Extrinsic parameters were calculated using the calibration plate as a marker.

The function of the calibration board is to provide the camera with multiple three-dimensional points with a fixed relative position, so as to calculate the required parameters. The commonly used calibration boards include checkerboard calibration board and two-dimensional code calibration board. The two-dimensional code calibration board is compared with the chessboard. The grid calibration board has the advantage that it can be distinguished after upside-down and left-right. At the same time, because of the complexity of the graphics, it puts forward higher requirements for lighting and camera resolution. Under the conditions of use in this article, since the surveillance cameras are all facing the surveillance scene, there is no upper and lower In the reversed situation, at the same time, the checkerboard calibration board is more suitable for the use environment of the present invention in order to adapt the calibration method to lower-cost surveillance cameras and to arrange the system in this article in a darker indoor ambient light condition.

Modify the ORB_SLAM3 framework code to save the photo taken by the current camera and the camera pose at a certain moment. In actual operation, select the origin position of the world coordinate system, and the surveillance camera slowly moves from the origin to the direction of the checkerboard calibration board. In this process, ORB_SLAM3 is used to estimate the pose of the surveillance camera in real time. When the surveillance camera moves to the chessboard When the grid is in front of the calibration board, close the program and save the photos taken by the current surveillance camera and the pose of the camera;

S22. Internal parameter calibration of surveillance camera:

Place the checkerboard calibration board within the range of the surveillance camera, move the checkerboard calibration board from multiple angles, record a video, extract the frames in the video, identify the checkerboard, and use the Zhang Zhengyou calibration method to calibrate the internal parameters and distortion of the surveillance camera.

S23. Surveillance camera external parameter calibration:

The calculation of the pose is a method to solve the camera coordinate system and the target coordinate system according to the actual three-dimensional position information of the target feature point and the two-dimensional position of the target feature point in the image. If the three-dimensional coordinates of the feature points on the calibration board are known, this paper will use the direct linear transformation method to construct an augmented matrix [R|T] containing 12 unknowns to represent the transformation between the camera coordinate system and the target coordinate system. . Select at least 6 pairs of corresponding point pairs of known three-dimensional space point coordinates and two-dimensional pixel point coordinates to solve the unknowns in the augmented matrix, thereby realizing the calculation of the camera pose.

Assuming that the homogeneous coordinate corresponding to a feature point P ₁ on the calibration plate in space is P ₁ =(X, Y, Z, 1) ^T , mark the corresponding two-dimensional point of this point in the image of the surveillance camera as x ₁ = (u ₁ , v ₁ , 1) ^T , according to the calculation method of direct linear transformation, let the 3×4 augmented matrix [R|T] be expanded into the form:

In the above formula, the pixel coordinates X, Y, and Z of a certain two-dimensional point in the image of the monitoring camera u ₁ , v ₁ are the three-dimensional coordinates of the corresponding point and s is the scale.

According to the linear transformation of the matrix, the scale coefficient s is eliminated by the last row, and we get:

In order to simplify the representation, each row of the augmented matrix of formula (1) is represented by vectors t ₁ , t ₂ , t ₃ :

Then the equation for the vector representation can be obtained as follows:

In equations (5) and (6), t is the vector to be solved. According to (5) and (6), each feature point contains two unknown constraint equations. If there are N pairs of corresponding point pairs of three-dimensional coordinates and two-dimensional coordinates, The characteristic equation can be written in the form shown below:

According to equation (7), it can be seen that there are 12 unknowns in total, so at least six pairs of corresponding point pairs are needed to solve the above equation. The SVD method solves the equation by least squares.

For the 42 three-dimensional points P on the checkerboard calibration board and their projections p on the normalized plane, the pose R and t of the camera were previously calculated using the method of direct linear transformation, and its Lie algebra is expressed as ξ. Assume that the spatial coordinates of a certain calibration plate corner point in space are P _i =[X _i , Y _i , Z _i ] ^T , and its projected coordinates _ui =[u _i , v _i ] ^T . The relationship between pixel coordinates and spatial point positions is shown in the formula:

Written in matrix form:

s _i u _i =Kexp(ξ^)P _i

There is an error in this equation due to the unknown camera pose and the noise of the observation point. Sum the errors, construct a least squares problem, and then find the best camera pose to minimize it. The sum of the error terms is as follows:

The solution is solved by Gauss-Newton algorithm, and the camera pose transformation matrix with the smallest error term is obtained.

The above can solve the camera pose on the premise of knowing the coordinates of the feature points of the checkerboard calibration board. Due to the scale uncertainty of the monocular camera, it is impossible to determine each feature based on the size and style of the calibration board observed in the image. The relative positional relationship between the points, according to the actual situation where the system of the present invention is located, as shown in Figure 4, the scale information can be obtained from two aspects:

(1) The Z-axis absolute distance between the feature point and the optical center of the RGBD camera can be obtained from the depth map, so as to obtain the absolute position information of each feature point.

(2) Measure the size of the checkerboard on the calibration board to obtain the relative positional relationship between each feature point on the checkerboard.

Taking the first characteristic corner point of the upper left corner of the checkerboard calibration plate as the coordinate origin, the horizontal axis is the x-axis, the vertical axis is the y-axis, and the vertical upward is the z-axis to form the calibration plate coordinate system. Knowing the three-dimensional coordinates of the feature points on the checkerboard calibration board to solve the camera pose, the pose transformation matrix of the camera in the calibration board coordinate system can be solved. The transformation matrix under the calibration board coordinate system obtained when the checkerboard calibration board is photographed with a surveillance camera is denoted as T _mb . The matrix is denoted as T _cb , and the transformation matrix of the RGBD camera relative to the world coordinate system given by the visual odometry is T _cw .

Let the coordinates of a feature point on the checkerboard calibration board in the calibration board coordinate system be P _b =(X, Y, 0, 1) ^T , the coordinate mark P _m of this point in the surveillance camera camera coordinate system, in RGBD The coordinate mark P _c under the camera coordinate system and the coordinate mark P _w under the world coordinate system of the visual odometer are:

P _c =T _cb P _b (10)

P _m =T _mb P _b (11)

Multiply the left and right sides of equation (10) by T _cb ^-1 to get:

P _b =T _cb ^-1 P _c (12)

Substitute equation (12) into (11) to have:

P _m =T _mb T _cb ^-1 P _c (13)

According to the Euclidean transformation definition (13), T _mb T _cb ^-1 represents the transformation method from the RGBD camera coordinate system to the monitoring camera coordinate system, so T _mb T _cb ^-1 is denoted as T _mc .

From the formula (14), the transformation matrix T _mw from the world coordinate system to the monitoring camera coordinate system can be solved.

T _mw =T _mc T _cw (14)

S3. Position calculation: According to the position and attitude data obtained in step S2 and the monitoring video stream provided by the monitoring camera, the position of the person in the image is identified and the position of the person in the three-dimensional point cloud map is given.

Personnel positioning and trajectory tracking based on surveillance video has the advantages of high precision and good stability. The following will focus on the pedestrian detection and localization algorithm based on surveillance video.

As shown in Figure 5, the process of tracking and calculating the pedestrian's position in the surveillance camera is as follows:

S31, according to the type of self-monitoring camera and whether the movement has occurred, select whether to enter the posture correction process, if yes, enter step S32, otherwise enter step S33;

S32, if entering the attitude correction process, extract the vanishing point in the image and compare the position of the vanishing point, determine whether the monitoring screen rotates according to whether the vanishing point moves, and update the rotation;

Specifically, regarding the attitude correction of surveillance cameras, some surveillance cameras have the function of manual or automatic rotation, which can cause changes in the yaw angle and pitch angle of the camera attitude. The workload is large, and it is unrealistic to re-calibrate the external parameters every time the posture of the surveillance camera changes. Therefore, it is necessary to provide a workflow for automatically updating and correcting posture.

1) Vanishing point extraction

According to the principle of projective geometry, in the case of perspective deformation, the group of parallel lines in the real world will intersect at an infinite point, and the projection of the intersection point on the imaging plane is called the vanishing point. When parallel lines in the real world are parallel to the imaging plane, the vanishing point is at infinity of the imaging plane. However, when there is a non-parallel relationship between the group of parallel lines and the imaging plane, the vanishing point will lie within a limited distance of the imaging plane, even within the imaging area.

Vanishing points have some important properties:

a. In the real world, lines parallel to each other and lines parallel to each other all point to the same vanishing point;

b. The vanishing point corresponding to the straight line must be located in the direction of the projected light of the straight line on the image plane;

c. The position of the vanishing point has nothing to do with the roll angle, only the pitch angle and the yaw angle.

The vanishing point is an important feature formed on the image plane after perspective projection, which can provide a large amount of structural information and orientation information for scene analysis or be used to measure the parameters of the camera itself. Therefore, vanishing points are widely used in rectangular structure estimation and matching, 3D reconstruction, camera calibration, and azimuth estimation.

Since this system is used inside buildings, most of the building walls and floors are flat, so a fixed vanishing point can be extracted. According to whether the vanishing point moves or not, it is determined whether the monitoring screen rotates. First, the vanishing point is extracted. The process is as follows:

(1) Take the internal parameters and distortion of the surveillance camera, de-distort the original picture, and obtain the de-distorted picture;

(2) Use the LSD line segment extractor to extract line segments on the dedistorted picture;

(3) Screening line segments according to their lengths and taking more than 60 pixels as valid line segments;

(4) Calculate the angle of each line segment using the Hough transform;

(5) The line segments are clustered by angle and divided into three categories;

(6) Use the least squares method to find the nearest point for each type of straight line;

(7) Select the smallest sum of coordinates among the three vanishing points as the reference point;

As shown in Figure 6, red, green and blue are three types of line segments, and the center of the red circle is the specific location of the reference point.

2) Attitude correction

As shown in Figure 6, the coordinates of the vanishing point before the rotation are (x ₀ , y ₀ ), and the coordinates of the vanishing point after the rotation are (x ₁ , y ₁ ), then the variation of the yaw angle δyaw and the pitch angle can be calculated The amount of change δpitch is shown in equations (15) and (16).

δyaw=arctan(x ₁ -x ₀ )#(15)

δpitch=arctan(y ₁ -y ₀ )#(16)

Since the pose of the surveillance camera is represented by the transformation matrix T _mw , the change of the angle is converted into a matrix form to represent the same rotation, the rotation in the yaw direction is denoted as R _x , and the rotation in the pitch direction is denoted as R _y As shown in formula (17) (18):

In view of the fact that the camera can only rotate and cannot move, it can be considered that the translation amount of the optical center coordinate is approximately zero, and the rotation matrix in the two directions is multiplied and then combined with the translation vector t=(0, 0, 0) ^T to form a transformation The matrix T ₀₁ represents the transformation from the camera coordinates before rotation to the camera coordinates after rotation. The transformation matrix of the new surveillance camera required for positioning is denoted as T _mw ′, then there are:

T _mw ′=T ₀₁ T _mw #(19)

S33, enter the positioning process, first take a frame of surveillance video image;

S34, perform target detection on the pedestrian in the image, and obtain the coordinates of the target frame;

S35, target tracking is performed on the detected target frame, and the corresponding personnel position coordinates are given;

S36. Calculate the spatial position coordinates of the personnel according to the calibration parameters of the surveillance camera;

Specifically, the location calculation process is as follows:

1) Target recognition and tracking

Detect and track multiple targets in the video stream based on the SORT (Simple Online And Realtime Tracking) algorithm, and display the id of each target. This algorithm uses a powerful CNN detector - yolov3 to detect the target, and then uses the Kalman filter (Kalman filter) and the Hungarian algorithm to track the detected target. The algorithm can achieve accurate multi-pedestrian tracking on the premise of meeting the real-time requirements.

2) Pedestrian position calculation

The world coordinate system is the coordinate system of the point cloud map obtained by 3D reconstruction, that is, the position of the optical center of the first frame is the origin, and the Z-axis direction is opposite to the direction of gravity. The transformation matrix T _mw from the world coordinate system to the surveillance camera coordinate system consists of a 3×3 rotation matrix R and a 1×3 translation vector t. Through the actual measurement, the height of the ground in the world coordinate system is h, and the camera's internal parameter matrix K is calibrated through the calibration plate.

According to the projection equation (20), the coordinates X _w , Y _w of pedestrians on the ground can be solved:

The specific solution process is as follows:

According to the pose transformation matrix of the surveillance camera, the camera model optical center P _ow = (X _ow , Y _ow , Z _ow ) of the surveillance camera is obtained:

P _ow =-RT ^t #(21)

Take the midpoint M(u, v) of the lower bottom edge of the target frame of the person to be positioned, calculate the spatial position of point M on the normalized plane according to the projection equation, set the depth d to be 1 meter, and solve it according to formula (22) The spatial coordinates of the point M on the normalized plane P _m =(X _m , Y _m , Z _m );

According to the height of the ground in the world coordinate system as h, write the equation of the plane where the ground is located, as shown in Equation (24), convert the plane where the ground is located to a point and a normal vector on the plane to represent:

z=h#(23)

p ₀ = (0, 0, h),

写作参数方程如(25)所示，其中

为射线的方向向量，

t为参数t∈[0，∞)。will ray

The writing parameter equation is shown in (25), where

is the direction vector of the ray,

t is the parameter t∈[0, ∞).

与地面所在平面交点为P_g，则有：set ray

The intersection point with the plane where the ground is located is P _g , then there are:

After finishing formula (26), we get:

From the distributive law of vector dot product:

交点P_g的坐标即为行人在世界坐标系下的三维坐标。From this, the intersection point can be solved

The coordinates of the intersection point _Pg are the three-dimensional coordinates of the pedestrian in the world coordinate system.

S37, if the positioning is not terminated, acquire the next frame of image and return to step S33, otherwise end.

S4 Visual display: Display the 3D point cloud map of the scene in step S1 and the position of the characters in the 3D point cloud map in step S3, and provide a GUI interface for user interaction and supervision.

The system designed and implemented by the present invention includes a 3D display interface based on the Three.JS graphics library capable of displaying the three-dimensional point cloud, the camera model and the trajectory, and its functional logic is as follows.

1) 3D point cloud loading

The present invention can load multiple three-dimensional point cloud PCD files located on the server and correctly display them in the 3D display interface, and its function logic is as follows:

(1) Find all PCL files in the specified directory, and get the PCL file name;

(2) According to the file name, use the FileLoader that comes with JavaScript to read the PCD file to obtain the data of the 3D point cloud, such as the number of point clouds, point coordinate sets, color sets, etc.;

(3) Remap the color gamut of the point cloud to the color gamut range supported by the Three.Js graphics library, the formula is as follows;

(4) Combining the typed array of JavaScript and the data class of Three.Js, correspondingly fill the 3D point cloud data into the graphics class object provided by Three.Js; the typed array is shown in Figure 7.

(5) Each created graphic class object is named with a file name and added to the display interface;

(6) According to the system settings, only the point cloud viewed by default is displayed, and other point clouds are reserved for backup.

So far, the functional logic of loading 3D point clouds in the system has been sorted out. Figure 8 is an example of loading 3D point clouds.

2) Camera model loading

The present invention can add a camera model with the correct orientation in the correct position in the 3D display interface according to the camera calibration parameters located on the server side, so that the overall layout is clear at a glance, and its functional logic is as follows:

(1) Read the specified parameter xml file to obtain the number of cameras and the corresponding name and transformation matrix;

(2) Use the FileLoader that comes with JavaScript to read the camera model obj file, and create a specified number of Three.Js graphic class objects accordingly;

(3) Adjust the world coordinates and orientation of each graphic object according to the transformation matrix, and name it;

(4) Add each adjusted graphic object to the display interface.

So far, the functional logic of loading the camera model in the system has been sorted out. Figure 9 is a sample diagram of the loaded camera model, and the background is the point cloud in Figure 8.

The specific content displayed includes multiple 3D point clouds and their corresponding cameras and trajectories. When a certain point cloud is selected to be displayed, other point clouds and corresponding cameras and trajectories are hidden.

3) Trajectory information transmission and rendering optimization

The server of the present invention can receive Socket information, based on this, the transmission and drawing functions of the obtained trajectory information are realized, and the problem that the line thickness of the current Web graphics library cannot be adjusted is solved. The function flow is shown in Figure 10, and the function logic is as follows:

(1) Accept Socket information, and obtain track-related information in real time, including track ID, operation type and track data;

(2) If the operation type is track update: first search for the corresponding ID in the existing track library, if not found, create a Three.Js line object, fill in the corresponding track data, and add it to the display interface; if found, then Update the existing Three.Js line class object with the newly obtained trajectory data, and redraw it in the display interface;

(3) If the operation type is track deletion, search for the corresponding ID in the existing track library, and delete the Three.Js line class object corresponding to the ID in the display interface;

The flickering problem of track update: When the track is drawn, the update function provided by Three.Js will cause the track to keep flickering. The reason is that the redrawing of the track is constantly deleted when the track is updated. ” method, specifically: retain the old track when processing the newly obtained track information, and delete the old track from the display interface after the new track is overwritten and drawn. This can not only ensure the display effect of the track, but also reduce the memory usage of the system.

The problem of the thickness of the track: When the track is drawn, the line object provided by Three.Js cannot adjust the thickness, which makes the track difficult to identify in the display interface. This patent proposes a method of "copying a single track with partial overlap" to solve this problem. The purpose is: After updating the coordinate data of the Three.Js line object, copy the object multiple times when drawing in the display interface, and gradually pan around it, successfully achieving the effect of thickening the track and passing the performance test.

(4) Object binding and perspective operation

As mentioned above, the 3D display interface of the present invention can display multiple point clouds and their corresponding cameras and trajectories, and can ensure that when a certain point cloud is selected to be displayed, other point clouds and corresponding cameras and trajectories are hidden. This patent uses the object binding operation provided by the Three.Js graphics library to correspond the camera and the trajectory to the specified point cloud according to the ID, thereby realizing the function of switching the point cloud scene.

In addition, the 3D display interface of the present invention realizes the translation, rotation, and zoom operations of the viewing angle based on the event listener mechanism of Web and JavaScript, and has the ability of multi-terminal adaptation.

Regarding the GUI system interface, the overall system architecture is shown in Figure 11.

The system adopts the front-end and back-end separation architecture, and the front-end uses the Vue development framework to build a visual display platform, providing visualization functions such as 3D point cloud, pedestrian trajectory, and real-time monitoring.

The back-end is implemented based on the SpringBoot development framework, which provides monitoring management, pedestrian counting, pedestrian trajectory, log management and other computing and storage functions for front-end visualization. The front-end and back-end communicate through Http and WebSocket. Each function is described as follows:

Monitoring management: Manage the monitoring of system access, mainly including monitoring configuration management and status management.

Pedestrian trajectory: Use the pedestrian positioning and tracking algorithm to obtain the coordinates of the pedestrian trajectory in real time, and realize the visualization of the pedestrian trajectory and the persistent storage of the historical trajectory.

Pedestrian counting: Calculate the number of pedestrians based on the number of trajectories in the current scene, and display them in real time on the visual interface.

Log management: record all behaviors generated by the system and provide query functions to help improve system security.

The backend of the system uses the Mysql database to achieve data persistence, which can provide efficient data reading and writing; the kafka message queue is used to integrate the pedestrian location tracking algorithm into the system. The specific process is as follows:

(1) The pedestrian location tracking algorithm analyzes the video frame, and sends the coordinates of the pedestrian in the three-dimensional coordinate system to the topic named slam in Kafka. The number of partitions is set to 1 to ensure the orderliness of the message;

(2) The backend of the system monitors Kafka in real time, reads the pedestrian trajectory coordinates calculated by the pedestrian positioning tracking algorithm, and stores the pedestrian trajectory coordinates persistently in the MySql database on the one hand, and sends the pedestrian trajectory coordinates to the front end through WebSocket on the other hand;

(3) Whenever the front end receives the pedestrian trajectory coordinates pushed by the back end, it draws the trajectory points in the 3D point cloud map according to the coordinates. When a certain number of trajectory points accumulate, a more obvious pedestrian trajectory is formed.

Using Kafka to integrate algorithms is conducive to improving the scalability of the system, flexibly extending algorithm functions, and providing pluggable function implantation.

The front-end interface of the visual video surveillance system of the present invention is shown in FIG. 13 , including a 3D display interface, video surveillance, an abnormal item monitoring switch, a track list, and an abnormal event list. The upper right corner can switch floors. After switching, the point cloud, camera and trajectory on the left and video surveillance on the right can be updated simultaneously; the lower left corner can selectively enable the functions of this patent, such as target tracking, flame alarm, etc.

When pedestrians are detected in the video surveillance area on the right and the trajectories of pedestrians are calculated, these trajectories will be updated and drawn synchronously on the left side, and the corresponding entries will be updated in the track list area on the lower side, as shown in Figure 12, the trajectories are clearly visible , and use the camera model to show the position and orientation of the camera.

When a fire or smoke is detected in the video surveillance, an alarm prompt will pop up, which is convenient for the user of this patent to find the abnormal situation in time, and at the same time, the corresponding table item will be updated in the abnormal event list in the lower right corner, as shown in Figure 14. The overall system runs smoothly and meets the performance requirements.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Each embodiment in this specification is described in a related manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the apparatus embodiments, the electronic device embodiments, the computer-readable storage medium embodiments, and the computer program product embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and for relevant details, refer to the method embodiments part of the description.

The above-mentioned embodiments are only specific implementations of the present application, and are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited thereto. Detailed description, those of ordinary skill in the art should understand: any person skilled in the art is within the technical scope disclosed in this application, and it can still modify the technical solutions described in the foregoing embodiments or can easily think of changes, Alternatively, equivalent replacements are made to some of the technical solutions; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the present application. All should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (7)

1. a monitoring video personnel positioning and tracking method based on visual slam, is characterized in that, use the depth camera to carry out three-dimensional reconstruction to the environment, obtain the point cloud map of the environment; External parameter calibration is combined under the scene of the monitoring camera, by the calibration board Perform external parameter calibration on the surveillance camera to obtain the position and attitude information of the surveillance camera; track the person through the surveillance camera, use the deep neural network to identify the person in the surveillance image, and use the inverse perspective transformation based on the previously calibrated position and attitude. The principle of the method is to calculate the three-dimensional position of the person on the ground based on the a priori of the pedestrians appearing in the monitoring, so as to realize the drawing of the person's trajectory in the constructed point cloud map, and make it appear in the point cloud map. 2. the monitoring video personnel positioning tracking method based on visual slam according to claim 1, is characterized in that, comprises the following steps: S1. 3D reconstruction based on visual slam: construct a 3D point cloud map of the scene and record the data required for external parameter calibration, and use the RGBD image stream and inertial sensor data provided by the depth camera to construct a 3D point cloud map of the scene based on visual odometry And save it to the file; in the process of 3D reconstruction, shoot the checkerboard calibration board and record the position and attitude of the camera at this time; S2, calibration: the surveillance camera takes a picture of the surveillance image of the checkerboard calibration board, and combines the calibration data provided during the 3D reconstruction in step S1 to calculate the location and attitude of the surveillance camera in the 3D point cloud map; S3. Position tracking and calculation of pedestrians in the surveillance camera: track the person, and identify the position of the person in the image based on the position and attitude data obtained in step S2 and the surveillance video stream provided by the surveillance camera. the location in the point cloud map; The process of tracking and calculating the pedestrian's position in the surveillance camera is as follows: S31, according to the type of self-monitoring camera and whether the movement has occurred, select whether to enter the posture correction process, if yes, enter step S32, otherwise enter step S33; S32, if entering the attitude correction process, extract the vanishing point in the image and compare the position of the vanishing point, determine whether the monitoring screen rotates according to whether the vanishing point moves, and update the rotation; S33, enter the positioning process, first take a frame of surveillance video image; S34, perform target detection on the pedestrian in the image, and obtain the coordinates of the target frame; S35, target tracking is performed on the detected target frame, and the corresponding personnel position coordinates are given; S36. Calculate the spatial position coordinates of the personnel according to the calibration parameters of the surveillance camera; S37, if the positioning is not terminated, acquire the next frame of image and return to step S33, otherwise end. 3. the monitoring video personnel positioning tracking method based on visual slam according to claim 2, is characterized in that, the construction process of the three-dimensional point cloud map of step S1 scene increases point cloud map processing thread, for accepting each incoming Frame the camera's pose information and RGBD image frames, and output an accurate point cloud map. The specific process is as follows: S141. Screen the incoming pose information and RGBD image frame of each frame of the depth camera. When the camera angle change between the current frame and the last selected frame is greater than 10° and the displacement change is greater than 2 meters, select the current frame , and perform subsequent point cloud map generation operations; S142, calculate the point cloud block of the current frame, and rotate it to a unified world coordinate system; S143, splicing and merging the point cloud blocks generated by all frames to obtain an overall point cloud map, filtering and removing outliers on the point cloud map, so as to compress the data volume of the point cloud map and optimize the visual perception of the map at the same time ; S144. When a loopback occurs during the mapping process, ORB-SLAM3 re-optimizes the pose of the selected frame, re-splices the point cloud, and re-does the point cloud processing operation according to step S143. 4. the monitoring video personnel positioning and tracking method based on visual slam according to claim 2, is characterized in that, step S2 external parameter calibration method is: S21. The surveillance camera shoots the checkerboard calibration board: select the origin position of the world coordinate system, and the surveillance camera slowly moves from the origin position to the checkerboard calibration board. In this process, ORB_SLAM3 is used to estimate the pose of the surveillance camera in real time. , when the surveillance camera moves to the front of the checkerboard calibration board, close the program and save the photos taken by the current surveillance camera and the pose of the camera; S22. Surveillance camera internal parameter calibration: place the checkerboard calibration board within the range of the surveillance camera, move the checkerboard calibration board from multiple angles, record a video, extract frames from the video, identify the checkerboard, and use Zhang Zhengyou's calibration method to calibrate the surveillance camera internal reference and distortion; S23. Surveillance camera external parameter calibration: According to the actual three-dimensional position information of the target feature point and the two-dimensional position of the target feature point in the image, the direct linear transformation method is used to solve the camera coordinate system and the target coordinate system, so as to realize the monitoring camera and the three-dimensional Calculation of relative positional relationship between point cloud maps. 5. the monitoring video personnel positioning and tracking method based on visual slam according to claim 2, is characterized in that, in step S36, the spatial position coordinate calculation method of personnel is: According to the pose transformation matrix of the surveillance camera, the camera model optical center P _ow = (X _ow , Y _ow , Z _ow ) of the surveillance camera is obtained: P _ow =-RT ^t #(21) Take the midpoint M(u, v) of the lower bottom edge of the target frame of the person to be positioned, calculate the spatial position of point M on the normalized plane according to the projection equation, set the depth d to be 1 meter, and solve it according to formula (22) The spatial coordinates of the point M on the normalized plane P _m =(X _m , Y _m , Z _m );

According to the height h of the ground in the world coordinate system, the equation of the plane where the ground is located is given as shown in equation (24), and the plane where the ground is located is converted into a point and a normal vector on the plane to represent: z=h#(23) p ₀ = (0, 0, h),

will ray

The writing parameter equation is shown in (25), where

is the direction vector of the ray,

t is the parameter t∈[0, ∞),

set ray

The intersection point with the plane where the ground is located is P _g , then:

After finishing formula (26), we get:

From the distributive law of vector dot product:

Solve for the intersection

The coordinates of the intersection point _Pg are the three-dimensional coordinates of the pedestrian in the world coordinate system. 6. the monitoring video personnel locating and tracking method based on visual slam according to claim 2, is characterized in that, also comprises step S4 visual display: the three-dimensional point cloud map of step S1 scene and step S3 character are in three-dimensional point cloud map The location of the 3D point cloud is displayed, and a GUI interface is provided for user interaction and supervision. The specific content displayed includes multiple 3D point clouds and their corresponding cameras and trajectories. When a certain point cloud is selected for display, other point clouds and corresponding cameras and trajectories are displayed. hide. 7. a monitoring video personnel positioning tracking system based on visual slam, is characterized in that, comprises following module, to realize the method of any one of claim 1-6: The 3D reconstruction module is used to construct the 3D point cloud map of the scene and record the data required for external parameter calibration. It runs only once in the entire use cycle, and uses the RGBD image stream and inertial sensor data provided by the depth camera to construct the scene's The 3D point cloud map is saved to the file for display by the visual display module; in the process of 3D reconstruction, the checkerboard calibration board and the position and attitude of the camera are recorded for the calibration module to use; The calibration module is used to measure the position and attitude of each surveillance camera in the 3D point cloud map. It runs only once in the entire use cycle. The surveillance camera takes a picture of the surveillance image of the checkerboard calibration board, combined with the calibration data provided during 3D reconstruction. Calculate the position and attitude of the surveillance camera in the 3D point cloud map for use by the position calculation module; The position calculation module is used to identify the position of the person in the image and give the position of the person in the 3D point cloud map for the visualization display module to display according to the position and attitude data obtained by the calibration module and the monitoring video stream provided by the monitoring camera. ; The visual display module is used to display the 3D point cloud map of the scene and the position of the characters in the 3D point cloud map, and provide a GUI interface for user interaction and supervision.

Images