CN114299106A - High-altitude parabolic early warning system and method based on visual sensing and track prediction - Google Patents
High-altitude parabolic early warning system and method based on visual sensing and track prediction Download PDFInfo
- Publication number
- CN114299106A CN114299106A CN202111066314.8A CN202111066314A CN114299106A CN 114299106 A CN114299106 A CN 114299106A CN 202111066314 A CN202111066314 A CN 202111066314A CN 114299106 A CN114299106 A CN 114299106A
- Authority
- CN
- China
- Prior art keywords
- falling
- parabolic
- image
- early warning
- point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a high-altitude parabolic early warning system and a high-altitude parabolic early warning method based on visual sensing and track prediction, wherein the method comprises the following steps of: acquiring an image through binocular camera video monitoring; the receiving end applies a convolutional neural network algorithm retrained on the basis of acceptance v4 to identify the class of the high-altitude falling object; the system keeps a standby state in a no-threat mode; when a threat mode exists, identifying falling objects, analyzing the possible stress mode to establish a mathematical model, performing combined prediction on data received by the double cameras by using a Kalman filtering method, predicting the falling point coordinates of the objects and transmitting the coordinates; the acousto-optic early warning module receiving the coordinates processes data rapidly, marks the falling point by using a striking laser lamp before the falling object falls to the ground and is assisted with audible alarm to remind the pedestrian, and the blocking net is used for intercepting to avoid the pedestrian from being injured. The invention can timely detect and early warn the high-altitude parabolic state, ensure the safety of personnel and reduce the accident risk.
Description
Technical Field
The invention relates to the field of security monitoring, in particular to a high altitude parabolic early warning system and method based on visual sensing and track prediction.
Background
In recent years, high-altitude parabolic events frequently occur, public safety of people is seriously harmed, and legal benefits of people are invaded. High altitude toss are an event of high-rise throws, either man-made or not, called "pain hanging over cities". Modern high buildings build hundreds of meters high frequently, and the behaviors of non-standard living habits and lack of public safety awareness of residential areas can cause the events. The object is thrown from tens of meters to hundreds of meters high above the ground, and the speed of falling to the ground can reach 30-45 meters per second. The threat is huge for people walking down the street or downstairs, and retrospective tracing of parabolas is extremely difficult.
In order to eliminate the potential safety hazard, the issuing of laws and regulations is important, technical innovation and design are also indispensable, and the improvement and development of the existing technical means are urgently needed. Society has come to a greater and greater call for a technology that can provide early warning and precaution.
At present, few and few schemes focusing on high-altitude parabolic detection are researched at home and abroad.
The method is generally adopted in China that a plurality of camera groups are used for carrying out networking monitoring on a household face, and camera cross deployment proportion is used for marking out, positioning and throwing a parabolic range to eliminate a blind area. The detection scheme is the most traditional high-altitude parabolic detection scheme, and has obvious defects that the detection scheme does not have the capability of identifying and capturing dynamic parabolas in a video, the falling points and the directions of the detection scheme cannot be predicted, and the detection scheme cannot effectively react to falling events to reduce harm to people.
In foreign countries, according to foreign literature, foreign processing means for high altitude parabolas still utilize traditional cameras and capturing means, and the occurrence of events is reduced through tracing and recording the offenders. Some inventors adopt the cloud networking to track the parabolic times of high altitude parabolas to determine corresponding punishment measures, and lack the early warning emergency measures after the corresponding accident occurs.
Therefore, a high-altitude parabolic early warning system based on visual sensing and trajectory prediction is needed to realize efficient and timely parabolic detection and intelligent early warning.
Disclosure of Invention
The invention aims to solve the technical problem of providing a high-altitude parabolic early warning system and method based on visual sensing and track prediction aiming at the defects in the prior art.
The technical scheme adopted by the invention for solving the technical problems is as follows:
the invention provides a high-altitude parabolic early warning method based on visual sensing and track prediction, which comprises the following steps of:
step 1, acquiring an image through binocular camera video monitoring, and performing morphological processing on an initial image by using a graying and Gaussian filtering method;
step 2, data are transmitted through a 5G communication module, a receiving end applies a convolutional neural network algorithm based on initiation v4 and retrained again to identify the category of the high-altitude falling object, and the high-altitude falling object is divided into two modes of no threat and threat;
step 3, the system keeps a standby state in a no-threat mode, the condition of false triggering caused by environmental factors or tiny disturbance is reduced, and the loss of energy and the system is reduced;
when the robot enters a threatening mode, the falling object is identified, meanwhile, a mathematical model established by analyzing the possible stress mode is used for carrying out combined prediction on data received by the double cameras by using a Kalman filtering method, and the falling point coordinates of the object are predicted and transmitted;
and 4, rapidly processing data by the acousto-optic early warning module receiving the coordinates, marking a drop point by using a striking laser lamp before the falling object falls to the ground, assisting with sound alarm to remind the pedestrian, intercepting by using a blocking net, and avoiding the pedestrian from being injured.
Further, the method for processing the image in the step 1 of the present invention includes:
converting the RGB color three-channel image into a single-channel grey-scale image, and performing Gaussian blur, wherein the Gaussian blur effect is achieved by introducing template operation, and the template formula is as follows:
m (I, J) is the value of the matrix element in the corresponding template, and the value of the pixel at the center point of the image is replaced by the pixel weighting operation of the coverage area of the template determined by the movement of the constructed template; processing the original data stream by using a creatbackgroundsustractorknn algorithm API of OpenCV to obtain a differential image after background subtraction, and performing further processing on the differential image, wherein the digital image processing method comprises graying and threshold processing; finally, analyzing a connected domain according to the optimized image to obtain the maximum outline of the image; video data for detecting a parabolic event is obtained.
Further, in step 2 of the present invention, the convolutional neural network algorithm retrained based on initiation v4 identifies the high-altitude falling object class, and the method thereof is as follows:
based on retraining based on the concept v4, according to the CNN theory, using transfer learning in image recognition, regarding each filter as a small feature grabber, constructing the uppermost softmax layer of the concept v4 according to the weight of the model trained by the concept v4, and retraining by using a training set adaptive to the use environment.
Further, when the threat mode is entered in step 3 of the present invention, the mathematical model established while identifying the drop includes:
the method comprises the following steps that information is collected, a top view and a side view of a parabolic track are selected and obtained, and the top view and the side view are respectively provided by cameras in two directions; the selected special track point information is the three-dimensional coordinate and time (x) of the parabolic motion casting point0,y0,z0,t0) Highest point three dimensional coordinate and time (x)1,y1,z1,t1) Three-dimensional coordinates and time (x) of a point in the descending process2,y2,z2,t2) (ii) a In order to simplify the operation, the model is simplified on the basis of respectively solving motion track vectors into three planes x, y and z; the x, y and z planes are determined to be the plane opposite to the building, namely the y plane, the ground is the z plane, and any side surface is the x plane;
side view trajectory analysis:
for high altitude parabolas, the side view is understood to be a trajectory similar to a parabola, for which the vertical coordinate analysis is performed, and if the orientation is positive, the object is subjected to the force analysis in the ascent phase:
maz＝-mg-vz 2λ
wherein f ═ vz 2λ is air resistance, let mg ═ c2λ, then there is
Arranging and integrating to obtain
Is obtained by calculation and arrangement
Finally, the integral of the velocity in the vertical direction is used as the falling height to form the formula:
obtaining a landing moment t' of the parabolic motion;
and then carrying out y coordinate analysis on the parabolic motion direction, and carrying out stress analysis on the parabolic motion direction if the parabolic motion direction is positive:
ma＝-vy 2λ
wherein "-" represents the force direction, and f ═ vy 2λ is the magnitude of the air resistance received, and is obtained by performing deformation integration
To obtain
The data brought into the acquisition track points can be obtained as vy1And vy2About vyoAnd because the speed change process exists
ma＝-vy 2λ
V can be obtained by three formulasyRegarding the expression of t, the landing time t' is already obtained in the vertical direction analysis, so the y coordinate of the final landing point is
Wherein v isxIs the speed of movement, v, of the object in the direction of the x-axisyIn the y-axis directionSpeed of movement of upper body, vzThe moving speed of an object in the z-axis direction is shown, wherein (X, Y, z) are coordinates of the position of the object, c is a constant number, lambda is an air resistance coefficient, m is the mass of the object, g is the gravity acceleration, f is the air resistance, and (X, Y) are horizontal and vertical coordinates of a drop point;
analyzing a top view track:
in the process of throwing the object, the track of the object in the horizontal direction cannot be changed by the force of gravity, the air resistance is always opposite to the moving direction, so that the track of the object cannot be deviated in the horizontal dimension, and the overlooking track is a line segment according to a starting point (x)0,y0,z0,t0) And highest point (x)1,y1,z1,t1) Can yield a linear relationship between the parabolic coordinates x, y:
and substituting the Y coordinate Y of the drop point obtained from the side view to obtain the X coordinate X of the drop point, namely obtaining the three-dimensional coordinate and time (X, Y, Z, t) of the drop point, achieving the purpose of predicting the drop point, and transmitting the object after predicting the drop point coordinate.
Further, in step 3 of the present invention, a kalman filter method is used for joint prediction, and the method includes:
the preparation process comprises the following steps:
calculating the error between the true value of the position at time i and the estimated value of the position
modeling implementation:
first, a time update equation is established:
where u (i) is an input amount, i.e., input position information,
is the predicted value at time i; a is a state transition matrix and B is a gain matrix; w (i) is a set noise;
establishing an error correlation matrix P for judging whether the estimated value is accurate:
q is a covariance matrix with respect to system noise, and is a constantly changing value; p (i) is the covariance matrix of the estimation error,
is a covariance matrix of the prediction error;
the above is the prediction process; after the predicted value at the corresponding moment is obtained through calculation, correcting the predicted value; the process is to make a comparison between the measured value and the actual value and further update the data;
wherein
Is a measurement margin, and Z (i) is a measurement value;
calculating the necessary Kalman gain:
kalman filtering is itself a minimum mean square error, the mean square error being the trace of P (i), so solving the trace for the prediction error covariance equation yields a trace for KkEquation of (c), optimally estimated KkThe value minimizes the trace of P (i) and the equation is left and right simultaneously for KkDerivation, and using the algorithm of matrix differentiation to make the differentiation result be zero to obtain KkThe calculation expression of (1):
Kk＝P′(i)HT(HP′(i)HT+R)-1
the kalman gain obtained from the above can be used to find and update the estimate. For the estimated value of the time i, the estimated value can be processed by using the Kalman coefficient to perform an operation so as to obtain the Kalman predicted value
Thus, an update of the kalman estimation value using the estimation value and the measurement margin is completed;
and updating the covariance matrix, and obtaining the covariance expressed by Kalman gain and the prediction error according to the derivation:
P(i)＝(I-KkH)P′(i)
according to continuous iteration and recursion, finally calculating until P (n) and X (n) to complete the estimation of each position on the track;
the pose information of the target object is updated at any time according to the condition of the target object through the model, the corrected track of the target object is obtained through effective loop detection, and the relevant data information is obtained according to simulation of a Matlab tool, so that the target object drop point area is obtained.
Further, the method of step 4 of the present invention is:
when the predicted object falling point coordinates are received, a data instruction is transmitted to the single chip microcomputer system through the signal data transmitting and transmitting module, the single chip microcomputer box controls the holder, the strong light lamp on the holder projects red strong light to a falling point area, and the buzzer gives an alarm; before falling objects fall to the ground, striking laser lamps are used for marking the falling points and assisting with sound alarm to remind pedestrians, and blocking nets are used for intercepting the falling objects, so that the pedestrians are prevented from being injured, and the personal and property safety of the pedestrians is protected to the maximum extent; and after the early warning is finished, the execution program enables the cradle head to automatically initialize and return to the original position and continue to be in a standby state.
The invention provides a high-altitude parabolic early warning system based on visual sensing and track prediction, which comprises:
the camera module is provided with a binocular camera and is used for acquiring a video frame for monitoring the parabolic area, and preprocessing images of the video frame to obtain video data for detecting parabolic events; the pretreatment comprises the following steps: performing morphological processing on the initial image by using a graying and Gaussian filtering method;
the communication module adopts a 5G communication module and is used for sending the image acquired by the camera module to a moving object target tracking detection module at a receiving end; and transmitting the drop point prediction result and the instruction to a networking alarm module and an early warning light source projection control module;
the moving object target tracking detection module is used for applying a convolutional neural network algorithm based on initiation v4 and retrained again to identify the category of the high-altitude falling object and divide the high-altitude falling object into two modes of no threat and threat; the system keeps a standby state in a no-threat mode, thereby reducing the condition of false triggering caused by environmental factors or tiny disturbance and reducing the loss of energy and the system; when the robot enters a threatening mode, the falling object is identified, meanwhile, a mathematical model established by analyzing the possible stress mode is used for carrying out combined prediction on data received by the double cameras by using a Kalman filtering method, and an object falling point coordinate is predicted and transmitted;
the networking alarm module is used for generating corresponding sound alarm when receiving the signal of falling of the parabola;
the early warning light source projection control module is used for identifying and reminding a falling object falling point, dividing the warning range of the system into regions with high enough precision, setting cloud platform control parameters corresponding to each region in the system in advance, feeding back predicted falling point coordinates after the processing of an image is completed by an algorithm, obtaining required parameters in a table look-up mode, and using a laser lamp for irradiation to finish early warning of pedestrians.
The invention has the following beneficial effects:
1. detecting the source of the high-altitude parabolic motion, establishing a mathematical model of the parabolic motion of the three-dimensional object, predicting the motion track and the falling point of the falling object, and sending out a warning.
2. The method comprises the steps of capturing the condition of a building through a high-speed camera, monitoring and detecting the falling of an object through neural network operation, and simultaneously storing falling object data and video data to the local to leave an effective material evidence of a high-altitude parabola.
3. The parabolic type is judged according to deep learning, the size of the influence is predicted, and corresponding early warning is sent out, so that the pedestrian can make correct response, and the serious casualties caused by falling objects are avoided to the maximum extent.
Drawings
The invention will be further described with reference to the accompanying drawings and examples, in which:
FIG. 1 is a general design framework diagram (two-dimensional flow chart) of a high-altitude parabolic early warning system based on visual sensing and trajectory prediction according to the present invention;
FIG. 2 is an example of single target tracking of a moving car surveillance video asset using background subtraction;
FIG. 3 is an example of multi-target tracking of moving car surveillance video assets using background subtraction;
fig. 4 is a learning curve of loss and accuracy of the training set and loss and accuracy of the test set.
Detailed Description
In order to make the objects, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a schematic flow chart illustrating steps of a high-altitude parabolic warning method based on visual sensing and trajectory prediction according to a first embodiment of the present invention, in an embodiment of the present invention, the high-altitude parabolic warning method based on visual sensing and trajectory prediction includes:
after acquiring the video image of the parabolic area recorded by the double cameras, preprocessing the image, wherein the preprocessing comprises the following steps: converting the RGB color three-channel image into a single-channel grey-scale image, and performing Gaussian blur, wherein the effect of Gaussian filtering is achieved by introducing template operation, and the template formula is as follows:
m (I, J) is the value of the matrix element in the corresponding template, and the value of the pixel at the central point of the image is replaced by the pixel weighting operation of the template coverage area determined by the movement of the constructed template, thereby achieving the purpose of Gaussian filtering. And then, obtaining an image of the moving object after background subtraction by using a background subtraction method, wherein the background subtraction method is similar to an inter-frame difference method, but the background subtraction is obtained by subtracting the original image and the background which is continuously updated by the original image. It is a way to obtain the contour of the moving object of the image by using the difference relation. Meanwhile, the processing mode of reducing the background to a certain extent can effectively improve and solve the 'false target interference' caused by the brightness change of the frame difference method. The background subtraction method is simple in calculation, because no moving target exists in the background image and the moving target exists in the current image, the two images are subtracted, the complete moving target can be obviously extracted, and the problem that the target extracted by the interframe difference method contains 'holes' is solved.
The method for realizing target detection by using the background subtraction method mainly comprises four links: background modeling, background updating, target detection and post-processing. Among them, background modeling and background updating are core problems in background subtraction. The quality of the background model establishment directly influences the target detection effect. Background modeling, namely, building a model capable of representing the background through a mathematical method. The most ideal method for obtaining the background is to obtain a frame of "clean" image without moving object as the background, fig. 2 and fig. 3 are examples of single-object tracking and multi-object tracking of moving car surveillance video resources by background subtraction, respectively.
And processing the original data stream by using a creatbackgroundsustractonKNN algorithm API of OpenCV to obtain a picture with the background subtracted, and performing further processing on the basis of the differential image, wherein the further processing comprises basic digital image processing technologies such as graying, threshold processing and the like. And finally, analyzing a connected domain according to the optimized image to obtain the maximum outline of the image. Video data for detecting a parabolic event is obtained.
After the video data of the parabolic event is intercepted, a moving object is judged through a trained convolutional neural network, transfer learning can be used in image recognition according to the CNN theory, and each filter is a small feature grabber. Google's decapmed has trained acceptance v4 to identify 1000 items, but their trained items do not fit well with the requirements of the high altitude parabola in the project, such as it is impossible to throw a car.
Therefore, the invention directly uses the weight of the model trained by the initiation v4, constructs the softmax layer at the top of the initiation v4 by itself, and retrains by using the training set of the invention.
A popular platform for this neural network is the GPU version of tensorflow2.0, using Adam gradient method to achieve gradient descent.
The training process is monitored by using the tensorboard, and the learning curves of loss and accuracy of the training set and loss and accuracy of the testing set in the 10 epochs before one training are shown in FIG. 4.
After the data is judged by using the algorithm, if no parabola is judged, the system enters a no-threat mode, and the system keeps a standby state in the no-threat mode, so that the condition of false triggering caused by environmental factors or tiny disturbance is reduced, and the loss of energy and the system is reduced; after the parabola is judged to appear, joint prediction is carried out on data received by the double cameras by using a Kalman filtering method through a mathematical model established by analyzing a possible stress mode of the parabola, the calculation of a drop point is beneficial to the prediction of the trajectory of the parabola, and the influence of conditions such as wind speed, damping and the like needs to be considered;
the kinematic and mechanical analysis, solving and establishing of the above mathematical model are as follows:
(1) modeling environment description
Description of the resistance: since the model aims to solve the problem of solving the trajectory and the drop point of the high-altitude parabola as accurately as possible, influence factors in practice, namely resistance, which is a non-negligible factor, cannot be simply described and solved by using the parabola model should be fully considered in the process of establishing the model, so that the problem of air resistance is also worth considering
Describing a problem analysis method, in the practical problem, the distribution of the cameras is wide enough, the data of the two cameras can be used for determining the throwing and storing time and the real-time three-dimensional coordinate of a thrown object, and the simulation calculation of the two aspects is integrated to obtain a complete high-altitude parabolic model
(2) Environmental assumptions
Assume one: during the falling process of the object, various influencing factors (such as the shape, the posture and the like of the object) can not change along with the change of the height of the falling object.
Assume two: during the falling process of the object throwing, the buoyancy of the air can not influence the object. (parabolic mass sufficient to allow air to have negligible resistance to it.)
Suppose three: the falling process of the parabola does not have any influence on the trajectory of the parabola by uncertain factors (such as birds, other objects and the like).
(3) Symbols used in simulation and their meanings
TABLE 1 symbols used in the simulation and their meanings
(4) Model building and solving
The method comprises the following steps that information is collected, a top view and a side view of a parabolic track are selected and obtained, and the top view and the side view are respectively provided by cameras in two directions; the selected special track point information is the three-dimensional coordinate and time (x) of the parabolic motion casting point0,y0,z0,t0) Highest point three dimensional coordinate and time (x)1,y1,z1,t1) Three-dimensional coordinates and time (x) of a point in the descending process2,y2,z2,t2) (ii) a In order to simplify the operation, the model is simplified on the basis of respectively solving motion track vectors into three planes x, y and z; the x, y and z planes are determined to be the plane opposite to the building, namely the y plane, the ground is the z plane, and any side surface is the x plane;
side view trajectory analysis:
for high altitude parabolas, the side view is understood to be a trajectory similar to a parabola, for which the vertical coordinate analysis is performed, and if the orientation is positive, the object is subjected to the force analysis in the ascent phase:
maz＝-mg-vz 2λ
wherein f ═ vz 2λ is air resistance, let mg ═ c2λ, then there is
Arranging and integrating to obtain
Is obtained by calculation and arrangement
Finally, the integral of the velocity in the vertical direction is used as the falling height to form the formula:
obtaining a landing moment t' of the parabolic motion;
and then carrying out y coordinate analysis on the parabolic motion direction, and carrying out stress analysis on the parabolic motion direction if the parabolic motion direction is positive:
ma＝-vy 2λ
wherein "-" represents the force direction, and f ═ vy 2λ is the magnitude of the air resistance received, and is obtained by performing deformation integration
To obtain
The data brought into the acquisition track points can be obtained as vy1And vy2About vyoAnd because the speed change process exists
ma＝-vy 2λ
V can be obtained by three formulasyRegarding the expression of t, the landing time t' is already obtained in the vertical direction analysis, so the y coordinate of the final landing point is
Wherein v isxIs the speed of movement, v, of the object in the direction of the x-axisyIs the speed of movement, v, of the object in the direction of the y-axiszThe moving speed of an object in the z-axis direction is shown, wherein (X, Y, z) are coordinates of the position of the object, c is a constant number, lambda is an air resistance coefficient, m is the mass of the object, g is the gravity acceleration, f is the air resistance, and (X, Y) are horizontal and vertical coordinates of a drop point;
analyzing a top view track:
in the process of throwing the object, the track of the object in the horizontal direction cannot be changed by the force of gravity, the air resistance is always opposite to the moving direction, so that the track of the object cannot be deviated in the horizontal dimension, and the overlooking track is a line segment according to a starting point (x)0,y0,z0,t0) And highest point (x)1,y1,z1,t1) Can yield a linear relationship between the parabolic coordinates x, y:
and substituting the Y coordinate Y of the drop point obtained from the side view to obtain the X coordinate X of the drop point, namely obtaining the three-dimensional coordinate and time (X, Y, Z, t) of the drop point, achieving the purpose of predicting the drop point, and transmitting the object after predicting the drop point coordinate.
(5) Influence of wind speed on motion trajectory
During the falling process of the object, the wind speed may affect the trajectory and the falling point, if the falling object has a large mass, the influence of the wind speed on the falling object is smaller than the influence of the air resistance on the falling object, and the influence of the wind speed on the falling trajectory of the object can be ignored, and if the wind speed is large and the object has a light mass, the influence of the wind speed on the falling trajectory of the object cannot be ignored.
For modeling after wind speed becomes an influence factor of a parabola, two schemes exist, namely, like a parabola motion track, the wind speed is projected to three planes of a three-dimensional space, and the scheme causes large calculation amount and is different from the modeling in the prior art. The other is to consider the wind speed as a moving three-dimensional space, and the parabolic motion is regarded as a parabolic motion relative to the moving three-dimensional space, compared with the previous solution, which can systematically migrate the system for calculating the landing point, and is simpler than the previous solution.
(6) Modeling assessment
The model can directly calculate the final drop point of the parabola, can meet the requirement of actual problems on drop point prediction, but still has certain errors because the model does not take emergency situations into consideration and optimizes the problems in some places, so that the warning range is enlarged by expanding the judged drop point, and the higher effectiveness of drop point judgment is achieved.
After modeling is completed, an equation about the projectile landing point is obtained according to the double-camera landing point motion modeling data. The drop point equation can be expanded to obtain the motion trail of the throwing point, and the position of the throwing point can be obtained by reversely deducing the drop point and the motion equation. In order to improve the model of predicting the drop point, a loop and feedback process is added on the basis of the drop point, so that the process of continuously correcting the drop point in the process of projectile motion can be realized, and the stability and the robustness of the system are improved.
Therefore, a Kalman filtering algorithm is introduced, the Kalman filtering algorithm is wide and powerful, the Kalman filtering algorithm can be used for estimating the past and current states of the signals, and the future motion track can be estimated and presumed according to the existing states so as to correct the falling point.
Kalman filtering essentially accomplishes two processes, namely prediction and rectification. The basic idea is to use the minimum mean square error as the best estimation criterion, use the estimation value of the previous moment and the observation value of the current moment to jointly estimate and update the state variable, calculate the estimation value of the current moment, and establish a system equation and an observation equation according to an algorithm to estimate the minimum mean square error of the signals to be processed.
The preparation process comprises the following steps:
calculating the error between the true value of the position at time i and the estimated value of the position
modeling implementation:
first, a time update equation is established:
where u (i) is an input amount, i.e., input position information,
is the predicted value at time i; a is a state transition matrix and B is a gain matrix; w (i) is a set noise;
establishing an error correlation matrix P for judging whether the estimated value is accurate:
q is a covariance matrix with respect to system noise, and is a constantly changing value; p (i) is the covariance matrix of the estimation error,
is a covariance matrix of the prediction error;
the above is the prediction process; after the predicted value at the corresponding moment is obtained through calculation, correcting the predicted value; the process is to make a comparison between the measured value and the actual value and further update the data;
wherein
Is a measurement margin, and Z (i) is a measurement value;
calculating the necessary Kalman gain:
kalman filtering is itself a minimum mean square error, the mean square error being the trace of P (i), so solving the trace for the prediction error covariance equation yields a trace for KkEquation of (c), optimally estimated KkThe value minimizes the trace of P (i) and the equation is left and right simultaneously for KkDerivation, and using the algorithm of matrix differentiation to make the differentiation result be zero to obtain KkThe calculation expression of (1):
Kk＝P′(i)HT(HP′(i)HT+R)-1
the kalman gain obtained from the above can be used to find and update the estimate. For the estimated value of the time i, the estimated value can be processed by using the Kalman coefficient to perform an operation so as to obtain the Kalman predicted value
Thus, an update of the kalman estimation value using the estimation value and the measurement margin is completed;
and updating the covariance matrix, and obtaining the covariance expressed by Kalman gain and the prediction error according to the derivation:
P(i)＝(I-KkH)P′(i)
the process of kalman predicting the trajectory of a projectile can be summarized as follows:
step 1, first P (0) and X (0) are known, then P '(1) is calculated from P (0), and then K1 is calculated from P' (1), after these parameters exist, X (1) can be estimated by combining observed values, and P (1) is updated by K1.
And 2, calculating P '(2) from P (1) and K2 from P' (2) in the next updating process, estimating X (2) by combining the observed values after the parameters exist, and updating P (2) by using K2.
According to continuous iteration and recursion, finally calculating until P (n) and X (n) to complete the estimation of each position on the track;
the pose information of the target object is updated at any time according to the condition of the target object through the model, the corrected track of the target object is obtained through effective loop detection, and the relevant data information is obtained according to simulation of a Matlab tool, so that the target object drop point area is obtained.
After receiving the simulation result, the data instruction can be transmitted to the single chip microcomputer system by the transmitting and transmitting module of the signal data, the single chip microcomputer box controls the holder, the highlight lamp on the holder can project red highlight to a drop point area, and the buzzer gives an alarm. Use striking laser lamp to mark and assist with audible alarm and remind the pedestrian the falling point before falling object to fall to the ground to use the barrier to intercept, avoid the pedestrian to receive the injury. Thereby protecting the personal and property safety of the pedestrian to the maximum extent. And after the early warning is finished, executing the program to enable the cloud platform to automatically initialize and return to the original position and continue to be in a standby state.
It should be noted that, for the sake of simplicity, the above-mentioned method embodiments are described as a series of acts or combinations, but those skilled in the art should understand that the present invention is not limited by the described order of acts, as some steps may be performed in other orders or simultaneously according to the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no acts or modules are necessarily required of the invention.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
The foregoing is provided for illustration of the present invention, and variations in the detailed description and the application scope will occur to those skilled in the art based on the concept of the embodiment of the present invention.
Claims (8)
1. A high-altitude parabolic early warning method based on visual sensing and trajectory prediction is characterized by comprising the following steps:
step 1, acquiring an image through binocular camera video monitoring, and performing morphological processing on an initial image by using a graying and Gaussian filtering method;
step 2, data are transmitted through a 5G communication module, a receiving end applies a convolutional neural network algorithm based on initiation v4 and retrained again to identify the category of the high-altitude falling object, and the high-altitude falling object is divided into two modes of no threat and threat;
step 3, the system keeps a standby state in a no-threat mode, the condition of false triggering caused by environmental factors or tiny disturbance is reduced, and the loss of energy and the system is reduced;
when the robot enters a threatening mode, the falling object is identified, meanwhile, a mathematical model established by analyzing the possible stress mode is used for carrying out combined prediction on data received by the double cameras by using a Kalman filtering method, and the falling point coordinate of the object is predicted and transmitted;
and 4, rapidly processing data by the acousto-optic early warning module receiving the coordinates, marking a drop point by using a striking laser lamp before the falling object falls to the ground, assisting with sound alarm to remind the pedestrian, intercepting by using a blocking net, and avoiding the pedestrian from being injured.
2. The high altitude parabolic early warning method based on visual sensing and trajectory prediction as claimed in claim 1, wherein the method for processing the image in step 1 comprises:
converting the RGB color three-channel image into a single-channel grey-scale image, and performing Gaussian blur, wherein the Gaussian blur effect is achieved by introducing template operation, and the template formula is as follows:
m (I, J) is the value of the matrix element in the corresponding template, and the value of the pixel at the center point of the image is replaced by the pixel weighting operation of the coverage area of the template determined by the movement of the constructed template; processing the original data stream by using a creatbackgroundsustractorknn algorithm API of OpenCV to obtain a differential image after background subtraction, and performing further processing on the differential image, wherein the digital image processing method comprises graying and threshold processing; finally, analyzing a connected domain according to the optimized image to obtain the maximum outline of the image; video data for detecting a parabolic event is obtained.
3. The high-altitude parabolic early warning method based on visual sensing and trajectory prediction as claimed in claim 1, wherein the convolutional neural network algorithm retrained based on initiation v4 in the step 2 identifies the high-altitude falling object class by:
based on retraining based on the concept v4, according to the CNN theory, using transfer learning in image recognition, regarding each filter as a small feature grabber, constructing the uppermost softmax layer of the concept v4 according to the weight of the trained model of the concept v4, and retraining by using a training set adaptive to the use environment.
4. The high altitude parabolic early warning method based on visual sensing and trajectory prediction as claimed in claim 1, wherein when the threatening mode is entered in step 3, the mathematical model established while identifying the falling object comprises:
the method comprises the following steps that information is collected, a top view and a side view of a parabolic track are selected and obtained, and the top view and the side view are respectively provided by cameras in two directions; the selected special track point information is the three-dimensional coordinate and time (x) of a parabolic motion casting point0,y0,z0,t0) Highest point three dimensional coordinate and time (x)1,y1,z1,t1) Three-dimensional coordinates and time (x) of a point in the descending process2,y2,z2,t2) (ii) a In order to simplify the operation, the model is simplified on the basis of decomposing the motion track vector into three planes x, y and z for respectively solving; the x, y and z planes are determined to be the plane opposite to the building, namely the y plane, the ground is the z plane, and any side surface is the x plane;
side view trajectory analysis:
for high altitude parabolas, the side view is understood to be a trajectory similar to a parabola, for which the vertical coordinate analysis is performed, and if the orientation is positive, the object is subjected to the force analysis in the ascent phase:
maz＝-mg-vz 2λ
wherein f ═ vz 2λ is air resistance, let mg ═ c2λ, then there is
Arranging and integrating to obtain
Is obtained by calculation and arrangement
Finally, the integral of the velocity in the vertical direction is used as the falling height to form the formula:
obtaining a landing moment t' of the parabolic motion;
and then carrying out y coordinate analysis on the parabolic motion direction, and carrying out stress analysis on the parabolic motion direction if the parabolic motion direction is positive:
ma＝-vy 2λ
wherein "-" represents the force direction, and f ═ vy 2λ is the magnitude of the air resistance received, and is obtained by performing deformation integration
To obtain
The data brought into the acquisition track points can be obtained as vy1And vy2About vyoAnd because the speed change process exists
ma＝-vy 2λ
V can be obtained by three formulasyRegarding the expression of t, the landing time t' is already obtained in the vertical direction analysis, so the y coordinate of the final landing point is
Wherein v isxIs the speed of movement, v, of the object in the direction of the x-axisyIs the speed of movement of the object in the y-axis direction，vzThe moving speed of an object in the z-axis direction is shown, wherein (X, Y, z) are coordinates of the position of the object, c is a constant, lambda is an air resistance coefficient, m is the mass of the object, g is the gravity acceleration, f is the air resistance, and (X, Y) are horizontal and vertical coordinates of a drop point;
analyzing a top view track:
in the process of throwing the object, the track of the object in the horizontal direction cannot be changed by the force of gravity, the air resistance is always opposite to the moving direction, so that the track of the object cannot be deviated in the horizontal dimension, and the overlooking track is a line segment according to a starting point (x)0,y0,z0,t0) And highest point (x)1,y1,z1,t1) Can yield a linear relationship between the parabolic coordinates x, y:
and substituting the Y coordinate Y of the drop point obtained from the side view to obtain the X coordinate X of the drop point, so as to obtain the three-dimensional coordinate and time (X, Y, Z, t) of the drop point, achieving the purpose of predicting the drop point, and transmitting the object after predicting the drop point coordinate.
5. The high altitude parabolic early warning method based on visual sensing and trajectory prediction as claimed in claim 1, wherein the combined prediction is performed in step 3 by using a kalman filtering method, and the method comprises:
the preparation process comprises the following steps:
calculating the error between the true value of the position at time i and the estimated value of the position
modeling implementation:
first, a time update equation is established:
where u (i) is an input amount, i.e., input position information,
is the predicted value at time i; a is a state transition matrix and B is a gain matrix; w (i) is a set noise;
establishing an error correlation matrix P for judging whether the estimated value is accurate:
q is a covariance matrix with respect to system noise, and is a constantly changing value; p (i) is the covariance matrix of the estimation error,
is a covariance matrix of the prediction error;
the above is the prediction process; after the predicted value at the corresponding moment is obtained through calculation, correcting the predicted value; the process is to make a comparison between the measured value and the actual value and further update the data;
wherein
Is a measurement margin, and Z (i) is a measurement value;
calculating the necessary Kalman gain:
kalman filtering is itself a minimum mean square error, the mean square error being the trace of P (i), so solving the trace for the prediction error covariance equation yields a trace for KkEquation of (c), optimally estimated KkThe value minimizes the trace of P (i) and the equation is left and right simultaneously for KkDerivation, and using the algorithm of matrix differentiation to make the differentiation result be zero to obtain KkThe calculation expression of (1):
Kk＝P′(i)HT(HP′(i)HT+R)-1
the kalman gain obtained from the above can be used to find and update the estimate. For the estimated quantity of the moment i, the estimated quantity can be processed in such a way that a Kalman coefficient is utilized to perform an operation so as to obtain a Kalman predicted value
Thus, an update of the kalman estimation value using the estimation value and the measurement margin is completed;
and updating the covariance matrix, and deriving a covariance expressed by Kalman gain and prediction error according to the derivation:
P(i)＝(I-KkH)P′(i)
according to continuous iteration and recursion, finally calculating until P (n) and X (n) to complete the estimation of each position on the track;
the pose information of the target object is updated at any time according to the condition of the target object through the model, the corrected track of the target object is obtained through effective loop detection, and the relevant data information is obtained according to simulation of a Matlab tool, so that the target object drop point area is obtained.
6. The high altitude parabolic early warning method based on visual sensing and trajectory prediction as claimed in claim 1, wherein the method of step 4 is:
when the predicted object falling point coordinates are received, a data instruction is transmitted to the single chip microcomputer system through the signal data transmitting and transmitting module, the single chip microcomputer box controls the holder, the strong light lamp on the holder projects red strong light to a falling point area, and the buzzer gives an alarm; before falling objects fall to the ground, striking laser lamps are used for marking the falling points and assisting with sound alarm to remind pedestrians, and blocking nets are used for intercepting the falling objects, so that the pedestrians are prevented from being injured, and the personal and property safety of the pedestrians is protected to the maximum extent; and after the early warning is finished, the execution program enables the cradle head to automatically initialize and return to the original position and continue to be in a standby state.
7. A high altitude parabolic early warning system based on visual sensing and track prediction is characterized by comprising:
the camera module is provided with a binocular camera and is used for acquiring a video frame for monitoring the parabolic area and preprocessing the image of the video frame to obtain video data for detecting the parabolic event; the pretreatment comprises the following steps: performing morphological processing on the initial image by using a graying and Gaussian filtering method;
the communication module adopts a 5G communication module and is used for sending the image acquired by the camera module to a moving object target tracking detection module of a receiving end; and transmitting the drop point prediction result and the instruction to a networking alarm module and an early warning light source projection control module;
the moving object target tracking detection module is used for applying a convolutional neural network algorithm based on initiation v4 and retrained again to identify the category of the high-altitude falling object and divide the high-altitude falling object into two modes of no threat and threat; the system keeps a standby state in a no-threat mode, thereby reducing the condition of false triggering caused by environmental factors or tiny disturbance and reducing the loss of energy and the system; when the robot enters a threatening mode, the falling object is identified, meanwhile, a mathematical model established by analyzing the possible stress mode is used for carrying out combined prediction on data received by the double cameras by using a Kalman filtering method, and the falling point coordinate of the object is predicted and transmitted;
the networking alarm module is used for generating corresponding sound alarm when receiving the signal of falling of the parabola;
the early warning light source projection control module is used for identifying and reminding a falling object falling point, dividing the warning range of the system into regions with high enough precision, setting a holder control parameter corresponding to each region in the system in advance, feeding back predicted falling point coordinates after processing of an image is completed by an algorithm, obtaining required parameters in a table look-up mode, irradiating by using a laser lamp, and completing early warning of pedestrians.
8. A high altitude parabolic warning apparatus based on visual sensing and trajectory prediction, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the steps of the method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111066314.8A CN114299106A (en) | 2021-09-13 | 2021-09-13 | High-altitude parabolic early warning system and method based on visual sensing and track prediction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111066314.8A CN114299106A (en) | 2021-09-13 | 2021-09-13 | High-altitude parabolic early warning system and method based on visual sensing and track prediction |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114299106A true CN114299106A (en) | 2022-04-08 |
Family
ID=80964325
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111066314.8A Pending CN114299106A (en) | 2021-09-13 | 2021-09-13 | High-altitude parabolic early warning system and method based on visual sensing and track prediction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114299106A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115082864A (en) * | 2022-07-25 | 2022-09-20 | 青岛亨通建设有限公司 | Building construction safety monitoring system |
| CN116597340A (en) * | 2023-04-12 | 2023-08-15 | 深圳市明源云科技有限公司 | High altitude parabolic position prediction method, electronic device and readable storage medium |
| CN116911004A (en) * | 2023-07-06 | 2023-10-20 | 山东建筑大学 | Trajectory drop point correction method based on neural network |
-
2021
- 2021-09-13 CN CN202111066314.8A patent/CN114299106A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115082864A (en) * | 2022-07-25 | 2022-09-20 | 青岛亨通建设有限公司 | Building construction safety monitoring system |
| CN116597340A (en) * | 2023-04-12 | 2023-08-15 | 深圳市明源云科技有限公司 | High altitude parabolic position prediction method, electronic device and readable storage medium |
| CN116597340B (en) * | 2023-04-12 | 2023-10-10 | 深圳市明源云科技有限公司 | High altitude parabolic position prediction method, electronic device and readable storage medium |
| CN116911004A (en) * | 2023-07-06 | 2023-10-20 | 山东建筑大学 | Trajectory drop point correction method based on neural network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114299106A (en) | High-altitude parabolic early warning system and method based on visual sensing and track prediction | |
| CN109522793B (en) | Method for detecting and identifying abnormal behaviors of multiple persons based on machine vision | |
| WO2020253308A1 (en) | Human-machine interaction behavior security monitoring and forewarning method for underground belt transportation-related personnel | |
| US10007850B2 (en) | System and method for event monitoring and detection | |
| CN109785363A (en) | A kind of unmanned plane video motion Small object real-time detection and tracking | |
| CN107609522A (en) | A kind of information fusion vehicle detecting system based on laser radar and machine vision | |
| CN107352032A (en) | A kind of monitoring method and unmanned plane of flow of the people data | |
| CN107818326A (en) | A kind of ship detection method and system based on scene multidimensional characteristic | |
| CN111461023A (en) | Method for quadruped robot to automatically follow pilot based on three-dimensional laser radar | |
| CN110765964A (en) | Method for detecting abnormal behaviors in elevator car based on computer vision | |
| CN106462978A (en) | Object detection device, object detection method, and object detection system | |
| CN102447835A (en) | Non-blind-area multi-target cooperative tracking method and system | |
| CN104318578A (en) | Video image analyzing method and system | |
| CN100565557C (en) | System for tracking infrared human body target based on corpuscle dynamic sampling model | |
| CN109800665A (en) | A kind of Human bodys' response method, system and storage medium | |
| CN206968975U (en) | A kind of unmanned plane | |
| CN111079722B (en) | Hoisting process personnel safety monitoring method and system | |
| CN116647651B (en) | Unmanned aerial vehicle construction monitoring method and system based on Beidou satellite | |
| CN109830078A (en) | Intelligent behavior analysis method and intelligent behavior analytical equipment suitable for small space | |
| CN114280611A (en) | Road side sensing method integrating millimeter wave radar and camera | |
| CN110197121A (en) | Moving target detecting method, moving object detection module and monitoring system based on DirectShow | |
| CN112541938A (en) | Pedestrian speed measuring method, system, medium and computing device | |
| CN112785564B (en) | Pedestrian detection tracking system and method based on mechanical arm | |
| CN109887004A (en) | A kind of unmanned boat sea area method for tracking target based on TLD algorithm | |
| CN116866520B (en) | AI-based monorail crane safe operation real-time monitoring management system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |