CN115717867A - Bridge deformation measurement method based on airborne double cameras and target tracking - Google Patents
Bridge deformation measurement method based on airborne double cameras and target tracking Download PDFInfo
- Publication number
- CN115717867A CN115717867A CN202211373051.XA CN202211373051A CN115717867A CN 115717867 A CN115717867 A CN 115717867A CN 202211373051 A CN202211373051 A CN 202211373051A CN 115717867 A CN115717867 A CN 115717867A
- Authority
- CN
- China
- Prior art keywords
- target
- bridge
- camera
- measurement
- displacement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a bridge deformation measurement method based on airborne double cameras and target tracking, which divides bridge deformation measurement into three parts, namely pre-pasting of a sign board, acquisition of bridge video by an unmanned aerial vehicle and data analysis. The method comprises the steps of firstly arranging a plurality of measuring targets at a to-be-measured measuring point on the side face of a bridge, then adopting an unmanned aerial vehicle to carry a double-camera system comprising a long-focus camera and a wide-angle camera to collect a bridge video, finally adopting a calculation bridge surface target displacement based on deep learning multi-target tracking, respectively measuring a bridge side face moving target and a bridge pier fixed target through the double cameras, eliminating the shaking of the unmanned aerial vehicle, and obtaining the absolute displacement of the bridge target. The method has the advantages of rapidness, convenience, low cost and non-contact, overcomes the problem that the traditional method for arranging the sensors is difficult to be applied to bridge deformation measurement, and has good prospect of being widely applied to actual bridge deformation monitoring.
Description
Technical Field
The invention belongs to the technical field of structural safety monitoring, and particularly relates to a bridge deformation measuring method and system based on airborne double cameras and target tracking.
Background
The deformation measurement of the bridge under the action of vehicle load and environment (wind and temperature) is an important content for bridge safety evaluation, and particularly in a load test of the bridge, the accurately measured bridge deformation response can be compared with the bridge deformation condition of finite element analysis, and deep characteristic parameters (such as structural frequency response function and modal flexibility) of the bridge can be calculated according to a structural dynamics theory, so that a foundation is provided for damage identification of the bridge. The displacement measurement of the bridge is divided into long-term monitoring and short-term detection, the long-term monitoring generally needs to install a fixed measuring sensor on the bridge, measure the deformation of a small amount of key parts, and send out early warning in advance when the bridge is subjected to extreme load or accidents. Short-term deformation response detection is more commonly used in load tests before bridge service and regular safety evaluation in the service process, and most of the existing bridge deformation measurement methods aim at short-term dynamic deformation measurement of bridges.
The existing bridge deformation measurement method is mostly based on traditional measurement and surveying instruments including a static GPS, a total station instrument, a liquid level communicating pipe and the like, and the method is generally low in measurement frequency which is generally not more than a few hertz, so that the method is only suitable for static measurement. With the development of instrument science in recent years, some leading edge sensors are gradually applied to bridge detection. The high-sensitivity acceleration sensor is the most commonly used sensor in the bridge vibration test at present, and some scholars utilize acceleration data quadratic integration to reversely calculate bridge deformation, but the method also points out the limitation of using under the conditions of short duration interval and small amplitude displacement. The LVDT sensor is a commonly used displacement sensor, some researches use it to measure the deformation of a local position of a bridge, but the LVDT needs to be installed on a fixed base point near a measuring point, so that the absolute displacement of a bridge body, such as a bridge spanning a medium position, cannot be measured, and the measuring range is limited.
With the rapid development of computer vision technology, vision-based measurement methods have begun to be applied to structural deformation detection. Such methods are inexpensive, simple to construct, and have been demonstrated in the measurement of some civil engineering structures to achieve the required measurement accuracy. Researches prove that the visual measurement method has obvious application prospect in bridge deformation measurement, but two inherent defects of the method are also exposed. (1) The vision measurement method is generally based on the matching between images to calculate the motion condition of a point to be measured in the images at the front moment and the back moment, but the imaging quality of the images is easily interfered by illumination, so that the traditional matching method based on image processing is very sensitive to the interference of illumination, shielding and the like. (2) When a vision measurement method is adopted to detect a long and large span bridge, a proper camera layout position is difficult to find, on one hand, the bridge is generally positioned on a river, most cameras can only tilt to shoot the bridge when the cameras are arranged, on the other hand, when the bridge span needs to be measured, the measurement distance can exceed 600m for the long and large span bridge, the atmospheric interference and the micro-vibration of the cameras have great influence on the result at the distance, and therefore the problem needs to be solved urgently by a response method.
In order to solve the problem that the conventional optical measurement method is easily affected by interference such as illumination and shielding, some scholars think that the problem can be solved by a deep learning method which is widely researched in recent years. The target detection algorithm based on deep learning generally has good anti-interference capability, and in the disease detection, even if an image is illuminated, shielded and eroded by stains, the disease in the image can be accurately identified. Therefore, some scholars try to integrate the deep learning technology into the visual measurement method, and the methods effectively utilize various methods in the deep learning to improve the stability of the optical measurement, but most of the methods are not applied to the bridge deformation measurement scene, and the deep learning method has obvious sample selectivity, so that further research needs to be carried out on the methods in the bridge deformation measurement scene.
Aiming at the problem of long measurement distance which is difficult to overcome in large-span bridge measurement by a vision measurement method, the unmanned aerial vehicle technology which is applied more and more widely in recent years is expected to become a breakthrough point for solving. Unmanned aerial vehicle carries the camera and has played important role in the structure detects, carries the camera by unmanned aerial vehicle and replaces standing camera to measure bridge deformation, will greatly reduce measuring distance to can select displacement measurement position wantonly. However, the unmanned aerial vehicle inevitably shakes in flight, which causes the problem that the measurement base point is unstable, thereby limiting the application of the measurement method based on the unmanned aerial vehicle. Most of the existing researches adopt a structural background motionless point shot by an unmanned aerial vehicle as a reference to calculate the displacement of the unmanned aerial vehicle. But for large span bridges, it is generally difficult to find a fixed point in the picture as a reference because they are generally located on the water surface.
Disclosure of Invention
The technical problem to be solved is as follows: aiming at the defects in the prior art, the invention provides the bridge deformation measuring method and the bridge deformation measuring system based on the airborne double cameras and the target tracking, the bridge deformation can be quickly and conveniently measured, and the measuring result precision is high.
The technical scheme is as follows:
a bridge deformation measurement method based on airborne double cameras and target tracking comprises the following steps:
s1, arranging a plurality of measuring targets at a measuring point to be measured on the side face of a bridge, and carrying a double-camera system comprising a long-focus camera and a wide-angle camera by using an unmanned aerial vehicle to acquire a bridge video; the long-focus camera and the wide-angle camera are coaxial and fixed in position and continuously shoot the bridge; the view field of the wide-angle camera comprises a left bridge pier and a right bridge pier of the bridge; the long-focus camera is only used for collecting image videos of the measuring target at the measuring point to be measured on the side face of the bridge;
s2, constructing a target recognition model based on a YOLO v5S network, respectively importing video frame images acquired by a telephoto camera and a wide-angle camera into the target recognition model, respectively recognizing the target positions from bridge video frame images acquired by the telephoto camera and the wide-angle camera by adopting the target recognition model, and segmenting the target images from video frames;
s3, performing inter-class distinction on the segmented measurement target images at different positions by adopting a feature extraction network, distributing a fixed ID (identity) to each measurement target, acquiring the mark frame center coordinates of the measurement target of each frame, and drawing to obtain a rough displacement track of the measurement target;
s4, obtaining the central coordinates of the marking frame of the measuring target of the current frame based on the DeepsORT multi-target tracking algorithm, predicting by using Kalman filtering to obtain the predicted track of the current coordinate, calculating the association degree of the predicted track and the actual coordinates of the marking frame center of the next frame, and correcting the rough displacement track of the measuring target according to the association degree to obtain the actual displacement track of the central point of the measuring target;
s5, recognizing the central point of the measurement target image by using the sub-pixels to obtain the sub-pixel point level displacement track of the central point of the measurement target corresponding to the two camera collected images;
and S6, removing the displacement error caused by the shaking of the unmanned aerial vehicle from the result of the motion target measurement by combining the sub-pixel point level displacement tracks corresponding to the two cameras and the position relation of the two cameras.
Further, in step S1, before the dual-camera system is used for measurement, the telephoto camera and the wide-angle camera are independently calibrated by using the zhang' S calibration method to obtain the internal parameter K of the wide-angle camera w And internal reference K of telephoto camera t 。
Furthermore, the relative displacement measurement of the pier position does not use a measurement target, and the measurement is carried out by utilizing the matching relation of the texture of the pier.
Further, in step S2, the process of constructing the target recognition model based on the YOLO v5S network includes the following steps:
sticking a target on the surface of the bridge, measuring the position of the target, and taking the center of the target as a displacement measuring point;
collecting a plurality of target images under different illumination conditions, marking point positions manually, manufacturing a target data set by using a PASCAL VOC (volatile organic compound) format, and clustering the target images in the data set by adopting a k-means clustering method;
and importing the target data set into a YOLO v5s network for training and verification to obtain a target recognition model after training.
Further, in step S4, the process of correcting the rough displacement trajectory of the measurement target according to the association degree to obtain the actual displacement trajectory of the center point of the measurement target includes:
calculating the correlation degree of the predicted track and the actual coordinate of the mark frame center of the next frame, and if the correlation degree reaches a predicted correlation degree threshold value, judging that the detection result of the next frame is correct; otherwise, IOU matching is carried out on the predicted track and the re-detected measurement target so as to continuously trace the track.
Further, the graph of the measurement target includes four rectangular markers located at four corners and a circular marker located in the middle.
Further, in step S5, the process of identifying the central point of the measurement target image by using the sub-pixels to obtain the sub-pixel point level displacement trajectory of the measurement target central point corresponding to the two camera-collected images includes the following steps:
taking coordinates of four rectangular marks at four corners of the target as known points, calculating a proportion parameter of a measurement target image, and performing inclination correction on the measurement target image;
screening coordinates P of four rectangular marks by adopting a graph detection method 1 (x 1 ，y 1 )、P 2 (x 2 ，y 2 )、P 3 (x 3 ，y 3 )、P 4 (x 4 ，y 4 ) Fitting to obtain a measurement target prototype, and detecting the circle center coordinate P of the fitted prototype by Hough transform 5 (x 5 ，y 5 ) The average value of these five coordinates is taken as the center coordinate P of the measurement target image c (x c ，y c )。
Further, if some of the rectangular markers and the circular markers of the measurement target image are missing, the average of the coordinates of the remaining markers is taken as the center coordinates of the measurement target image.
Further, in step S6, the pure bridge displacement for eliminating the displacement of the unmanned aerial vehicle platform base point is:
ΔP b ＝k ti ΔY t -R t，w k wi ΔY w
wherein R is t，w And T t，w Is a roto-translational relationship of the two cameras, where k i Is T 1 Proportional coefficient of time target, Δ Y w And Δ Y t Is the vertical displacement of the target measured by the wide angle camera and the tele camera.
The invention also provides a bridge deformation measuring system based on the airborne double cameras and the deep learning tracking of the unmanned aerial vehicle, wherein the bridge deformation measuring system comprises a plurality of measuring targets, the unmanned aerial vehicle carrying the double camera system comprising a long-focus camera and a wide-angle camera and a processor;
the plurality of measuring targets are distributed and arranged at the positions of the side faces of the bridge to be measured; the unmanned aerial vehicle hovers at the side face of the bridge according to a control instruction of the processor, a long-focus camera and a wide-angle camera are adopted to collect bridge videos, and the collected bridge videos are sent to the processor;
the processor calculates and obtains the vertical displacement of the measuring target at the measuring point to be measured on the side surface of the bridge by adopting the bridge deformation measuring method
Has the beneficial effects that:
according to the bridge deformation measuring method based on the airborne dual-camera and the target tracking, on one hand, a large-view-angle video including different points such as bridge piers and the like and a refined video aiming at a bridge side displacement target are shot simultaneously through the airborne coaxial dual-camera of the unmanned aerial vehicle, and the displacement of a base point caused by shaking of an unmanned aerial vehicle platform is removed from the bridge displacement, and on the other hand, the problem that a traditional displacement calculation method is easy to generate measuring errors under illumination change and accidental shielding is solved through a method based on deep learning target detection and multi-target tracking. Compared with the prior art, the invention has the following advantages:
(1) Theoretical derivation and experimental verification prove that the mode that the unmanned aerial vehicle carries the coaxial dual-camera and shoots the immobile point of the pier and the bridge body target simultaneously can effectively eliminate the vibration of the unmanned aerial vehicle when the unmanned aerial vehicle is used for measuring the bridge displacement, the method simultaneously considers the requirements of shooting the bridge in a neat large view field and shooting the bridge in a small view field at the position of the local target of the bridge, and compared with the existing method that the unmanned aerial vehicle carries the single camera and shoots the whole bridge, the displacement measurement precision can be improved to the maximum.
(2) The displacement measurement method based on deep learning multi-target tracking integrates the latest target detection algorithm, the latest multi-target tracking algorithm and the latest sub-pixel detection algorithm, and on the established bridge displacement target data set, the testing precision of target detection is 0.998, and the testing precision of target feature classification is 0.823. The displacement calculation method combining deep learning and sub-pixel detection solves the problem that the traditional DIC and optical flow displacement calculation methods need to be adjusted manually according to the measured object, and also solves the problem that the traditional methods are easily affected by complex conditions such as change of illumination conditions and shielding of measured objects. In the indoor displacement table test, the proposed method can effectively avoid the appearance of outliers in the displacement curve while maintaining the similar precision as the DIC method.
Drawings
Fig. 1 is a schematic diagram of a bridge deformation measurement method based on airborne dual cameras and target tracking according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a camera pose settlement principle and a dual-camera measurement principle, wherein (a) is the camera pose settlement principle and (b) is the dual-camera measurement principle;
FIG. 3 is a flowchart of a displacement calculation method based on deep learning multi-target tracking;
FIG. 4 is a flow chart of trajectory processing based on DeepSORT multi-target tracking algorithm;
FIG. 5 is a schematic diagram of a target recognition model training result based on a YOLO v5s network;
FIG. 6 is a diagram illustrating training results of the inter-class recognition network based on the DeepsORT multi-target tracking algorithm.
Detailed Description
The following examples will give the skilled person a more complete understanding of the present invention, but do not limit the invention in any way.
Referring to fig. 1, the present embodiment discloses a bridge deformation measurement method based on airborne dual cameras and target tracking, which includes the following steps:
s1, arranging a plurality of measuring targets at a measuring point to be measured on the side face of a bridge, and carrying a double-camera system comprising a long-focus camera and a wide-angle camera by using an unmanned aerial vehicle to acquire a bridge video; the long-focus camera and the wide-angle camera are coaxial and fixed in position and continuously shoot the bridge; the view field of the wide-angle camera comprises a left bridge pier and a right bridge pier of the bridge; the long-focus camera is only used for acquiring image videos of the measuring target at the position to be measured on the side face of the bridge.
S2, constructing a target recognition model based on a YOLO v5S network, respectively importing the video frame images collected by the telephoto camera and the wide-angle camera into the target recognition model, respectively recognizing the target positions from the bridge video frame images collected by the telephoto camera and the wide-angle camera by adopting the target recognition model, and segmenting the target images from the video frames.
And S3, performing class-to-class distinction on the segmented measurement target images at different positions by adopting a feature extraction network, distributing a fixed ID (identity) to each measurement target, acquiring the central coordinates of the marking frame of the measurement target of each frame, and drawing to obtain the rough displacement track of the measurement target.
And S4, obtaining the central coordinates of the marking frame of the measuring target of the current frame by using a DeepSORT-based multi-target tracking algorithm, predicting by using Kalman filtering to obtain the predicted track of the current coordinate, calculating the association degree of the predicted track and the actual coordinates of the center of the marking frame of the next frame, and correcting the rough displacement track of the measuring target according to the association degree to obtain the actual displacement track of the central point of the measuring target.
And S5, recognizing the central point of the measurement target image by using the sub-pixels to obtain the sub-pixel point level displacement track of the central point of the measurement target corresponding to the two camera collected images.
And S6, removing the displacement error caused by the shaking of the unmanned aerial vehicle from the result of the measurement of the moving target by combining the level displacement tracks of the sub-pixel points corresponding to the two cameras and the position relation of the two cameras. Therefore, based on the absolute displacement value of each time point of the removed error, a dynamic displacement track curve of the bridge can be drawn.
The method comprises the steps of arranging the targets, collecting bridge videos and analyzing data, wherein the targets are arranged at measuring points to be measured, the unmanned aerial vehicle hovers on the side face of the bridge to shoot video information of the bridge under load, and the algorithm in the data analysis can realize the real-time calculation of the displacement of the preset targets on the bridge at a high frame rate, can be used for displacement detection of the bridge in service, and can effectively evaluate the safety performance of the bridge structure.
Measurement target and unmanned aerial vehicle set up
The target is used for being arranged at a measuring point position on the side face of a bridge and is made of a light aluminum plate, and the graph of the target consists of four rectangles at four corners and a circle in the middle. And then, the center point formed by the rectangular coordinates of the four corners is superposed with the center point coordinate of the middle circle, so that the center point coordinate can be calculated by a small number of corner point coordinates under the condition that the target part is shielded. Assuming the coordinate of the central point of the target as P c (x c ，y c ) The corner point of the detected four sides is P 1 (x 1 ，y 1 )，P 2 (x 2 ，y 2 )，P 3 (x 3 ，y 3 )，P 4 (x 4 ，y 4 ) The coordinate of the central dot is P 5 (x 5 ，y 5 ) Then P is c The calculation of (c) is to average these five coordinates.
A unmanned aerial vehicle for shooing the side deformation video of bridge can adopt general type unmanned aerial vehicle, and the camera system that unmanned aerial vehicle carried is a parallel arrangement double camera system who contains long focus camera and wide angle camera. The long-focus camera is used for shooting targets arranged on the side face of the bridge, and the wide-angle camera is used for shooting the whole bridge. Before the double cameras are used for measurement, the double cameras need to be independently calibrated by adopting a Zhang calibration method to obtain the inside of the camerasThe ginseng is respectively: wide-angle camera internal reference K w Zoom camera internal reference K t . The internal parameters are used for distortion removal of source data during data analysis, homography calculation is carried out on the two cameras, the correlation between the calculated displacement results of the two cameras is obtained through analysis, and the shaking of the unmanned aerial vehicle is eliminated. Specifically, aiming at the basic point vibration rejection of the unmanned aerial vehicle, a double-camera system with a telephoto lens and a wide-angle lens is adopted, a shooting mode of shooting a deformation detection point of a bridge body and a fixed reference point of a bridge pier is adopted, and the relation of displacement measured by double cameras under the condition is derived theoretically, so that the displacement of the unmanned aerial vehicle is rejected. In actual operation, the unmanned aerial vehicle carrying the coaxial double cameras can hover at one side of the bridge, and the cameras continuously shoot videos right opposite to the bridge. Unmanned aerial vehicle's camera sets up to free mode, and the horizontal rotation of camera, vertical rotation angle are unchangeable promptly, and do not receive unmanned aerial vehicle gesture change and take place to rotate, keep all the time just to the gesture of bridge, consequently can not consider the relative displacement change that the rotation of camera brought. The wide-angle lens of the camera is set to just shoot a visual field containing left and right piers.
Absolute displacement calculation of bridge
Referring to FIG. 2, assume the world coordinate system (x) w ，y w ，z w ) Unstable coordinate system of unmanned aerial vehicle platform (x) u ，y u ，z u ) The coordinate system of the two cameras is (x) k ，y k ，z k ) Where k =1,2. The coordinates of the camera are changed to (α, β, γ, Δ x, Δ y, Δ z), where (α, β, γ) is the rotation angle of the camera around the three directions and (Δ x, Δ y, Δ z) is the translation component of the camera in the three directions. Let P i Is a fixed target control point, p, within the field of view of the camera W i Is the projection point of the control point in the W camera image, where i =1,2,3 i Relative movement occurs, the projection point p i Will contain the rotation of the camera about the axis of rotation of that direction plus the translational component of that direction, which can be expressed as (for the y-axis for example):
P′ y ＝-P x sinγ w +P y cosα w cosγ w +P z sinα w cosγ w +ΔY w
wherein alpha is w ，β w And gamma w Is the rotation angle of the camera and the fixed target in the three-axis direction, P x ，P y And P z Is P i Coordinates before the movement takes place, Δ Y w Is the translation component in the y-axis direction. The relative vertical displacement of the camera W from the fixed target is:
H w ＝P′ y -P y ＝P y (cosα w cosγ w -1)+P z sinα w cosγ w +ΔY w
the point displacement in the image coordinate system is then:
Δp iy ＝v′ i -v i ＝k wi H w
wherein the proportionality coefficient k wi The target size and the camera parameters can be calculated through a homography relation, and when the camera is right opposite to the target during shooting, the target size and the camera parameters can also be simply calculated through object distance and focal distance. Since the movement in the object distance direction is small, k can be considered to be wi Does not change and is constant. Due to Δ p iy Can be directly calculated from two frame images, alpha w ，β w And gamma w Also very little under the stable compensation of triaxial cloud platform, consequently can simplify the above equation into:
Δp iy ＝k wi ΔY w
thus, the vertical displacement of the camera platform in the shooting process can be obtained.
The camera W and the camera T are fixedly connected, and the relationship between the two cameras is as follows:
wherein R is t，w And T t，w The relationship between the two cameras is the rotational-translational relationship, so that the vertical displacement of the camera W can be transmitted to the camera T after the rotational-translational relationship between the two cameras is obtained by calibrating the two cameras.Similar to the camera W, the vertical displacement of the three-dimensional point captured by the camera T is:
then the pure displacement of the bridge for eliminating the displacement of the base point of the unmanned aerial vehicle platform is as follows:
ΔP b ＝k ti ΔY t -R t，w k wi ΔY w 。
(III) deep learning multi-target tracking displacement
Referring to fig. 3, for an image detected using the object detection network, it is possible to quickly extract the size of the detected mark frame and calculate the center coordinates of the detected object. Connecting the frame-by-frame center coordinates can be considered a coarse object pixel displacement. Aiming at the problems that the ID of the shielded target detection is changed and the pixel-level displacement result can only be obtained by directly calculating the central point, a target tracking method and a sub-pixel fine extraction method are added on the basis of a target detection network. The method for analyzing target displacement from the video comprises three steps:
the first step of data analysis is to apply a deep learning target detection network to an original video to identify the target position in the video and segment the target from a video frame, the used network is a YOLO v5s network trained in a bridge target data set, and the target automatic identification based on the YOLO v5s network is used, so that the method has advantages in speed and platform consistency. The established pictures in the data set come from target videos actually shot by the unmanned aerial vehicle on the multi-type bridge, and the pictures contain a large number of target images with different illumination conditions, so that the universality and the practicability of the method are improved. Specifically, the target is adhered to the surface of the bridge, the position of the target is measured, and the center of the target is used as a displacement measuring point. Establishing 1336 Zhang Babiao images under different illumination conditions, marking point positions manually, making a data set by using a PASCAL VOC format, and finally clustering the data set by adopting a k-means clustering method. FIG. 5 is a schematic diagram of a target recognition model training result based on a YOLO v5s network.
The second step of the data analysis is to apply a feature extraction network to the segmented target image to discriminate between classes of targets at a plurality of different locations, so that each target is assigned a fixed ID. The core of the second step of data analysis is a DeepsORT-based multi-target tracking algorithm, which adopts a convolutional neural network to learn the identified targets in advance, so as to judge the difference of different targets in a deep learning classification mode. After the identification frame or the target center coordinates of the current frame are obtained, the prediction track of the current coordinates is obtained by Kalman filtering prediction, and the predicted track is input into the target detection of the next frame. If the correlation of the predicted track of the next frame is good, the detection result is considered to be correct, and the processing is continued according to the flow.
Specifically, referring to fig. 4, after the real-time identification of the target is realized, the center of the target marker frame of each frame is calculated, and the marker frame centers of each frame are connected, so that a rough displacement trajectory of the target is actually obtained. And (3) accurately processing the re-identification problem and the identity transformation problem under large displacement or shielding in the target movement by using a DeepSORT-based multi-target tracking algorithm. After the identification frame or the target center coordinates of the current frame are obtained, a prediction track of the current coordinates is obtained by Kalman filtering prediction, and the prediction track is input into target detection of the next frame. If the correlation of the predicted track of the next frame is good, the detection result is considered to be correct, and the processing is continued according to the flow. For the case that the target is not identified, IOU matching is carried out on the track and the target detected again later, and the track is continuously drawn. FIG. 6 is a diagram illustrating training results of the inter-class recognition network based on the DeepsORT multi-target tracking algorithm.
And the third step of data analysis is to further refine the center position of the pixel-level target obtained in the second step to a sub-pixel level, so as to improve the precision. And identifying the central point of the target image by using the sub-pixels so as to obtain a sub-pixel level displacement track of the central point of the target. And calculating the scale parameters of the image by taking the coordinates of the rectangular corner points of the four corners of the target as known points and performing inclination correction. And screening coordinate information of the rectangular corner points by adopting a graph detection method, detecting the coordinates of the circle center of the fitted prototype by adopting Hough transform, averaging the extracted five coordinates, and obtaining the final result, namely the target center coordinates at the sub-pixel level.
The data analysis method has the advantages of high measurement precision and no influence of ambient illumination and the displacement of the unmanned aerial vehicle, and can effectively improve the bridge displacement monitoring efficiency; in addition, the bridge displacement monitoring at a sub-pixel level can be realized based on a deep learning tracking method, and the method can be further used for performance evaluation of the bridge in a long-term working state.
The following describes specific implementation steps of the rapid displacement measurement method based on the drone platform and the deep learning target tracking algorithm according to this embodiment by using a typical bridge case.
Step 1: and (5) image acquisition. Hovering the unmanned aerial vehicle at a distance of about 20m from the bridge, and continuously shooting the video by the camera over the bridge. The camera is set to be in a free mode, and the camera keeps the posture of always facing the bridge. The wide-angle lens of the camera is set to be just shooting the visual field containing the left bridge pier and the right bridge pier, the zoom camera is set to zoom by 40 times, the focal length is 240mm, and the visual field is adjusted to be that the target is clearly visible. The fixed camera on ground aims at the target and pastes the position and shoot, and the collection frame rate of ground camera and unmanned aerial vehicle camera sets up to 30fps. The size of the targets pasted on the bridge deck is 10cm, and 3 targets are respectively pasted near the midspan. The relative displacement measurement of the pier position does not use a target, and the matching relation of the texture of the pier is utilized for measurement. 3 targets near the midspan position are shot respectively in the whole measuring process, and the shooting time is 3 minutes each time
Step 2: and calculating the absolute displacement of the bridge. By adopting the method provided by the embodiment, the video of the tele camera and the video of the wide camera are respectively processed, wherein the video of the wide camera takes the piers at two sides as measuring positions, the calculated result is the relative displacement of the unmanned aerial vehicle relative to the unmoving point of the pier, and the relative displacement is subtracted from the target displacement obtained by video analysis of the tele camera, so that the real displacement of the target relative to the unmoving point of the pier is obtained.
For the target 1, the correlation of the displacement curve of the proposed method and the displacement curve of the fixed camera using the DIC method is 0.885, and the correlation of the displacement curve of the proposed method and the displacement curve of the fixed camera using the proposed method is 0.917. For target 2, the correlations were 0.766 and 0.772, respectively, and for target 3, the correlations were 0.761 and 0.792, respectively. And for the displacement curves before and after rejecting the unmanned aerial vehicle displacement, the three targets are 0.374,0.135 and 0.248 respectively. The results of the correlation calculation also confirm the above conclusion.
The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above-mentioned embodiments, and all technical solutions belonging to the idea of the present invention belong to the protection scope of the present invention. It should be noted that modifications and embellishments within the scope of the invention may be made by those skilled in the art without departing from the principle of the invention.
Claims (10)
1. A bridge deformation measurement method based on airborne double cameras and target tracking is characterized by comprising the following steps:
s1, arranging a plurality of measuring targets at a measuring point to be measured on the side face of a bridge, and carrying a double-camera system comprising a long-focus camera and a wide-angle camera by using an unmanned aerial vehicle to acquire a bridge video; the long-focus camera and the wide-angle camera are coaxial and fixed in position and continuously shoot the bridge; the field of view of the wide-angle camera comprises a left bridge pier and a right bridge pier of the bridge; the long-focus camera is only used for acquiring an image video of a measuring target at a measuring point to be measured on the side face of the bridge;
s2, constructing a target recognition model based on a YOLO v5S network, respectively importing video frame images acquired by a telephoto camera and a wide-angle camera into the target recognition model, respectively recognizing the target positions from bridge video frame images acquired by the telephoto camera and the wide-angle camera by adopting the target recognition model, and segmenting the target images from video frames;
s3, performing inter-class distinction on the segmented measurement target images at different positions by adopting a feature extraction network, distributing a fixed ID (identity) to each measurement target, acquiring the mark frame center coordinates of the measurement target of each frame, and drawing to obtain a rough displacement track of the measurement target;
s4, obtaining the central coordinates of the marking frame of the measuring target of the current frame by using a DeepSORT-based multi-target tracking algorithm, predicting by using Kalman filtering to obtain the predicted track of the current coordinate, calculating the association degree of the predicted track and the actual coordinates of the center of the marking frame of the next frame, and correcting the rough displacement track of the measuring target according to the association degree to obtain the actual displacement track of the central point of the measuring target;
s5, recognizing the central point of the measurement target image by using the sub-pixels to obtain sub-pixel point level displacement tracks of the central point of the measurement target corresponding to the two camera collected images;
and S6, removing the displacement error caused by the shaking of the unmanned aerial vehicle from the result of the motion target measurement by combining the sub-pixel point level displacement tracks corresponding to the two cameras and the position relation of the two cameras.
2. The bridge deformation measurement method based on airborne double cameras and target tracking according to claim 1, wherein in step S1, before the double camera system is applied for measurement, the tensiometer calibration method is adopted to independently calibrate the tele camera and the wide camera to obtain the internal parameter K of the wide camera w And internal reference K of telephoto camera t 。
3. The bridge deformation measurement method based on airborne dual-camera and target tracking according to claim 1, characterized in that the relative displacement measurement of the pier position is performed by using the matching relationship of the texture of the pier itself without using a measurement target.
4. The bridge deformation measurement method based on airborne dual-camera and target tracking according to claim 1, wherein in step S2, the process of constructing the target identification model based on the YOLO v5S network comprises the following steps:
sticking a target on the surface of the bridge, measuring the position of the target, and taking the center of the target as a displacement measuring point;
collecting a plurality of target images under different illumination conditions, marking point positions manually, manufacturing a target data set by using a PASCAL VOC (volatile organic compound) format, and clustering the target images in the data set by adopting a k-means clustering method;
and importing the target data set into a YOLO v5s network for training and verification to obtain a target recognition model after training.
5. The bridge deformation measurement method based on airborne dual-camera and target tracking according to claim 1, wherein in step S4, the process of correcting the rough displacement trajectory of the measurement target according to the correlation degree to obtain the actual displacement trajectory of the center point of the measurement target comprises:
calculating the correlation degree of the predicted track and the actual coordinate of the center of the marking frame of the next frame, and if the correlation degree reaches a predicted correlation degree threshold value, judging that the detection result of the next frame is correct; otherwise, IOU matching is carried out on the predicted track and the re-detected measurement target so as to continuously trace the track.
6. The bridge deformation measurement method based on airborne dual-camera and target tracking according to claim 1, wherein the graph of the measurement target comprises four rectangular marks at four corners and a circular mark in the middle.
7. The bridge deformation measurement method based on airborne dual cameras and target tracking according to claim 6, wherein in step S5, the process of identifying the center point of the measurement target image by sub-pixels to obtain the sub-pixel point level displacement trajectory of the measurement target center point corresponding to the two camera collected images comprises the following steps:
taking coordinates of four rectangular marks at four corners of the target as known points, calculating a proportion parameter of a measurement target image, and performing inclination correction on the measurement target image;
screening coordinates P of four rectangular marks by adopting a graph detection method 1 (x 1 ，y 1 )、P 2 (x 2 ，y 2 )、P 3 (x 3 ，y 3 )、P 4 (x 4 ，y 4 ) Fitting to obtain a measurement target prototype, and detecting circle center coordinate P by Hough transform on the fitted prototype 5 (x 5 ，y 5 ) Taking the average value of these five coordinates as the center coordinate P of the measurement target image c (x c ，y c )。
8. The bridge deformation measurement method based on airborne double cameras and object tracking according to claim 6, wherein if some of the rectangular markers and the circular markers of the measurement target image are missing, the average value of the coordinates of the remaining markers is used as the center coordinates of the measurement target image.
9. The bridge deformation measurement method based on airborne double cameras and target tracking according to claim 1, wherein in step S6, the pure bridge displacement excluding the displacement of the platform base point of the unmanned aerial vehicle is:
ΔP b ＝k ti ΔY t -R t，w k wi ΔY w
wherein R is t，w And T t，w Is a roto-translational relationship of two cameras, where k i Is T 1 Proportional coefficient of time target, Δ Y w And Δ Y t Is the vertical displacement of the target measured by the wide angle camera and the tele camera.
10. A bridge deformation measurement system based on unmanned aerial vehicle airborne double cameras and deep learning tracking is characterized by comprising a plurality of measurement targets, an unmanned aerial vehicle carrying a double camera system comprising a long-focus camera and a wide-angle camera, and a processor;
the plurality of measuring targets are distributed and arranged at the positions of the side faces of the bridge to be measured; the unmanned aerial vehicle hovers at the side face of the bridge according to a control instruction of the processor, a long-focus camera and a wide-angle camera are adopted to collect bridge videos, and the collected bridge videos are sent to the processor;
the processor calculates the vertical displacement of the measuring target at the point to be measured on the side surface of the bridge by using the bridge deformation measuring method as claimed in any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211373051.XA CN115717867A (en) | 2022-11-03 | 2022-11-03 | Bridge deformation measurement method based on airborne double cameras and target tracking |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211373051.XA CN115717867A (en) | 2022-11-03 | 2022-11-03 | Bridge deformation measurement method based on airborne double cameras and target tracking |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115717867A true CN115717867A (en) | 2023-02-28 |
Family
ID=85254745
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211373051.XA Pending CN115717867A (en) | 2022-11-03 | 2022-11-03 | Bridge deformation measurement method based on airborne double cameras and target tracking |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115717867A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116309418A (en) * | 2023-03-09 | 2023-06-23 | 中建铁路投资建设集团有限公司 | Intelligent monitoring method and device for deformation of girder in bridge cantilever construction |
| CN117387491A (en) * | 2023-12-11 | 2024-01-12 | 南京理工大学 | Binocular vision marker positioning device and method suitable for bridge girder erection machine |
-
2022
- 2022-11-03 CN CN202211373051.XA patent/CN115717867A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116309418A (en) * | 2023-03-09 | 2023-06-23 | 中建铁路投资建设集团有限公司 | Intelligent monitoring method and device for deformation of girder in bridge cantilever construction |
| CN117387491A (en) * | 2023-12-11 | 2024-01-12 | 南京理工大学 | Binocular vision marker positioning device and method suitable for bridge girder erection machine |
| CN117387491B (en) * | 2023-12-11 | 2024-04-05 | 南京理工大学 | Binocular vision marker positioning device and method suitable for bridge girder erection machine |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110142785A (en) | A kind of crusing robot visual servo method based on target detection | |
| CN115717867A (en) | Bridge deformation measurement method based on airborne double cameras and target tracking | |
| WO2017080102A1 (en) | Flying device, flying control system and method | |
| CN110470226B (en) | Bridge structure displacement measurement method based on unmanned aerial vehicle system | |
| CN106529538A (en) | Method and device for positioning aircraft | |
| CN107589069B (en) | Non-contact type measuring method for object collision recovery coefficient | |
| CN111076880A (en) | Multi-point deflection measuring method of long-span bridge considering camera attitude change | |
| CN108305288B (en) | Fixed star centroid extraction method for stationary orbit earth observation satellite line instrument | |
| CN112629431B (en) | Civil structure deformation monitoring method and related equipment | |
| CN102788572B (en) | Method, device and system for measuring attitude of lifting hook of engineering machinery | |
| CN105823416A (en) | Method for measuring object through multiple cameras and device thereof | |
| CN103411621A (en) | Indoor-mobile-robot-oriented optical flow field vision/inertial navigation system (INS) combined navigation method | |
| CN109269525B (en) | Optical measurement system and method for take-off or landing process of space probe | |
| CN110889829A (en) | Monocular distance measurement method based on fisheye lens | |
| CN106403900A (en) | Flyer tracking and locating system and method | |
| CN115578315A (en) | Bridge strain close-range photogrammetry method based on unmanned aerial vehicle image | |
| CN111161305A (en) | Intelligent unmanned aerial vehicle identification tracking method and system | |
| CN117095315A (en) | High-precision spillway bottom plate concrete structure surface crack characteristic detection method | |
| CN111598956A (en) | Calibration method, device and system | |
| CN111402324A (en) | Target measuring method, electronic equipment and computer storage medium | |
| CN116310263A (en) | Pointer type aviation horizon instrument indication automatic reading implementation method | |
| CN109974660A (en) | Method based on unmanned plane hovering video measuring unmanned plane hovering precision | |
| CN115861407A (en) | Safe distance detection method and system based on deep learning | |
| CN113436276B (en) | Visual relative positioning-based multi-unmanned aerial vehicle formation method | |
| CN115388891A (en) | Space positioning method and system for large-view-field moving target |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |