CN111862143A - Automatic river bank collapse monitoring method - Google Patents
Automatic river bank collapse monitoring method Download PDFInfo
- Publication number
- CN111862143A CN111862143A CN202010669018.6A CN202010669018A CN111862143A CN 111862143 A CN111862143 A CN 111862143A CN 202010669018 A CN202010669018 A CN 202010669018A CN 111862143 A CN111862143 A CN 111862143A
- Authority
- CN
- China
- Prior art keywords
- river bank
- collapse
- image
- monitoring
- moving target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 106
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000001514 detection method Methods 0.000 claims abstract description 87
- 230000011218 segmentation Effects 0.000 claims abstract description 32
- 230000008859 change Effects 0.000 claims abstract description 21
- 238000001914 filtration Methods 0.000 claims abstract description 13
- 238000012216 screening Methods 0.000 claims abstract description 10
- 239000012634 fragment Substances 0.000 claims abstract description 9
- 238000012545 processing Methods 0.000 claims abstract description 8
- 238000007781 pre-processing Methods 0.000 claims description 9
- 238000004891 communication Methods 0.000 claims description 8
- 238000013528 artificial neural network Methods 0.000 claims description 7
- 238000011176 pooling Methods 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 6
- 238000007477 logistic regression Methods 0.000 claims description 5
- 239000011800 void material Substances 0.000 claims description 4
- 230000004913 activation Effects 0.000 claims description 3
- 230000004927 fusion Effects 0.000 claims description 3
- 230000002146 bilateral effect Effects 0.000 claims description 2
- 239000003086 colorant Substances 0.000 claims description 2
- 238000004364 calculation method Methods 0.000 description 6
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 6
- 238000012806 monitoring device Methods 0.000 description 3
- 238000010606 normalization Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000003628 erosive effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000005299 abrasion Methods 0.000 description 1
- 230000002411 adverse Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000009991 scouring Methods 0.000 description 1
- 238000010008 shearing Methods 0.000 description 1
- 238000002791 soaking Methods 0.000 description 1
- 239000002689 soil Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000010356 wave oscillation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/215—Motion-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
- G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30181—Earth observation
- G06T2207/30184—Infrastructure
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an automatic river bank collapse monitoring method, which comprises the following steps: step 1, acquiring a monitoring video of a plurality of paths of monitoring equipment in real time, and primarily screening the monitoring video by using a time sequence convolution dynamic change monitoring video frame detection model to acquire monitoring video fragments with the possibility of river bank collapse; step 2, carrying out moving target detection on the obtained monitoring video segment by using a moving target detection model to obtain a moving target area image; performing semantic segmentation detection on the obtained surveillance video segments by using a river bank detection semantic segmentation model, and extracting river bank area images; step 3, performing image subtraction on the extracted river bank area and the obtained moving target area to obtain an image of the river bank area without the moving target; step 4, creating a Gaussian filter, carrying out gray processing and Gaussian filtering on the acquired river bank area image which does not contain the moving target, and establishing a background template; and 5, finally, determining the collapse position and the collapse area by using a frame difference method.
Description
Technical Field
The invention relates to the field of video image processing technology and deep learning algorithm, in particular to an automatic monitoring method for river bank collapse.
Background
The collapse of the river levee refers to the unfavorable geological phenomena of reconstruction deformation such as erosion (abrasion), collapse (collapse) collapse, displacement and the like of the river levee caused by the change of the dynamic pressure of underground water caused by the fluctuation of the water level of river levee and the aggravated weathering of the soil body of the river levee and the reduction of the scouring resistance strength of the river levee under the influence of factors such as river water soaking, wave impact, water flow erosion, dry-wet alternation and the like. The river bank collapse activity not only causes instability of a main river channel of a river, but also mainly damages the river bank, threatens dikes, even breaks the dikes, aggravates flood disasters, endangers life and property safety of people and causes adverse social influence.
According to published data, the prior various types of flood control of China are about 29.91 km, wherein the backbone dike is 6.57 km, and plays an important role in flood control in the flood season of the past year. However, the river bank collapse phenomenon happens occasionally due to the problems of river impact, artificial construction damage and the like, and if the river bank collapse phenomenon cannot be timely and effectively treated, the loss cannot be estimated. For important dam buildings, deformation monitoring can be carried out for a long time, but for common river levees, due to the fact that the monitoring range is too long, river levees are still detected in a manual inspection mode at present, the manual inspection mode usually has hysteresis, and therefore the river levees cannot be timely processed when collapse occurs, and personal safety of social public is harmed.
In order to solve the above problems, people are always seeking an ideal technical solution.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, and provides an automatic river bank collapse monitoring method, which solves the problem of the hysteresis of collapse detection and the problem of human resources.
In order to achieve the purpose, the invention adopts the following technical scheme:
an automatic river bank collapse monitoring method specifically comprises the following steps:
step 1, acquiring a monitoring video of a plurality of paths of monitoring equipment in real time, and primarily screening the monitoring video by using a time sequence convolution dynamic change monitoring video frame detection model to acquire monitoring video fragments with the possibility of river bank collapse;
step 2, carrying out moving target detection on the obtained monitoring video segment by using a moving target detection model to obtain a moving target area image;
performing semantic segmentation detection on the obtained surveillance video segments by using a river bank detection semantic segmentation model, and extracting river bank area images;
step 3, performing image subtraction on the extracted river bank area and the obtained moving target area to obtain an image of the river bank area without the moving target;
step 4, creating a Gaussian filter, carrying out gray processing and Gaussian filtering on the acquired river bank area image which does not contain the moving target, and establishing a background template;
And 5, finally, determining the collapse position and the collapse area by using a frame difference method.
Based on the above, the time sequence convolution dynamic change surveillance video frame detection model is formed by training a depth 3-dimensional convolution network model, the depth 3-dimensional convolution network comprises 8 convolution layers, 5 pooling layers and two full-connection layers, all 3D convolution filters are 3 × 3 × 3, and the step length is 1 × 1 × 1; the input of the depth 3-dimensional convolution network model is as follows: a surveillance video segment with a data dimension of 3 × 16 × 112 is obtained by preprocessing a surveillance video sequence, wherein 3 is the number of image channels, 16 is the length of the surveillance video sequence, and 112 × 112 is the image size;
performing logistic regression activation on the output data value of the depth 3-dimensional convolution network model to predict collapse phenomena and judge the type of the monitoring video segment, wherein if the output result is 1, the collapse phenomena possibly exist in the monitoring video segment; and if the output result is 0, the monitoring video clip has no collapse phenomenon.
Based on the above, the moving object detection model is trained by a yolo v3 model, and the input of the yolo v3 model is the monitoring video segment obtained in step 1, wherein the monitoring video segment is possible to have river bank collapse.
Based on the above, the river bank detection semantic segmentation model is trained by a deep neural network based on deep lab V3+, wherein the input of the semantic segmentation model is the monitoring video segment obtained in step 1, which is likely to have river bank collapse;
the deep neural network based on the deep lab V3+ takes Resnet101 as a basic network, selects a basic network Conv1 and a basic network Conv2 as feature extractors, wherein the basic network Conv1 performs 7 multiplied by 7 convolution operation, and the number of filters is 64; introducing void convolution to perform 3 × 3 multi-scale pooling, modifying the void volume rate to be (8,16,24), and constructing a characteristic encoder; and constructing a decoder, connecting the decoder with the feature extractor and the feature encoder, thereby realizing the fusion of the image features and the encoding features, and introducing deconvolution to carry out up-sampling operation.
Based on the above, step 5 specifically includes the following steps:
step 5.1, performing interframe difference on the gray image sequence obtained in the step 4, and taking two adjacent images in time sequence for difference;
step 5.2, marking the pixel point with the gray value greater than or equal to the preset pixel threshold value as 1, and indicating that the pixel point is the position where the river bank collapses; marking the pixel point with the gray value smaller than the preset pixel threshold value as 0, and indicating that the pixel point is a position without river bank collapse;
Step 5.3, communicating all the pixel points marked as 1, and calculating the area of a communication area;
and 5.4, extracting the outline of the communication area with the largest area to obtain a collapse coordinate position.
The invention also provides an automatic river bank collapse monitoring system which comprises a plurality of paths of monitoring equipment, edge computing equipment connected with the plurality of paths of monitoring equipment and a cloud monitoring platform connected with the edge computing equipment;
the multi-path monitoring equipment is arranged on the river bank and used for acquiring a monitoring video of the river bank;
the edge meter equipment is provided with a time sequence convolution dynamic change surveillance video frame detection module, and a time sequence convolution dynamic change surveillance video frame detection model is arranged in the time sequence convolution dynamic change surveillance video frame detection module and is used for primarily screening surveillance videos and obtaining surveillance video fragments which are possible to have river bank collapse;
the cloud monitoring platform is provided with a moving target detection module, a river bank detection semantic segmentation module, a river bank image difference making module and an image difference collapse detection module, wherein the moving target detection module is internally provided with a moving target detection model and is used for detecting a moving target of the obtained monitoring video segment and obtaining a moving target area image; a river bank detection semantic segmentation model is arranged in the river bank detection semantic segmentation module and is used for performing semantic segmentation detection on the acquired surveillance video segments and extracting river bank area images; the river bank image difference making module is used for making image difference between the extracted river bank area and the obtained moving target area to obtain a river bank area image not containing the moving target; the image differential collapse detection module is internally provided with a Gaussian filter and is used for carrying out gray processing and Gaussian filtering on the acquired river bank area image which does not contain the moving target, establishing a background template and determining a collapse position and an area by using a frame difference method.
Compared with the prior art, the method has outstanding substantive characteristics and remarkable progress, and particularly adopts a time sequence convolution dynamic change monitoring video frame detection model to primarily screen and obtain monitoring video segments which are possible to have river bank collapse, then adopts a semantic segmentation network and a target detection network to respectively carry out river bank extraction and moving target detection on the monitoring video segments which are possible to have river bank collapse so as to obtain river bank area images which do not contain moving targets, and utilizes image differential collapse detection to quickly and accurately determine collapse positions and areas based on the river bank area images which do not contain moving targets; the method solves the problem of river bank collapse detection, and simultaneously places the function of primarily screening and obtaining monitoring video segments which possibly have river bank collapse by adopting a time sequence convolution dynamic change monitoring video frame detection model on the edge computing equipment, so that the communication pressure of the multi-path monitoring equipment and the cloud monitoring platform is reduced; meanwhile, more accurate monitoring of the river bank collapse position and calculation of the collapse area are placed on a cloud monitoring platform; by combining the characteristics of low delay of edge calculation and strong performance of cloud calculation, the rapid and accurate detection of river bank collapse is further ensured.
Drawings
Fig. 1 is a schematic flow chart of an automatic river bank collapse monitoring method according to the present invention.
FIG. 2 is a schematic block diagram of a semantic segmentation network according to the present invention.
Fig. 3 is a schematic block diagram of an automatic river bank collapse monitoring system according to the present invention.
Detailed Description
The following describes in further detail specific embodiments of the present invention with reference to the accompanying drawings.
Example 1
The invention provides an automatic river bank collapse monitoring method, which specifically comprises the following steps of:
step 1, acquiring a monitoring video of a plurality of paths of monitoring equipment in real time, and primarily screening the monitoring video by using a time sequence convolution dynamic change monitoring video frame detection model to acquire monitoring video fragments with the possibility of river bank collapse;
step 2, carrying out moving target detection on the obtained monitoring video segment by using a moving target detection model to obtain a moving target area image;
performing semantic segmentation detection on the obtained surveillance video segments by using a river bank detection semantic segmentation model, and extracting river bank area images;
step 3, performing image subtraction on the extracted river bank area and the obtained moving target area to obtain an image of the river bank area without the moving target;
step 4, creating a Gaussian filter, carrying out gray processing and Gaussian filtering on the acquired river bank area image which does not contain the moving target, and establishing a background template;
And 5, finally, determining the collapse position and the collapse area by using a frame difference method.
Specifically, in step 1, the time series convolution dynamic change surveillance video frame detection model is trained by a depth 3-dimensional convolution network model, the depth 3-dimensional convolution network includes 8 convolution layers, 5 pooling layers and two full-connection layers, all 3D convolution filters are 3 × 3 × 3, and the step length is 1 × 1 × 1; the input of the depth 3-dimensional convolution network model is as follows: a surveillance video segment with a data dimension of 3 × 16 × 112 is obtained by preprocessing a surveillance video sequence, wherein 3 is the number of image channels, 16 is the length of the surveillance video sequence, and 112 × 112 is the image size;
performing logistic regression activation on the output data value of the depth 3-dimensional convolution network model to predict collapse phenomena and judge the type of the monitoring video segment, wherein if the output result is 1, the collapse phenomena possibly exist in the monitoring video segment; and if the output result is 0, the monitoring video clip has no collapse phenomenon.
Specifically, the formula of the logistic regression is as follows:
in the above formula, x is output data of the depth 3-dimensional convolution network model, and y is a logistic regression output value.
In this embodiment, the probability threshold is set to be 0.5, and when the calculated collapse probability prediction value is greater than 0.5, 1 is output to indicate that there is a collapse phenomenon in the monitoring video segment, so that subsequent accurate detection can be performed; otherwise it means that no subsequent accurate detection is required.
Specifically, in step 1, the monitoring video of the multi-channel monitoring device is an RGB color image, and before the RGB color image is sent to the depth 3-dimensional convolution network model, the following formula is used to perform image preprocessing:
in the above equation, R, G and B are the red, green, and blue components of the RGB color space, respectively.
Specifically, in step 2, the moving object detection model is trained by a yolo v3 model, and the input of the yolo v3 model is the surveillance video segment obtained in step 1, where river bank collapse may exist.
Specifically, in the step 2, the river levee detection semantic segmentation model is formed by deep neural network training based on deep lab V3+, a random gradient descent algorithm is selected for the learning rate in the training process, and an mIoU is selected for the performance evaluation standard; inputting the semantic segmentation model into the monitoring video segment which is obtained in the step 1 and is possible to have river bank collapse;
before inputting the surveillance video segment obtained in step 1 and having the possibility of river bank collapse into the deep neural network based on deep lab V3+, an original data set needs to be formed through a frame-extracting scaling screening operation and through data enhancement operations such as rotation, scaling, shearing and shifting, color filtering, frequency domain rate enhancement and gaussian filtering;
Setting semantically segmented labels, wherein the labels are classified into sky, trees, river levee and river water; manually carrying out pixel-level semantic annotation on the original data set by using a defined label to manufacture a label data set;
carrying out format conversion on the tag data set to form an input data set adapted to the river bank detection semantic segmentation network;
calculating the proportion of pixels and occupation of each category of the input data set, carrying out weighted average, and balancing the proportion of the pixels of the categories;
dividing an input data set into a training set, a verification set and a test set according to a ratio of 7:2: 1;
as shown in fig. 2, the deep neural network based on deep lab V3+ takes Resnet101 as a base network, and selects a base network Conv1 and a base network Conv2 as feature extractors, wherein the base network Conv1 performs 7 × 7 convolution operations, and the number of filters is 64; introducing a cavity convolution for 3 x 3 multi-scale pooling, modifying the cavity volume rate to be (8,16,24), and constructing a characteristic encoder;
specifically, 3 × 3 convolution operation is performed on the features subjected to the Conv5 convolution operation, the number of filters is 256, the hole sampling rate is 8, and batch normalization operation is performed;
carrying out 3 × 3 convolution operation on the features subjected to Conv5 convolution operation, wherein the number of filters is 256, the hole sampling rate is 16, and carrying out batch normalization operation;
The features that were subjected to the Conv5 convolution operation were subjected to a 3 × 3 convolution operation with 256 filters and a hole sampling rate of 24, and a batch normalization operation was performed.
And constructing a decoder, connecting the decoder with the feature extractor and the feature encoder, thereby realizing the fusion of the image features and the encoding features, and introducing deconvolution to carry out up-sampling operation.
Specifically, in step 3, the mask map of the river bank region extracted by semantic segmentation detection is used, the mask map of the moving target region obtained by moving target detection is subtracted, pixel points with the gray difference value larger than 0 in the mask map of the river bank region are reserved, and the gray value of the pixel points with the gray difference value smaller than 0 in the mask map of the river bank region is set to be 0.
Specifically, in step 4, the gaussian filtering employs bilateral positions and colors for combined filtering, and the kernel is:
wherein p isiIndicating the position of the ith pixel, pjIndicating the position of the jth pixel, IiIndicating the color intensity, I, of the ith pixeljIndicating the color intensity of the jth pixel,
and
are all hyper-parameters used to control the gaussian kernel scale.
Specifically, the step 5 specifically includes the following steps:
step 5.1, performing interframe difference on the gray image sequence obtained in the step 4, and taking two adjacent images in time sequence for difference;
Step 5.2, marking the pixel point with the gray value greater than or equal to the preset pixel threshold value as 1, and indicating that the pixel point is the position where the river bank collapses; marking the pixel point with the gray value smaller than the preset pixel threshold value as 0, and indicating that the pixel point is a position without river bank collapse;
step 5.3, communicating all the pixel points marked as 1, and calculating the area of a communication area;
and 5.4, extracting the outline of the communication area with the largest area to obtain a collapse coordinate position.
Compared with the prior art, the method has outstanding substantive characteristics and remarkable progress, and particularly adopts a time sequence convolution dynamic change monitoring video frame detection model to primarily screen and obtain monitoring video fragments with the possibility of river bank collapse, and then adopts a semantic segmentation network to extract the river bank from the monitoring video fragments with the possibility of river bank collapse so as to remove the influence of water wave oscillation, branch shaking and illumination on the detection of the river bank collapse; meanwhile, a target detection network is adopted to extract a moving target in the image, and a river bank area image which does not contain the moving target is further obtained, so that the interference of moving objects on a river bank is reduced; and based on the river bank area image without the moving target, determining the collapse position and area by using image differential collapse detection, and finally, rapidly and accurately realizing river bank detection.
Example 2
The embodiment provides an automatic river bank collapse monitoring system, as shown in fig. 3, which includes a plurality of paths of monitoring devices, an edge computing device connected to the plurality of paths of monitoring devices, and a cloud monitoring platform connected to the edge computing device;
the multi-path monitoring equipment is arranged on the river bank and used for acquiring a monitoring video of the river bank;
the edge meter equipment is provided with a time sequence convolution dynamic change surveillance video frame detection module, and a time sequence convolution dynamic change surveillance video frame detection model is arranged in the time sequence convolution dynamic change surveillance video frame detection module and is used for primarily screening surveillance videos and obtaining surveillance video fragments which are possible to have river bank collapse;
the cloud monitoring platform is provided with a moving target detection module, a river bank detection semantic segmentation module, a river bank image difference making module and an image difference collapse detection module, wherein the moving target detection module is internally provided with a moving target detection model and is used for detecting a moving target of the obtained monitoring video segment and obtaining a moving target area image; a river bank detection semantic segmentation model is arranged in the river bank detection semantic segmentation module and is used for performing semantic segmentation detection on the acquired surveillance video segments and extracting river bank area images; the river bank image difference making module is used for making image difference between the extracted river bank area and the obtained moving target area to obtain a river bank area image not containing the moving target; the image differential collapse detection module is internally provided with a Gaussian filter and is used for carrying out gray processing and Gaussian filtering on the acquired river bank area image which does not contain the moving target, establishing a background template and determining a collapse position and an area by using a frame difference method.
The edge meter equipment is also provided with an image preprocessing module, and the image preprocessing module preprocesses the monitoring video by using the following formula:
in the above formula, R, G and B are the red, green and blue components of the RGB color space, respectively;
and dividing the preprocessed surveillance video into non-overlapping 16-frame video segments, wherein the image size of each frame of video is 112 × 112.
The method comprises the steps of placing a function of primarily screening and obtaining monitoring video segments which possibly have river bank collapse by adopting a time sequence convolution dynamic change monitoring video frame detection model on edge computing equipment, so that the communication pressure of a plurality of paths of monitoring equipment and a cloud monitoring platform is reduced; meanwhile, more accurate monitoring of the river bank collapse position and calculation of the collapse area are placed on a cloud monitoring platform; by combining the characteristics of low delay of edge calculation and strong performance of cloud calculation, the rapid and accurate detection of river bank collapse is further ensured.
Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present invention and not to limit the same; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art will understand that: modifications to the specific embodiments of the invention or equivalent substitutions for parts of the technical features may be made; without departing from the spirit of the present invention, it is intended to cover all aspects of the invention as defined by the appended claims.
Claims (10)
1. An automatic river bank collapse monitoring method is characterized by comprising the following steps:
step 1, acquiring a monitoring video of a plurality of paths of monitoring equipment in real time, and primarily screening the monitoring video by using a time sequence convolution dynamic change monitoring video frame detection model to acquire monitoring video fragments with the possibility of river bank collapse;
step 2, carrying out moving target detection on the obtained monitoring video segment by using a moving target detection model to obtain a moving target area image;
performing semantic segmentation detection on the obtained surveillance video segments by using a river bank detection semantic segmentation model, and extracting river bank area images;
step 3, performing image subtraction on the extracted river bank area and the obtained moving target area to obtain an image of the river bank area without the moving target;
step 4, creating a Gaussian filter, carrying out gray processing and Gaussian filtering on the acquired river bank area image which does not contain the moving target, and establishing a background template;
and 5, finally, determining the collapse position and the collapse area by using a frame difference method.
2. The automatic river bank collapse monitoring method according to claim 1, wherein: the time sequence convolution dynamic change monitoring video frame detection model is formed by training a depth 3-dimensional convolution network model, the depth 3-dimensional convolution network comprises 8 convolution layers, 5 pooling layers and two full-connection layers, all 3D convolution filters are 3 multiplied by 3, and the step length is 1 multiplied by 1; the input of the depth 3-dimensional convolution network model is as follows: a surveillance video segment with a data dimension of 3 × 16 × 112 is obtained by preprocessing a surveillance video sequence, wherein 3 is the number of image channels, 16 is the length of the surveillance video sequence, and 112 × 112 is the image size;
Performing logistic regression activation on the output data value of the depth 3-dimensional convolution network model to predict collapse phenomena and judge the type of the monitoring video segment, wherein if the output result is 1, the collapse phenomena possibly exist in the monitoring video segment; and if the output result is 0, the monitoring video clip has no collapse phenomenon.
3. The automatic river bank collapse monitoring method according to claim 1, wherein: the moving object detection model is trained by a yolo v3 model, and the input of the yolo v3 model is the monitoring video segment obtained in the step 1 and having the possibility of river bank collapse.
4. The automatic river bank collapse monitoring method according to claim 1, wherein: the river levee detection semantic segmentation model is formed by training a deep neural network based on deep Lab V3+, wherein the input of the semantic segmentation model is the monitoring video segment which is obtained in the step 1 and is possible to have river levee collapse;
the deep neural network based on the deep lab V3+ takes Resnet101 as a basic network, selects a basic network Conv1 and a basic network Conv2 as feature extractors, wherein the basic network Conv1 performs 7 multiplied by 7 convolution operation, and the number of filters is 64; introducing void convolution to perform 3 × 3 multi-scale pooling, modifying the void volume rate to be (8,16,24), and constructing a characteristic encoder; and constructing a decoder, connecting the decoder with the feature extractor and the feature encoder, thereby realizing the fusion of the image features and the encoding features, and introducing deconvolution to carry out up-sampling operation.
5. The automatic river bank collapse monitoring method according to claim 2, wherein:
in step 1, the monitoring video of the multi-channel monitoring equipment is an RGB color image, and before the RGB color image is sent to the depth 3-dimensional convolution network model, the image preprocessing is further performed using the following formula:
in the above equation, R, G and B are the red, green, and blue components of the RGB color space, respectively.
6. The automatic river bank collapse monitoring method according to claim 1, wherein: in step 3, the mask image of the river bank area extracted through semantic segmentation detection is used, the mask image of the moving target area obtained through moving target detection is subtracted, pixel points with the gray difference value larger than 0 in the mask image of the river bank area are reserved, and the gray value of the pixel points with the gray difference value smaller than 0 in the mask image of the river bank area is set to be 0.
7. The automatic river bank collapse monitoring method according to claim 1, wherein: in step 4, the gaussian filtering adopts bilateral positions and colors for combined filtering, and the kernel is as follows:
wherein p isiIndicating the position of the ith pixel, pjIndicating the position of the jth pixel, IiIndicating the color intensity, I, of the ith pixel jIndicating the color intensity of the jth pixel,
and
are all hyper-parameters used to control the gaussian kernel scale.
8. The method for automatically monitoring river bank collapse according to claim 1, wherein the step 5 specifically comprises the following steps:
step 5.1, performing interframe difference on the gray image sequence obtained in the step 4, and taking two adjacent images in time sequence for difference;
step 5.2, marking the pixel point with the gray value greater than or equal to the preset pixel threshold value as 1, and indicating that the pixel point is the position where the river bank collapses; marking the pixel point with the gray value smaller than the preset pixel threshold value as 0, and indicating that the pixel point is a position without river bank collapse;
step 5.3, communicating all the pixel points marked as 1, and calculating the area of a communication area;
and 5.4, extracting the outline of the communication area with the largest area to obtain a collapse coordinate position.
9. The utility model provides a river levee automatic monitoring system that collapses which characterized in that: the system comprises a plurality of paths of monitoring equipment, edge computing equipment connected with the plurality of paths of monitoring equipment and a cloud monitoring platform connected with the edge computing equipment;
the multi-path monitoring equipment is arranged on the river bank and used for acquiring a monitoring video of the river bank;
The edge meter equipment is provided with a time sequence convolution dynamic change surveillance video frame detection module, and a time sequence convolution dynamic change surveillance video frame detection model is arranged in the time sequence convolution dynamic change surveillance video frame detection module and is used for primarily screening surveillance videos and obtaining surveillance video fragments which are possible to have river bank collapse;
the cloud monitoring platform is provided with a moving target detection module, a river bank detection semantic segmentation module, a river bank image difference making module and an image difference collapse detection module, wherein the moving target detection module is internally provided with a moving target detection model and is used for detecting a moving target of the obtained monitoring video segment and obtaining a moving target area image; a river bank detection semantic segmentation model is arranged in the river bank detection semantic segmentation module and is used for performing semantic segmentation detection on the acquired surveillance video segments and extracting river bank area images; the river bank image difference making module is used for making image difference between the extracted river bank area and the obtained moving target area to obtain a river bank area image not containing the moving target; the image differential collapse detection module is internally provided with a Gaussian filter and is used for carrying out gray processing and Gaussian filtering on the acquired river bank area image which does not contain the moving target, establishing a background template and determining a collapse position and an area by using a frame difference method.
10. The automatic river bank collapse monitoring system according to claim 9, wherein: the edge meter equipment is also provided with an image preprocessing module, and the image preprocessing module preprocesses the monitoring video by using the following formula:
in the above formula, R, G and B are the red, green and blue components of the RGB color space, respectively;
and dividing the preprocessed surveillance video into non-overlapping 16-frame video segments, wherein the image size of each frame of video is 112 × 112.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010669018.6A CN111862143B (en) | 2020-07-13 | 2020-07-13 | Automatic monitoring method for river dike collapse |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010669018.6A CN111862143B (en) | 2020-07-13 | 2020-07-13 | Automatic monitoring method for river dike collapse |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111862143A true CN111862143A (en) | 2020-10-30 |
| CN111862143B CN111862143B (en) | 2024-03-19 |
Family
ID=72983963
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010669018.6A Active CN111862143B (en) | 2020-07-13 | 2020-07-13 | Automatic monitoring method for river dike collapse |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111862143B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112557815A (en) * | 2020-11-27 | 2021-03-26 | 广东电网有限责任公司肇庆供电局 | Distribution network line tree obstacle identification and fault positioning method based on fixed and movable inspection images |
| CN112907621A (en) * | 2021-02-24 | 2021-06-04 | 华南理工大学 | Moving object extraction method based on difference and semantic information fusion |
| CN113191294A (en) * | 2021-05-11 | 2021-07-30 | 山东浪潮科学研究院有限公司 | Machine vision-based large-sized floating object collision dam prevention and detection method |
| CN113191367A (en) * | 2021-05-25 | 2021-07-30 | 华东师范大学 | Semantic segmentation method based on dense scale dynamic network |
| CN113720851A (en) * | 2021-08-02 | 2021-11-30 | 重庆市地质灾害防治中心 | Dangerous rock body acousto-optic combined intelligent monitoring and early warning method |
| CN114255250A (en) * | 2021-12-28 | 2022-03-29 | 黄河勘测规划设计研究院有限公司 | River protection engineering slope collapse detection method based on deep learning |
| CN114638436A (en) * | 2022-03-31 | 2022-06-17 | 邯郸市水利水电勘测设计研究院 | Emergency early warning method and system for river regulation project |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20110099904A (en) * | 2010-03-03 | 2011-09-09 | (주)위니텍 | Method for sensing a moving object on the basis of real-time moving picture and breakwater watching system using the method |
| CN104134080A (en) * | 2014-08-01 | 2014-11-05 | 重庆大学 | Method and system for automatically detecting roadbed collapse and side slope collapse of road |
| KR102008973B1 (en) * | 2019-01-25 | 2019-08-08 | (주)나스텍이앤씨 | Apparatus and Method for Detection defect of sewer pipe based on Deep Learning |
| CN110717886A (en) * | 2019-09-03 | 2020-01-21 | 南京理工大学 | Pavement pool detection method based on machine vision in complex environment |
| CN111127493A (en) * | 2019-11-12 | 2020-05-08 | 中国矿业大学 | Remote sensing image semantic segmentation method based on attention multi-scale feature fusion |
-
2020
- 2020-07-13 CN CN202010669018.6A patent/CN111862143B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20110099904A (en) * | 2010-03-03 | 2011-09-09 | (주)위니텍 | Method for sensing a moving object on the basis of real-time moving picture and breakwater watching system using the method |
| CN104134080A (en) * | 2014-08-01 | 2014-11-05 | 重庆大学 | Method and system for automatically detecting roadbed collapse and side slope collapse of road |
| KR102008973B1 (en) * | 2019-01-25 | 2019-08-08 | (주)나스텍이앤씨 | Apparatus and Method for Detection defect of sewer pipe based on Deep Learning |
| CN110717886A (en) * | 2019-09-03 | 2020-01-21 | 南京理工大学 | Pavement pool detection method based on machine vision in complex environment |
| CN111127493A (en) * | 2019-11-12 | 2020-05-08 | 中国矿业大学 | Remote sensing image semantic segmentation method based on attention multi-scale feature fusion |
Non-Patent Citations (1)
| Title |
|---|
| 陈莹;孙洪泉;赵祥;唐宏;周廷刚;: "地震灾区河谷滑坡检测的遥感分析――以北川县滑坡为例", 自然灾害学报, no. 01 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112557815A (en) * | 2020-11-27 | 2021-03-26 | 广东电网有限责任公司肇庆供电局 | Distribution network line tree obstacle identification and fault positioning method based on fixed and movable inspection images |
| CN112557815B (en) * | 2020-11-27 | 2022-05-20 | 广东电网有限责任公司肇庆供电局 | Fixed and movable inspection image-based distribution network line tree obstacle identification and fault positioning method |
| CN112907621A (en) * | 2021-02-24 | 2021-06-04 | 华南理工大学 | Moving object extraction method based on difference and semantic information fusion |
| CN113191294A (en) * | 2021-05-11 | 2021-07-30 | 山东浪潮科学研究院有限公司 | Machine vision-based large-sized floating object collision dam prevention and detection method |
| CN113191294B (en) * | 2021-05-11 | 2022-06-17 | 山东浪潮科学研究院有限公司 | Machine vision-based large-sized floating object collision dam prevention and detection method |
| CN113191367A (en) * | 2021-05-25 | 2021-07-30 | 华东师范大学 | Semantic segmentation method based on dense scale dynamic network |
| CN113720851A (en) * | 2021-08-02 | 2021-11-30 | 重庆市地质灾害防治中心 | Dangerous rock body acousto-optic combined intelligent monitoring and early warning method |
| CN114255250A (en) * | 2021-12-28 | 2022-03-29 | 黄河勘测规划设计研究院有限公司 | River protection engineering slope collapse detection method based on deep learning |
| CN114638436A (en) * | 2022-03-31 | 2022-06-17 | 邯郸市水利水电勘测设计研究院 | Emergency early warning method and system for river regulation project |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111862143B (en) | 2024-03-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111862143B (en) | Automatic monitoring method for river dike collapse | |
| CN112232391B (en) | Dam crack detection method based on U-net network and SC-SAM attention mechanism | |
| CN110033040B (en) | Flame identification method, system, medium and equipment | |
| CN112861690B (en) | Multi-method fused remote sensing image change detection method and system | |
| CN109740485B (en) | Reservoir or small reservoir identification method based on spectral analysis and deep convolutional neural network | |
| CN111145176A (en) | Method and system for automatically identifying lymph node staining pathological image of gastric cancer based on deep neural network | |
| CN110210608A (en) | The enhancement method of low-illumination image merged based on attention mechanism and multi-level features | |
| CN110335294B (en) | Mine water pump house water leakage detection method based on frame difference method and 3D convolutional neural network | |
| CN112488046B (en) | Lane line extraction method based on high-resolution images of unmanned aerial vehicle | |
| CN111797712A (en) | Remote sensing image cloud and cloud shadow detection method based on multi-scale feature fusion network | |
| CN111860143B (en) | Real-time flame detection method for inspection robot | |
| CN106650812A (en) | City water body extraction method for satellite remote sensing image | |
| CN113139528B (en) | Unmanned aerial vehicle thermal infrared image dam dangerous case detection method based on fast _ RCNN | |
| CN110726725A (en) | Transmission line hardware corrosion detection method and device | |
| CN113111878B (en) | Infrared weak and small target detection method under complex background | |
| CN107564018A (en) | It is a kind of to utilize the method for improving iterative algorithm extraction target image | |
| CN109886221A (en) | Sand dredger recognition methods based on saliency detection | |
| CN113435452A (en) | Electrical equipment nameplate text detection method based on improved CTPN algorithm | |
| Cheng et al. | Embankment crack detection in UAV images based on efficient channel attention U2Net | |
| CN114519694A (en) | Seven-segment digital tube liquid crystal display screen identification method and system based on deep learning | |
| CN111080562B (en) | Substation suspender identification method based on enhanced image contrast | |
| CN116416270B (en) | Method and device for monitoring underground working surface leakage water | |
| CN112132867A (en) | Remote sensing image transformation detection method and device | |
| CN116797941A (en) | Marine oil spill risk source rapid intelligent identification and classification method for high-resolution remote sensing image | |
| CN114782875B (en) | Fish fine granularity information acquisition method based on fishway construction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |