CN111339905A - CIM well lid state visual detection system based on deep learning and multi-view angle - Google Patents
CIM well lid state visual detection system based on deep learning and multi-view angle Download PDFInfo
- Publication number
- CN111339905A CN111339905A CN202010109313.6A CN202010109313A CN111339905A CN 111339905 A CN111339905 A CN 111339905A CN 202010109313 A CN202010109313 A CN 202010109313A CN 111339905 A CN111339905 A CN 111339905A
- Authority
- CN
- China
- Prior art keywords
- well lid
- state
- unit
- manhole cover
- mccim
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000000007 visual effect Effects 0.000 title claims abstract description 39
- 238000001514 detection method Methods 0.000 title claims abstract description 37
- 238000013135 deep learning Methods 0.000 title claims abstract description 16
- 230000009466 transformation Effects 0.000 claims abstract description 13
- 238000012549 training Methods 0.000 claims description 24
- 238000005516 engineering process Methods 0.000 claims description 14
- 238000002372 labelling Methods 0.000 claims description 13
- 238000003062 neural network model Methods 0.000 claims description 10
- 238000006073 displacement reaction Methods 0.000 claims description 8
- 238000013528 artificial neural network Methods 0.000 claims description 7
- 239000011159 matrix material Substances 0.000 claims description 7
- 238000011179 visual inspection Methods 0.000 claims description 6
- 125000004122 cyclic group Chemical group 0.000 claims description 3
- 238000009877 rendering Methods 0.000 claims description 3
- 230000001131 transforming effect Effects 0.000 claims description 3
- 238000012800 visualization Methods 0.000 claims description 3
- 238000000034 method Methods 0.000 abstract description 17
- 238000007726 management method Methods 0.000 description 8
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 3
- 238000012795 verification Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000007635 classification algorithm Methods 0.000 description 2
- 238000013145 classification model Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 241000422846 Sequoiadendron giganteum Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000009430 construction management Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000011478 gradient descent method Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Medical Informatics (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a CIM well lid state visual detection system based on deep learning and multiple visual angles, which is used for realizing well lid state visual detection based on a well lid city information model MCCIM, wherein the well lid city information model MCCIM comprises geographical position information of a well lid, a well lid state, well lid attribute information and nearby traffic light signal geographical position information, and the system comprises a sensing unit, a perspective transformation unit, a well lid positioning unit, a well lid state judgment unit, a traffic light control unit and a visual management unit. By using the method and the device, the cost required by the system is reduced, the accuracy of judging the state of the well lid is improved, and the safety of urban traffic is improved by controlling the traffic signal lamp near the road with the missing or displaced well lid in the well lid state detection.
Description
Technical Field
The invention relates to the technical field of smart cities, in particular to a CIM well lid state visual detection system based on deep learning and multiple visual angles.
Background
In recent years, well lids on urban roads have been frequently lost due to rampant of lawless persons. This has a serious impact on road safety. Especially at night, under the dim environment illumination condition, because can't observe the disappearance of well lid, very easily take place the incident for vehicle and pedestrian that normally pass on the road. Because some units are not standard in construction management, the well lid can be shifted, and a hidden trouble that the well lid is not easy to observe is formed. Therefore, the loss and damage of the well cover cause huge potential safety hazards to normal driving.
Some schemes use a plurality of sensors installed at the bottom of the well cover to monitor the state of the well cover. On one hand, the method needs to additionally supply power to the sensor, and on the other hand, the sensor is easily affected by the environment and high in manufacturing cost, and is not suitable for large-scale application in cities.
Some schemes adopt a single camera to detect or position the well lid, and on the one hand, the single camera is easy to be shielded by big trees, birds and the like to cause detection failure. On the other hand, the data of a single camera is too single, and misjudgment is easily caused to the well lid.
Therefore, the existing well lid state detection technology has the problems of high cost and easy misjudgment.
Disclosure of Invention
The invention provides a CIM well lid state visual detection system based on deep learning and multiple visual angles, which realizes well lid state detection, reduces the cost required by the system and improves the accuracy of well lid state judgment.
A CIM well lid state visual detection system based on deep learning and multi-view angle realizes well lid state visual detection based on a well lid city information model MCCIM, wherein the well lid city information model MCCIM comprises geographic position information of a well lid, a well lid state, well lid attribute information and geographic position information of nearby traffic lights, the system comprises a sensing unit, a perspective transformation unit, a well lid positioning unit, a well lid state judgment unit, a traffic light control unit and a visual management pipe unit, wherein,
the sensing unit is used for acquiring well lid area images by using cameras with different visual angles;
the perspective transformation unit is used for carrying out perspective transformation on the image shot by each camera based on the homography matrix and transforming the image into a top view;
the well lid positioning unit is used for positioning the position of the well lid by adopting a target detection cyclic convolution neural network based on deep learning;
the well lid state judging unit is used for judging the state of the well lid and comprises a pre-training module and a weight updating module:
the pre-training module is used for pre-training the classification neural network model according to the position of the well lid and the state mark of the well lid acquired by the well lid positioning unit;
the weight updating module is used for training the classification neural network model at a fixed time interval based on the expanded sample set and updating the network weight;
the traffic light control unit is used for determining the geographical position information of a nearby traffic signal light according to the geographical position information of the manhole cover in the manhole cover urban information model MCCIM when the manhole cover is displaced or lost, and sending a control signal to the traffic signal light to limit the traffic of a road direction in which the manhole cover is positioned;
and the visual management unit is used for displaying the well lid distribution and the well lid state of the city according to the well lid city information model MCCIM, so that the supervision of an administrator is facilitated.
The target detection circular convolution neural network adopts fast RCNN, and the size of an anchor frame is set to be 64, 128 and 256.
The well lid state judgment unit further comprises: and the sample expansion module is used for correcting the labeling result based on the network output result of the well lid images with the multiple visual angles, and expanding the sample set by taking the corrected labeling result and the well lid images as new samples.
The step of correcting the labeling result by the sample expansion module comprises the following steps: when most of the network output results of the multi-view images of the same well lid are the well lid missing, the well lid missing is determined to be a correct result, and the network output results of the few images with wrong judgment are modified to be missing and used as sample marking results; and when a small number of network output results of the multi-view images of the same well lid are well lid displacement, determining that the well lid displacement is a correct result, and modifying the sample output result of the image with the most wrong judgment into the displacement to serve as a sample marking result.
The visualization management unit includes:
the data acquisition module is used for acquiring the state of the well lid and transmitting the state to the well lid city information model MCCIM;
and the display module is used for rendering the manhole cover urban information model MCCIM by combining Web GIS and CIM technologies and displaying the manhole cover state and the control state of nearby traffic lights on a foreground Web page.
The invention has the beneficial effects that:
1. compared with the traditional sensor detection, the system has the advantages that the cost required by the system is reduced, the camera is only required to be maintained daily, and meanwhile, the equipment maintenance cost is reduced.
2. The method and the device combine the computer vision technology and the machine learning technology to analyze the graph to obtain the well lid state, and have higher detection precision compared with the traditional sensor detection.
3. According to the invention, the cameras with multiple visual angles are arranged to acquire images of the well lid area, on one hand, the multi-visual angle detection can obtain more abundant data, the anti-blocking property and robustness of the detection system are enhanced, and the well lid state detection precision is improved, on the other hand, the analysis results of the images with multiple visual angles can be utilized to correct the sample output result, so that the labeled sample required by the classification network is automatically generated, the complexity of manual labeling is avoided, the sample set is expanded by utilizing the corrected result and the images, the precision of the neural network model is improved, and the well lid state judgment result precision is further improved.
4. The invention constructs a manhole cover urban information model MCCIM, and realizes real-time monitoring of the state of the manhole cover of the whole city by combining Web GIS and CIM technologies, thereby realizing intelligent and real-time centralized management of the state of the manhole cover.
5. According to the invention, the traffic signal lamp near the road with the missing or shifted well cover is controlled by utilizing the well cover state obtained by judgment and utilizing the traffic signal lamp position recorded in the well cover urban information model and the control method, so that the safety of urban traffic is improved, and meanwhile, the intelligent auxiliary driving control method has high application value.
Drawings
FIG. 1 is a system framework diagram of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention provides a CIM well lid state visual detection system based on deep learning and multiple visual angles, and a system frame diagram is shown in figure 1. The following description will be made by way of specific examples.
The first embodiment is as follows:
a visual inspection system for the well lid state based on deep learning and multi-view is based on a well lid city information model MCCIM to realize visual inspection of the well lid state. Cim (city Information modeling), a model for effectively organizing a large amount of city Information, is a further upgrade of BIM technology, and can present various Information states of a city in real time by combining with GIS technology. The manhole Cover City Information model MCCIM (manhole Cover City Information modeling) comprises: well lid geographical position information, well lid state, well lid attribute information, nearby traffic signal lamp geographical position information. The well lid attribute information includes a well lid ID, a well lid type, a well lid brand, a well lid service life, well lid damage information, and the like. The system comprises a sensing unit, a well lid positioning unit, a well lid state judging unit, a traffic light control unit and a visual management unit.
The arrangement of the camera is important in order to obtain an image that can show the state of the well lid. The camera that is the visual angle of looking down that sets up to be located directly over the well lid can obtain the best image that presents the well lid state undoubtedly. However, the manhole cover is usually located on the road, and therefore, it is almost impossible to provide a camera directly above the manhole cover based on city planning considerations. And the camera with a single visual angle is difficult to accurately sense the well lid displacement state. In addition, street views where a plurality of well covers on the roadside are located are relatively complex, and a single camera may be shielded by a target object, so that detection failure is caused. Therefore, the invention adopts the cameras with different visual angles to collect the well lid area images. In a preferred embodiment, the cameras with different viewing angles can cover 360 degrees of environmental information around the manhole cover. In order to effectively utilize resources, the camera can utilize cameras for monitoring surrounding shops and roads, and an additional camera is arranged according to requirements on the basis so as to comprehensively cover the environment around the well cover. According to the invention, the well lid area images acquired by the cameras with different visual angles are utilized, and the perception information of the cameras at each angle is fused by a specific method based on a multi-view geometric method, so that the well lid area images have the characteristics of mutual correlation. By the multi-view geometry-based method, even if a certain camera is blocked or stops working, the system can still normally operate, and the situation of detection failure cannot occur. Meanwhile, the method based on the multi-view geometry can also automatically generate classification samples, expand sample sets and avoid manual labeling.
And the perspective transformation unit is used for carrying out perspective transformation on the image shot by each camera based on the homography matrix and transforming the image into a top view mode.
Because the cameras in the system may not face the manhole cover area, in order to ensure the uniformity of the images, the invention needs to perform perspective transformation on the images shot by each camera, and the images can be known according to a coordinate transformation formula of a homography matrix:
wherein (u, v) is the pixel position in the image shot by the camera, and (u ', v') is the pixel position after perspective transformation,
is a homography matrix representing the transformation relationship between the camera coordinate system and the top view mode. One camera corresponds to one homography matrix. The homography matrix can be obtained according to the mapping relation between the camera coordinate system and the camera pose parameters.
The transformed image is based on a top view of the camera plane, which contains a large number of street view elements, such as trees, sky, pedestrians, etc. The well lid area needs to be positioned in a complex street view, and the influence of the complex street view on the well lid state judgment is removed. Therefore, the present invention includes a well lid positioning unit.
And the well lid positioning unit is used for positioning the position of the well lid by adopting a target detection cyclic convolution neural network based on deep learning.
Specifically, the invention adopts fast RCNN to position the well cover area. Practitioners will appreciate that there are many similar target detection algorithms, such as SSD, YOLO, etc. The implementer may autonomously select the target detection algorithm based on speed and accuracy requirements.
In the original fast RCNN frame, the size of the anchor frame anchor is initially set to 128, 256, 512, and the well lid area occupies a small ratio in the whole picture, so in order to detect a small target and simultaneously meet the requirement of algorithm speed, the invention adds an anchor with the size of 64, and removes an anchor with the size of 512, namely, the size of the anchor is finally changed to 64, 128, 256. The modified anchor is more fit with the real data characteristic distribution, and the recall rate and the accuracy rate after the experiment improvement are higher than those of the original version.
Based on well lid positioning unit, can succeed in pinpointing well lid position. Further, the current state of the well lid needs to be classified according to the obtained position of the well lid.
The well lid state judging unit is used for judging the state of the well lid and comprises a pre-training module and a weight updating module.
And the pre-training module is used for pre-training the classification neural network model according to the position of the well lid obtained by the well lid positioning unit and the state mark of the well lid.
The pre-training classification neural network is obtained by means of training of a classification framework VGG 16. Firstly, a convolutional neural network model VGG16 is constructed, wherein the VGG16 network model comprises 13 convolutional layers, 5 pooling layers and 3 full-connection layers. The output of the original VGG16 last fully connected layer is 1000, representing a 1000-class item in the forecast imagenet. The invention changes the output of the last full-connection layer into 3 and outputs two main results of OK and NG, wherein OK represents that the well lid state is normal, and NG comprises two judgment results of shift or missing and can be respectively recorded as NG1 and NG 2. The invention trains based on the following loss functions:
wherein i is an index corresponding to the training sample in the training set, and yiTo predict value, tiIs the target value. In the implementation process, firstly, a well lid image sample is collected, the sample is labeled, a sample set and the label of the sample set are used as input, and a classification neural network model is trained to obtain an initial neural network.
Since the pre-training network model used at first cannot be applied to various working conditions, inconsistency inevitably occurs between the detection results of each viewing angle, and here an implementer should know that the precision of the pre-training model is directly related to the availability of the system of the present invention, and it should be ensured that the outputs of a plurality of viewing angles are mostly correct as much as possible, for example, the results observed and detected by a camera with 4 viewing angles may be three correct and one wrong, and worse cases only account for a few in a day, otherwise, the reliability of automatic labeling may be affected.
According to the result detected by the sensing information of the cameras, a plurality of manhole cover area images can be obtained at the same time. The invention carries out network training and well lid state judgment based on the camera images with multiple visual angles, and can correct the sample output result based on the network output result of the well lid images with multiple visual angles. Therefore, in order to improve the prediction effect of the model, the invention adopts a mode of automatically generating the labeled sample based on multi-visual sense to expand the sample set.
And the sample expansion module is used for correcting the labeling result based on the network output result of the well lid images with the multiple visual angles, and expanding the sample set by taking the corrected labeling result and the well lid images as new samples.
Assuming that the current well lid state is a missing state, because the corresponding characteristics are obvious, the cameras at multiple viewing angles can correctly judge the current well lid state according to the pre-training model. Meanwhile, a few cameras can judge the current well lid state wrongly. At this time, the current frame image collected by a few cameras with wrong judgment is marked as 'missing', and the image and the corresponding marking information are added into the sample set.
Assuming that the current well lid state is a shift state, the cameras at multiple viewing angles can misjudge the current well lid state due to the fact that the corresponding features of the current well lid state are fuzzy. Meanwhile, a few cameras can correctly judge the state of the well lid. At this time, most of the images collected by the cameras with wrong judgment are marked as 'displacement', and the images and the corresponding marking information are also added into the sample set.
Along with the lapse of time, prediction every time will produce the sample with label automatically to this carries out the sample set and expands, has both guaranteed the demand of model to sample quantity like this, has also saved a large amount of manual labeling, has practiced thrift a large amount of marking costs.
And the weight updating module is used for training the classification neural network model at a fixed time interval based on the expanded sample set and updating the network weight.
The sample set update timing may be implementation dependent. The implementer may take samples over a period of time, which may be determined at the discretion of the implementer based on the frequency of model updates. And each time the classification model is updated, the classification model is divided into a training set, a verification set and a test set according to the proportion according to the collected samples. In one embodiment, 20000 samples are collected, and the samples are processed according to the ratio of 8: 1: 1, divided into training set, verification set and test set.
The method adopts a random gradient descent method to optimize the parameters of the classification neural network model, the target error is 0.00001, the initial learning rate is 0.001, and in order to prevent the model from approaching a local minimum value, the learning rate needs to be decreased progressively according to a mode of reducing 10 times per 100 iterations. And the implementer observes the network convergence condition in time and the prediction accuracy on the verification set, and stops training at a proper node. At intervals, an implementer can train samples accumulated in the previous time once, update model parameters and improve the classification accuracy.
The invention does not require the specific time of updating, and an implementer can select a proper time interval updating model according to the actual situation and the calculation speed of the adopted hardware platform.
Further, the implementer can write a script for automatic training without manually observing and judging convergence conditions, namely setting an appropriate early-stopping strategy, and the specific method of early-stopping is not included in the protection content of the invention because the specific method of early-stopping is various and well-known.
The implementer should know that the classification algorithm can also be implemented by using networks such as mobilenet, alexnet and the like, and the implementer can flexibly select the classification algorithm according to the requirements of speed and precision.
The well lid is lost and shifts, can cause very big potential safety hazard to pedestrian and the vehicle that traveles on the road. On the one hand, can integrate pronunciation early warning module in the camera, when the well lid is lost and shift, remind vehicle and pedestrian to notice the detour through pronunciation. However, for the manhole cover on the road, since the driving environment is noisy, it is not enough to remind only by voice. Therefore, when the well lid is displaced or lost, the traffic light signal nearby should be controlled by combining the well lid city information model. The invention is provided with a traffic light control unit for controlling traffic signal lights near the manhole cover.
And the traffic light control unit is used for determining the geographical position information of a nearby traffic signal light according to the geographical position information of the well cover in the well cover urban information model MCCIM when the well cover is displaced or lost, and sending a control signal to the traffic signal light to limit the traffic of the road direction where the well cover is positioned.
In the implementation process, the traffic signal lamp can be controlled according to the positions of the traffic signal lamp and the manhole cover. In particular, in the manhole cover city information model MCCIM, it is possible to record for each manhole cover the associated traffic light position and how the traffic light should react when the manhole cover state is abnormal. For example, if the manhole cover is located near an intersection, its associated traffic lights include at least four, the location of each traffic light is recorded, and the emergency response that each traffic light should make is recorded. For example, if the position of the manhole cover is located on the left side of the traffic signal lamp, the left-turn traffic is limited, namely the left-turn signal lamp is red.
In order to visually present the well lid state of the current area, enable a user to visually acquire the well lid state of the current area, enable an administrator to monitor the well lid state more visually, and provide early warning information and road traffic safety guarantee, the MCCIM information model is displayed on the Web through a Web GIS technology in combination with a WebGIS visualization technology.
Visual management unit for the well lid state and the well lid distribution in display area, the administrator of being convenient for supervises, visual management unit includes:
the data acquisition module is used for acquiring the state of the well lid and transmitting the state to the well lid city information model MCCIM;
and the display module is used for rendering the manhole cover urban information model MCCIM by combining the Web GIS and CIM technologies and displaying the manhole cover state and the control state of nearby traffic lights on a foreground Web page.
Meanwhile, the system also reserves an output data interface, and municipal administration personnel such as municipal administration, fire control and the like can access MCCIM through the interface to access the well lid state information of a certain area in the current system, so that early warning information can be issued in time, repair work can be guided, and road traffic safety and property safety of people can be guaranteed.
Compared with the traditional sensor detection, the system has the advantages that the required cost is reduced, the influence of unstable environment on the algorithm is overcome, the camera only needs to be maintained daily, and meanwhile, the equipment maintenance cost is reduced. The method and the device combine the computer vision technology and the machine learning technology to analyze the graph to obtain the well lid state, and have higher detection precision compared with the traditional sensor detection. According to the invention, the cameras with multiple visual angles are arranged to acquire images of the well lid area, on one hand, the multi-visual angle detection can obtain more abundant data, the anti-blocking property and robustness of the detection system are enhanced, and the well lid state detection precision is improved, on the other hand, the analysis results of the images with multiple visual angles can be utilized to correct the sample output result, so that the labeled sample required by the classification network is automatically generated, the complexity of manual labeling is avoided, the sample set is expanded by utilizing the corrected result and the images, the precision of the neural network model is improved, and the well lid state judgment result precision is further improved. The invention constructs a manhole cover urban information model MCCIM, and realizes real-time monitoring of the state of the manhole cover of the whole city by combining Web GIS and CIM technologies, thereby realizing intelligent and real-time centralized management of the state of the manhole cover. According to the invention, the traffic signal lamp near the road with the missing or shifted well cover is controlled by utilizing the well cover state obtained by judgment and utilizing the traffic signal lamp position recorded in the well cover urban information model and the control method, so that the safety of urban traffic is improved, and meanwhile, the intelligent auxiliary driving control method has high application value.
The above embodiments are merely preferred embodiments of the present invention, which should not be construed as limiting the present invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (5)
1. A CIM well lid state visual detection system based on deep learning and multiple visual angles is characterized in that the system realizes well lid state visual detection based on a well lid city information model MCCIM, the well lid city information model MCCIM comprises geographic position information of a well lid, a well lid state, well lid attribute information and geographic position information of nearby traffic lights, the system comprises a sensing unit, a perspective transformation unit, a well lid positioning unit, a well lid state judgment unit, a traffic light control unit and a visual management pipe unit, wherein,
the sensing unit is used for acquiring well lid area images by using cameras with different visual angles;
the perspective transformation unit is used for carrying out perspective transformation on the image shot by each camera based on the homography matrix and transforming the image into a top view;
the well lid positioning unit is used for positioning the position of the well lid by adopting a target detection cyclic convolution neural network based on deep learning;
the well lid state judging unit is used for judging the state of the well lid and comprises a pre-training module and a weight updating module:
the pre-training module is used for pre-training the classification neural network model according to the position of the well lid and the state mark of the well lid acquired by the well lid positioning unit;
the weight updating module is used for training the classification neural network model at a fixed time interval based on the expanded sample set and updating the network weight;
the traffic light control unit is used for determining the geographical position information of a nearby traffic signal light according to the geographical position information of the manhole cover in the manhole cover urban information model MCCIM when the manhole cover is displaced or lost, and sending a control signal to the traffic signal light to limit the traffic of a road direction in which the manhole cover is positioned;
and the visual management unit is used for displaying the well lid distribution and the well lid state of the city according to the well lid city information model MCCIM, so that the supervision of an administrator is facilitated.
2. The deep learning and multi-view based CIM manhole cover state visual inspection system of claim 1, wherein the target detection circular convolution neural network employs fast RCNN with anchor frame size set to 64, 128, 256.
3. The deep learning and multi-view based CIM well lid state visual inspection system of claim 1, wherein the well lid state determination unit further comprises: and the sample expansion module is used for correcting the labeling result based on the network output result of the well lid images with the multiple visual angles, and expanding the sample set by taking the corrected labeling result and the well lid images as new samples.
4. The deep learning and multi-view based CIM manhole cover state visual inspection system of claim 1, wherein the sample expansion module correcting the labeling result comprises: when most of the network output results of the multi-view images of the same well lid are the well lid missing, the well lid missing is determined to be a correct result, and the network output results of the few images with wrong judgment are modified to be missing and used as sample marking results; and when a small number of network output results of the multi-view images of the same well lid are well lid displacement, determining that the well lid displacement is a correct result, and modifying the sample output result of the image with the most wrong judgment into the displacement to serve as a sample marking result.
5. The deep learning and multi-view based CIM manhole cover state visual inspection system of claim 1, wherein the visualization management unit comprises:
the data acquisition module is used for acquiring the state of the well lid and transmitting the state to the well lid city information model MCCIM;
and the display module is used for rendering the manhole cover urban information model MCCIM by combining Web GIS and CIM technologies and displaying the manhole cover state and the control state of nearby traffic lights on a foreground Web page.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010109313.6A CN111339905B (en) | 2020-02-22 | 2020-02-22 | CIM well lid state visual detection system based on deep learning and multiple visual angles |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010109313.6A CN111339905B (en) | 2020-02-22 | 2020-02-22 | CIM well lid state visual detection system based on deep learning and multiple visual angles |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111339905A true CN111339905A (en) | 2020-06-26 |
| CN111339905B CN111339905B (en) | 2022-07-08 |
Family
ID=71185473
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010109313.6A Active CN111339905B (en) | 2020-02-22 | 2020-02-22 | CIM well lid state visual detection system based on deep learning and multiple visual angles |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111339905B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112040172A (en) * | 2020-07-17 | 2020-12-04 | 深圳奇迹智慧网络有限公司 | Well lid monitoring method and device, computer equipment and storage medium |
| CN112906611A (en) * | 2021-03-05 | 2021-06-04 | 新疆爱华盈通信息技术有限公司 | Well lid detection method and device, electronic equipment and storage medium |
| CN113137983A (en) * | 2021-04-30 | 2021-07-20 | 深圳市恒星物联科技有限公司 | Self-learning manhole cover posture monitoring method and monitoring system |
| CN113362284A (en) * | 2021-05-20 | 2021-09-07 | 同济大学 | Method for measuring difference of road well frame |
| WO2022002618A1 (en) * | 2020-06-29 | 2022-01-06 | Robert Bosch Gmbh | Road side unit and its method, computer storage medium and vehicle networking system |
| CN114266980A (en) * | 2022-03-03 | 2022-04-01 | 科大天工智能装备技术（天津）有限公司 | Urban well lid damage detection method and system |
| CN115268372A (en) * | 2021-07-23 | 2022-11-01 | 中国航空油料集团有限公司 | Control method and system for moving well lid |
| CN116665164A (en) * | 2023-04-28 | 2023-08-29 | 深圳云天励飞技术股份有限公司 | Manhole cover disease detection method, device and system, electronic equipment and storage medium |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103345815A (en) * | 2013-06-08 | 2013-10-09 | 清华大学 | Urban storm flood monitoring and traffic controlling and guiding system and method |
| CN103366246A (en) * | 2013-06-28 | 2013-10-23 | 中国南方电网有限责任公司 | Electrical geographic information service platform system |
| CN103854400A (en) * | 2014-02-24 | 2014-06-11 | 三维通信股份有限公司 | Well lid intelligent monitoring device |
| CN106056517A (en) * | 2016-06-06 | 2016-10-26 | 苏州朗米尔照明科技有限公司 | Smart city management system |
| CN106446807A (en) * | 2016-09-09 | 2017-02-22 | 成都通甲优博科技有限责任公司 | Well lid theft detection method |
| CN107786640A (en) * | 2017-09-30 | 2018-03-09 | 上海邦芯物联网科技有限公司 | A kind of city intelligent manhole cover system and management method |
| CN207215174U (en) * | 2017-08-16 | 2018-04-10 | 北京市政路桥管理养护集团有限公司 | Management system for monitoring based on intelligent well cover |
| CN108474866A (en) * | 2018-03-23 | 2018-08-31 | 深圳市锐明技术股份有限公司 | A kind of manhole cover loss detecting system and method based on deep learning |
| CN108960232A (en) * | 2018-06-08 | 2018-12-07 | Oppo广东移动通信有限公司 | Model training method, device, electronic equipment and computer readable storage medium |
| CN109555212A (en) * | 2018-12-10 | 2019-04-02 | 郭恩彤 | One kind is drained off floodwaters intelligent well cover control method for municipal administration |
| CN109961049A (en) * | 2019-03-27 | 2019-07-02 | 东南大学 | Cigarette brand recognition methods under a kind of complex scene |
| CN110399822A (en) * | 2019-07-17 | 2019-11-01 | 思百达物联网科技(北京)有限公司 | Action identification method of raising one's hand, device and storage medium based on deep learning |
| US20190392256A1 (en) * | 2019-07-31 | 2019-12-26 | Lg Electronics Inc. | Monitoring method and apparatus in the vehicle, and a 3d modeling unit for generating an object detection model therefor |
-
2020
- 2020-02-22 CN CN202010109313.6A patent/CN111339905B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103345815A (en) * | 2013-06-08 | 2013-10-09 | 清华大学 | Urban storm flood monitoring and traffic controlling and guiding system and method |
| CN103366246A (en) * | 2013-06-28 | 2013-10-23 | 中国南方电网有限责任公司 | Electrical geographic information service platform system |
| CN103854400A (en) * | 2014-02-24 | 2014-06-11 | 三维通信股份有限公司 | Well lid intelligent monitoring device |
| CN106056517A (en) * | 2016-06-06 | 2016-10-26 | 苏州朗米尔照明科技有限公司 | Smart city management system |
| CN106446807A (en) * | 2016-09-09 | 2017-02-22 | 成都通甲优博科技有限责任公司 | Well lid theft detection method |
| CN207215174U (en) * | 2017-08-16 | 2018-04-10 | 北京市政路桥管理养护集团有限公司 | Management system for monitoring based on intelligent well cover |
| CN107786640A (en) * | 2017-09-30 | 2018-03-09 | 上海邦芯物联网科技有限公司 | A kind of city intelligent manhole cover system and management method |
| CN108474866A (en) * | 2018-03-23 | 2018-08-31 | 深圳市锐明技术股份有限公司 | A kind of manhole cover loss detecting system and method based on deep learning |
| CN108960232A (en) * | 2018-06-08 | 2018-12-07 | Oppo广东移动通信有限公司 | Model training method, device, electronic equipment and computer readable storage medium |
| CN109555212A (en) * | 2018-12-10 | 2019-04-02 | 郭恩彤 | One kind is drained off floodwaters intelligent well cover control method for municipal administration |
| CN109961049A (en) * | 2019-03-27 | 2019-07-02 | 东南大学 | Cigarette brand recognition methods under a kind of complex scene |
| CN110399822A (en) * | 2019-07-17 | 2019-11-01 | 思百达物联网科技(北京)有限公司 | Action identification method of raising one's hand, device and storage medium based on deep learning |
| US20190392256A1 (en) * | 2019-07-31 | 2019-12-26 | Lg Electronics Inc. | Monitoring method and apparatus in the vehicle, and a 3d modeling unit for generating an object detection model therefor |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022002618A1 (en) * | 2020-06-29 | 2022-01-06 | Robert Bosch Gmbh | Road side unit and its method, computer storage medium and vehicle networking system |
| CN112040172A (en) * | 2020-07-17 | 2020-12-04 | 深圳奇迹智慧网络有限公司 | Well lid monitoring method and device, computer equipment and storage medium |
| CN112906611A (en) * | 2021-03-05 | 2021-06-04 | 新疆爱华盈通信息技术有限公司 | Well lid detection method and device, electronic equipment and storage medium |
| CN112906611B (en) * | 2021-03-05 | 2024-04-26 | 新疆爱华盈通信息技术有限公司 | Well lid detection method and device, electronic equipment and storage medium |
| CN113137983A (en) * | 2021-04-30 | 2021-07-20 | 深圳市恒星物联科技有限公司 | Self-learning manhole cover posture monitoring method and monitoring system |
| CN113137983B (en) * | 2021-04-30 | 2023-08-22 | 深圳市恒星物联科技有限公司 | Self-learning well lid posture monitoring method and monitoring system |
| CN113362284A (en) * | 2021-05-20 | 2021-09-07 | 同济大学 | Method for measuring difference of road well frame |
| CN115268372A (en) * | 2021-07-23 | 2022-11-01 | 中国航空油料集团有限公司 | Control method and system for moving well lid |
| CN114266980A (en) * | 2022-03-03 | 2022-04-01 | 科大天工智能装备技术（天津）有限公司 | Urban well lid damage detection method and system |
| CN116665164A (en) * | 2023-04-28 | 2023-08-29 | 深圳云天励飞技术股份有限公司 | Manhole cover disease detection method, device and system, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111339905B (en) | 2022-07-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111339905B (en) | CIM well lid state visual detection system based on deep learning and multiple visual angles | |
| CN108961790B (en) | Bad weather early warning management system and method based on four-dimensional live-action traffic simulation | |
| CN104766086B (en) | The monitoring and managing method and system of a kind of way mark | |
| CN109117749A (en) | A kind of abnormal object monitoring and managing method and system based on unmanned plane inspection image | |
| KR102024695B1 (en) | System and method for modeling surveillance camera layout | |
| WO2021007185A1 (en) | Using high definition maps for generating synthetic sensor data for autonomous vehicles | |
| CN109724610A (en) | A kind of method and device of full information real scene navigation | |
| CN108491758A (en) | A kind of track detection method and robot | |
| CN107430815A (en) | Method and system for automatic identification parking area | |
| JP2019196680A (en) | Pavement information gathering inspection system, pavement information gathering inspection method, and program | |
| CN103700261A (en) | Video-based road traffic flow feature parameter monitoring and traffic comprehensive information service system | |
| KR102167292B1 (en) | Apparatus and method for providing road status information | |
| Song et al. | Real-time monitoring for crowd counting using video surveillance and GIS | |
| WO2021199915A1 (en) | Degradation display system, degradation display method, and storage medium | |
| CN107749055A (en) | A kind of fault detection method, system and the device of LED traffic guidances screen | |
| CN111881970A (en) | Intelligent outer broken image identification method based on deep learning | |
| CN111076096A (en) | Gas pipe network leakage identification method and device | |
| CN110427441A (en) | A kind of railway external environment perils detecting and management method based on space-air-ground integration technology | |
| CN103149603B (en) | Road weather detection method based on video | |
| CN113569956A (en) | Mountain fire disaster investigation and identification method based on AI algorithm | |
| CN116258980A (en) | Unmanned aerial vehicle distributed photovoltaic power station inspection method based on vision | |
| CN114419231A (en) | Traffic facility vector identification, extraction and analysis system based on point cloud data and AI technology | |
| CN114495421B (en) | Intelligent open type road construction operation monitoring and early warning method and system | |
| CN116993681A (en) | Substation inspection defect detection method and system | |
| KR100703818B1 (en) | Traffic management system using detect devices |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20240305 Address after: 230000 floor 1, building 2, phase I, e-commerce Park, Jinggang Road, Shushan Economic Development Zone, Hefei City, Anhui Province Patentee after: Dragon totem Technology (Hefei) Co.,Ltd. Country or region after: China Address before: No.56 Pengcheng Avenue, Zhengdong New District, Zhengzhou City, Henan Province Patentee before: ZHENGZHOU RAILWAY VOCATIONAL & TECHNICAL College Country or region before: China |
|
| TR01 | Transfer of patent right |