Disclosure of Invention
The invention provides a CIM well lid state visual detection system based on deep learning and multiple visual angles, which realizes well lid state detection, reduces the cost required by the system and improves the accuracy of well lid state judgment.
A CIM well lid state visual detection system based on deep learning and multi-view angle realizes well lid state visual detection based on a well lid city information model MCCIM, wherein the well lid city information model MCCIM comprises geographic position information of a well lid, a well lid state, well lid attribute information and geographic position information of nearby traffic lights, the system comprises a sensing unit, a perspective transformation unit, a well lid positioning unit, a well lid state judgment unit, a traffic light control unit and a visual management pipe unit, wherein,
the sensing unit is used for acquiring well lid area images by using cameras with different visual angles;
the perspective transformation unit is used for carrying out perspective transformation on the image shot by each camera based on the homography matrix and transforming the image into a top view;
the well lid positioning unit is used for positioning the position of the well lid by adopting a target detection cyclic convolution neural network based on deep learning;
the well lid state judging unit is used for judging the state of the well lid and comprises a pre-training module and a weight updating module:
the pre-training module is used for pre-training the classification neural network model according to the position of the well lid and the state mark of the well lid acquired by the well lid positioning unit;
the weight updating module is used for training the classification neural network model at a fixed time interval based on the expanded sample set and updating the network weight;
the traffic light control unit is used for determining the geographical position information of a nearby traffic signal light according to the geographical position information of the manhole cover in the manhole cover urban information model MCCIM when the manhole cover is displaced or lost, and sending a control signal to the traffic signal light to limit the traffic of a road direction in which the manhole cover is positioned;
and the visual management unit is used for displaying the well lid distribution and the well lid state of the city according to the well lid city information model MCCIM, so that the supervision of an administrator is facilitated.
The target detection circular convolution neural network adopts fast RCNN, and the size of an anchor frame is set to be 64, 128 and 256.
The well lid state judgment unit further comprises: and the sample expansion module is used for correcting the labeling result based on the network output result of the well lid images with the multiple visual angles, and expanding the sample set by taking the corrected labeling result and the well lid images as new samples.
The step of correcting the labeling result by the sample expansion module comprises the following steps: when most of the network output results of the multi-view images of the same well lid are the well lid missing, the well lid missing is determined to be a correct result, and the network output results of the few images with wrong judgment are modified to be missing and used as sample marking results; and when a small number of network output results of the multi-view images of the same well lid are well lid displacement, determining that the well lid displacement is a correct result, and modifying the sample output result of the image with the most wrong judgment into the displacement to serve as a sample marking result.
The visualization management unit includes:
the data acquisition module is used for acquiring the state of the well lid and transmitting the state to the well lid city information model MCCIM;
and the display module is used for rendering the manhole cover urban information model MCCIM by combining Web GIS and CIM technologies and displaying the manhole cover state and the control state of nearby traffic lights on a foreground Web page.
The invention has the beneficial effects that:
1. compared with the traditional sensor detection, the system has the advantages that the cost required by the system is reduced, the camera is only required to be maintained daily, and meanwhile, the equipment maintenance cost is reduced.
2. The method and the device combine the computer vision technology and the machine learning technology to analyze the graph to obtain the well lid state, and have higher detection precision compared with the traditional sensor detection.
3. According to the invention, the cameras with multiple visual angles are arranged to acquire images of the well lid area, on one hand, the multi-visual angle detection can obtain more abundant data, the anti-blocking property and robustness of the detection system are enhanced, and the well lid state detection precision is improved, on the other hand, the analysis results of the images with multiple visual angles can be utilized to correct the sample output result, so that the labeled sample required by the classification network is automatically generated, the complexity of manual labeling is avoided, the sample set is expanded by utilizing the corrected result and the images, the precision of the neural network model is improved, and the well lid state judgment result precision is further improved.
4. The invention constructs a manhole cover urban information model MCCIM, and realizes real-time monitoring of the state of the manhole cover of the whole city by combining Web GIS and CIM technologies, thereby realizing intelligent and real-time centralized management of the state of the manhole cover.
5. According to the invention, the traffic signal lamp near the road with the missing or shifted well cover is controlled by utilizing the well cover state obtained by judgment and utilizing the traffic signal lamp position recorded in the well cover urban information model and the control method, so that the safety of urban traffic is improved, and meanwhile, the intelligent auxiliary driving control method has high application value.
The first embodiment is as follows:
a visual inspection system for the well lid state based on deep learning and multi-view is based on a well lid city information model MCCIM to realize visual inspection of the well lid state. Cim (city Information modeling), a model for effectively organizing a large amount of city Information, is a further upgrade of BIM technology, and can present various Information states of a city in real time by combining with GIS technology. The manhole Cover City Information model MCCIM (manhole Cover City Information modeling) comprises: well lid geographical position information, well lid state, well lid attribute information, nearby traffic signal lamp geographical position information. The well lid attribute information includes a well lid ID, a well lid type, a well lid brand, a well lid service life, well lid damage information, and the like. The system comprises a sensing unit, a well lid positioning unit, a well lid state judging unit, a traffic light control unit and a visual management unit.
The arrangement of the camera is important in order to obtain an image that can show the state of the well lid. The camera that is the visual angle of looking down that sets up to be located directly over the well lid can obtain the best image that presents the well lid state undoubtedly. However, the manhole cover is usually located on the road, and therefore, it is almost impossible to provide a camera directly above the manhole cover based on city planning considerations. And the camera with a single visual angle is difficult to accurately sense the well lid displacement state. In addition, street views where a plurality of well covers on the roadside are located are relatively complex, and a single camera may be shielded by a target object, so that detection failure is caused. Therefore, the invention adopts the cameras with different visual angles to collect the well lid area images. In a preferred embodiment, the cameras with different viewing angles can cover 360 degrees of environmental information around the manhole cover. In order to effectively utilize resources, the camera can utilize cameras for monitoring surrounding shops and roads, and an additional camera is arranged according to requirements on the basis so as to comprehensively cover the environment around the well cover. According to the invention, the well lid area images acquired by the cameras with different visual angles are utilized, and the perception information of the cameras at each angle is fused by a specific method based on a multi-view geometric method, so that the well lid area images have the characteristics of mutual correlation. By the multi-view geometry-based method, even if a certain camera is blocked or stops working, the system can still normally operate, and the situation of detection failure cannot occur. Meanwhile, the method based on the multi-view geometry can also automatically generate classification samples, expand sample sets and avoid manual labeling.
And the perspective transformation unit is used for carrying out perspective transformation on the image shot by each camera based on the homography matrix and transforming the image into a top view mode.
Because the cameras in the system may not face the manhole cover area, in order to ensure the uniformity of the images, the invention needs to perform perspective transformation on the images shot by each camera, and the images can be known according to a coordinate transformation formula of a homography matrix:
wherein (u, v) is the pixel position in the image shot by the camera, and (u ', v') is the pixel position after perspective transformation,
is a homography matrix representing the transformation relationship between the camera coordinate system and the top view mode. One camera corresponds to one homography matrix. The homography matrix can be obtained according to the mapping relation between the camera coordinate system and the camera pose parameters.
The transformed image is based on a top view of the camera plane, which contains a large number of street view elements, such as trees, sky, pedestrians, etc. The well lid area needs to be positioned in a complex street view, and the influence of the complex street view on the well lid state judgment is removed. Therefore, the present invention includes a well lid positioning unit.
And the well lid positioning unit is used for positioning the position of the well lid by adopting a target detection cyclic convolution neural network based on deep learning.
Specifically, the invention adopts fast RCNN to position the well cover area. Practitioners will appreciate that there are many similar target detection algorithms, such as SSD, YOLO, etc. The implementer may autonomously select the target detection algorithm based on speed and accuracy requirements.
In the original fast RCNN frame, the size of the anchor frame anchor is initially set to 128, 256, 512, and the well lid area occupies a small ratio in the whole picture, so in order to detect a small target and simultaneously meet the requirement of algorithm speed, the invention adds an anchor with the size of 64, and removes an anchor with the size of 512, namely, the size of the anchor is finally changed to 64, 128, 256. The modified anchor is more fit with the real data characteristic distribution, and the recall rate and the accuracy rate after the experiment improvement are higher than those of the original version.
Based on well lid positioning unit, can succeed in pinpointing well lid position. Further, the current state of the well lid needs to be classified according to the obtained position of the well lid.
The well lid state judging unit is used for judging the state of the well lid and comprises a pre-training module and a weight updating module.
And the pre-training module is used for pre-training the classification neural network model according to the position of the well lid obtained by the well lid positioning unit and the state mark of the well lid.
The pre-training classification neural network is obtained by means of training of a classification framework VGG 16. Firstly, a convolutional neural network model VGG16 is constructed, wherein the VGG16 network model comprises 13 convolutional layers, 5 pooling layers and 3 full-connection layers. The output of the original VGG16 last fully connected layer is 1000, representing a 1000-class item in the forecast imagenet. The invention changes the output of the last full-connection layer into 3 and outputs two main results of OK and NG, wherein OK represents that the well lid state is normal, and NG comprises two judgment results of shift or missing and can be respectively recorded as NG1 and NG 2. The invention trains based on the following loss functions:
wherein i is an index corresponding to the training sample in the training set, and y
iTo predict value, t
iIs the target value. In the implementation process, firstly, a well lid image sample is collected, the sample is labeled, a sample set and the label of the sample set are used as input, and a classification neural network model is trained to obtain an initial neural network.
Since the pre-training network model used at first cannot be applied to various working conditions, inconsistency inevitably occurs between the detection results of each viewing angle, and here an implementer should know that the precision of the pre-training model is directly related to the availability of the system of the present invention, and it should be ensured that the outputs of a plurality of viewing angles are mostly correct as much as possible, for example, the results observed and detected by a camera with 4 viewing angles may be three correct and one wrong, and worse cases only account for a few in a day, otherwise, the reliability of automatic labeling may be affected.
According to the result detected by the sensing information of the cameras, a plurality of manhole cover area images can be obtained at the same time. The invention carries out network training and well lid state judgment based on the camera images with multiple visual angles, and can correct the sample output result based on the network output result of the well lid images with multiple visual angles. Therefore, in order to improve the prediction effect of the model, the invention adopts a mode of automatically generating the labeled sample based on multi-visual sense to expand the sample set.
And the sample expansion module is used for correcting the labeling result based on the network output result of the well lid images with the multiple visual angles, and expanding the sample set by taking the corrected labeling result and the well lid images as new samples.
Assuming that the current well lid state is a missing state, because the corresponding characteristics are obvious, the cameras at multiple viewing angles can correctly judge the current well lid state according to the pre-training model. Meanwhile, a few cameras can judge the current well lid state wrongly. At this time, the current frame image collected by a few cameras with wrong judgment is marked as 'missing', and the image and the corresponding marking information are added into the sample set.
Assuming that the current well lid state is a shift state, the cameras at multiple viewing angles can misjudge the current well lid state due to the fact that the corresponding features of the current well lid state are fuzzy. Meanwhile, a few cameras can correctly judge the state of the well lid. At this time, most of the images collected by the cameras with wrong judgment are marked as 'displacement', and the images and the corresponding marking information are also added into the sample set.
Along with the lapse of time, prediction every time will produce the sample with label automatically to this carries out the sample set and expands, has both guaranteed the demand of model to sample quantity like this, has also saved a large amount of manual labeling, has practiced thrift a large amount of marking costs.
And the weight updating module is used for training the classification neural network model at a fixed time interval based on the expanded sample set and updating the network weight.
The sample set update timing may be implementation dependent. The implementer may take samples over a period of time, which may be determined at the discretion of the implementer based on the frequency of model updates. And each time the classification model is updated, the classification model is divided into a training set, a verification set and a test set according to the proportion according to the collected samples. In one embodiment, 20000 samples are collected, and the samples are processed according to the ratio of 8: 1: 1, divided into training set, verification set and test set.
The method adopts a random gradient descent method to optimize the parameters of the classification neural network model, the target error is 0.00001, the initial learning rate is 0.001, and in order to prevent the model from approaching a local minimum value, the learning rate needs to be decreased progressively according to a mode of reducing 10 times per 100 iterations. And the implementer observes the network convergence condition in time and the prediction accuracy on the verification set, and stops training at a proper node. At intervals, an implementer can train samples accumulated in the previous time once, update model parameters and improve the classification accuracy.
The invention does not require the specific time of updating, and an implementer can select a proper time interval updating model according to the actual situation and the calculation speed of the adopted hardware platform.
Further, the implementer can write a script for automatic training without manually observing and judging convergence conditions, namely setting an appropriate early-stopping strategy, and the specific method of early-stopping is not included in the protection content of the invention because the specific method of early-stopping is various and well-known.
The implementer should know that the classification algorithm can also be implemented by using networks such as mobilenet, alexnet and the like, and the implementer can flexibly select the classification algorithm according to the requirements of speed and precision.
The well lid is lost and shifts, can cause very big potential safety hazard to pedestrian and the vehicle that traveles on the road. On the one hand, can integrate pronunciation early warning module in the camera, when the well lid is lost and shift, remind vehicle and pedestrian to notice the detour through pronunciation. However, for the manhole cover on the road, since the driving environment is noisy, it is not enough to remind only by voice. Therefore, when the well lid is displaced or lost, the traffic light signal nearby should be controlled by combining the well lid city information model. The invention is provided with a traffic light control unit for controlling traffic signal lights near the manhole cover.
And the traffic light control unit is used for determining the geographical position information of a nearby traffic signal light according to the geographical position information of the well cover in the well cover urban information model MCCIM when the well cover is displaced or lost, and sending a control signal to the traffic signal light to limit the traffic of the road direction where the well cover is positioned.
In the implementation process, the traffic signal lamp can be controlled according to the positions of the traffic signal lamp and the manhole cover. In particular, in the manhole cover city information model MCCIM, it is possible to record for each manhole cover the associated traffic light position and how the traffic light should react when the manhole cover state is abnormal. For example, if the manhole cover is located near an intersection, its associated traffic lights include at least four, the location of each traffic light is recorded, and the emergency response that each traffic light should make is recorded. For example, if the position of the manhole cover is located on the left side of the traffic signal lamp, the left-turn traffic is limited, namely the left-turn signal lamp is red.
In order to visually present the well lid state of the current area, enable a user to visually acquire the well lid state of the current area, enable an administrator to monitor the well lid state more visually, and provide early warning information and road traffic safety guarantee, the MCCIM information model is displayed on the Web through a Web GIS technology in combination with a WebGIS visualization technology.
Visual management unit for the well lid state and the well lid distribution in display area, the administrator of being convenient for supervises, visual management unit includes:
the data acquisition module is used for acquiring the state of the well lid and transmitting the state to the well lid city information model MCCIM;
and the display module is used for rendering the manhole cover urban information model MCCIM by combining the Web GIS and CIM technologies and displaying the manhole cover state and the control state of nearby traffic lights on a foreground Web page.
Meanwhile, the system also reserves an output data interface, and municipal administration personnel such as municipal administration, fire control and the like can access MCCIM through the interface to access the well lid state information of a certain area in the current system, so that early warning information can be issued in time, repair work can be guided, and road traffic safety and property safety of people can be guaranteed.
Compared with the traditional sensor detection, the system has the advantages that the required cost is reduced, the influence of unstable environment on the algorithm is overcome, the camera only needs to be maintained daily, and meanwhile, the equipment maintenance cost is reduced. The method and the device combine the computer vision technology and the machine learning technology to analyze the graph to obtain the well lid state, and have higher detection precision compared with the traditional sensor detection. According to the invention, the cameras with multiple visual angles are arranged to acquire images of the well lid area, on one hand, the multi-visual angle detection can obtain more abundant data, the anti-blocking property and robustness of the detection system are enhanced, and the well lid state detection precision is improved, on the other hand, the analysis results of the images with multiple visual angles can be utilized to correct the sample output result, so that the labeled sample required by the classification network is automatically generated, the complexity of manual labeling is avoided, the sample set is expanded by utilizing the corrected result and the images, the precision of the neural network model is improved, and the well lid state judgment result precision is further improved. The invention constructs a manhole cover urban information model MCCIM, and realizes real-time monitoring of the state of the manhole cover of the whole city by combining Web GIS and CIM technologies, thereby realizing intelligent and real-time centralized management of the state of the manhole cover. According to the invention, the traffic signal lamp near the road with the missing or shifted well cover is controlled by utilizing the well cover state obtained by judgment and utilizing the traffic signal lamp position recorded in the well cover urban information model and the control method, so that the safety of urban traffic is improved, and meanwhile, the intelligent auxiliary driving control method has high application value.
The above embodiments are merely preferred embodiments of the present invention, which should not be construed as limiting the present invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention should be included in the scope of the present invention.