CN114973067A - Earth pressure balance shield muck information extraction method based on deep multi-task learning - Google Patents
Earth pressure balance shield muck information extraction method based on deep multi-task learning Download PDFInfo
- Publication number
- CN114973067A CN114973067A CN202210484070.3A CN202210484070A CN114973067A CN 114973067 A CN114973067 A CN 114973067A CN 202210484070 A CN202210484070 A CN 202210484070A CN 114973067 A CN114973067 A CN 114973067A
- Authority
- CN
- China
- Prior art keywords
- muck
- task
- network
- image
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/55—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The invention belongs to the field of earth pressure balance shield method tunnel construction, and relates to an earth pressure balance shield muck information extraction method based on deep multitask learning. According to the method, important construction information in the earth pressure balance shield tunneling process is extracted according to the muck image, so that the information to be extracted needs to be determined based on construction requirements, and the types of all tasks are determined; collecting muck data, selecting and processing the data to obtain a multi-task recognition data set of muck with balanced soil pressure; then, constructing and training according to the identification task to be completed to obtain a muck multi-task identification model; and finally, inputting a muck monitoring video during the construction of the earth pressure balance shield to realize the real-time identification of muck and obtain various construction information required by constructors. The method has the characteristics of accuracy, rapidness, strong generalization, high speed, low cost and convenient application.
Description
Technical Field
The invention belongs to the field of earth pressure balance shield method tunnel construction, and relates to an earth pressure balance shield muck information extraction method based on deep multitask learning.
Background
With the continuous acceleration of the urbanization process in China, the mileage and scale of the subway are rapidly expanded, which brings great challenges to the safety of tunnel construction and the control of the influence on the surrounding environment. The shield method is widely applied to the construction of subway tunnels due to the characteristics of high construction speed, small influence on the surrounding environment and high degree of mechanization. During the shield tunneling process, the discharged muck is a hand of data capable of reflecting the tunneling stratum condition and the construction state, and the muck contains abundant construction information, so that the construction risk can be greatly reduced if the muck can be fully excavated, the construction quality is greatly improved, and the construction safety is more effectively ensured. At present, during shield construction, excavation of construction information contained in the slag soil is very limited, and constructors only occasionally check slag soil videos to ensure smooth slag discharge, so that the slag soil is not really utilized as an important construction information feedback source. Important information concerned by a plurality of constructors can be extracted from the muck image through a computer vision technology, such as soil property types in front of an excavation surface, so that the constructors can be helped to master stratum changes in time; guiding constructors to adjust the slag soil improvement strategy and the dosage under the condition of improving the slag soil; judge the state of slagging tap, can in time early warning when slagging tap is unusual, remind constructor in time to handle the problem in order to avoid developing bigger accident and so on. Therefore, the method for extracting the various muck information in real time, accurately and synchronously by using the muck image can provide a large amount of useful construction feedback information for constructors to assist the constructors in adjusting construction decisions and ensure the safe and stable tunneling of the tunnel.
The method mainly comprises the following steps of aiming at the problem of extracting the muck information of the shield tunneling machine at present:
the Chinese patent application with the application number of 202110558201.3 provides a real-time prediction system and a real-time prediction method for the soil quality of an excavation surface in shield tunnel construction, wherein the prediction system comprises a similar engineering data acquisition module, a soil quality information processing module, a construction data processing module, a soil quality predictor construction module and a soil quality prediction module. The soil property predictor construction module adopts a convolutional neural network to construct a first soil property predictor and a second soil property predictor based on a soil property characteristic gray-scale image and a muck image which are obtained by construction data, and combines prediction results of the two predictors to obtain a soil property real-time detection result.
Chinese patent application with application number 202111008790.4 provides an automatic detection method and system for muck fluidity plasticity based on deep learning. The method comprises the steps of classifying the muck fluidity plasticity to establish training sets of different types, constructing a deep convolutional neural network model by taking a Resnet network as a basic network, inputting the training sets into an input layer of the model, updating the model through a series of array transformation, and performing muck fluidity plasticity classification training on the model to obtain an automatic detection model for identifying the muck fluidity plasticity category.
The Chinese patent application with the application number of 201910445028.9 provides an automatic improvement method for earth pressure balance shield tunnel excavation muck, the shape of the muck is judged at the tail of a screw conveyor through a non-contact detection method, the fluidity grade is judged, the judgment result is transmitted to a muck improvement control system on a shield machine operation platform through a PLC program, and improvement parameters matched with the fluidity grade are selected to improve the muck fluidity in real time.
Disadvantages of the prior art
The existing muck information extraction method is to establish a muck image recognition model for a certain task, such as soil quality type recognition, improved state judgment and the like, and each model can only complete information extraction related to the certain task to realize a single information acquisition target. In practical application, if it is desired to obtain multiple pieces of information from the muck image, a model needs to be deployed for each muck identification task. The deep learning algorithm has high requirements on hardware devices such as a GPU (graphics processing unit) and the like, and with the increase of recognition tasks, the simultaneous image recognition of a plurality of models brings huge device consumption, which needs a large amount of cost investment. Secondly, the synchronous identification of a plurality of models can reduce the speed of image identification, real-time multi-task information acquisition cannot be really realized, and further, the construction decision cannot be dynamically adjusted in real time along with the shield tunneling process. For some tasks, the data set has the problems of small sample number or unbalanced category, for example, the image sample with abnormal slag tapping is far less than the image sample with normal slag tapping, and the trained model has the problems of low accuracy and weak generalization at the moment.
Disclosure of Invention
Aiming at the defects of the prior art, the invention designs an earth pressure balance shield muck information extraction method based on deep multitask learning, which fully excavates construction information contained in muck and aims to solve the problem that a shield driver is not sufficient in understanding soil quality information and shield construction conditions. The method can accurately and synchronously extract various types of information contained in the muck image in the shield machine in real time, and provides powerful basis for subsequent adjustment of construction parameters, muck improvement schemes and other construction decisions of a shield driver.
The technical scheme of the invention is as follows:
the method is used for carrying out image recognition on muck based on a deep multitask learning algorithm and extracting multi-type construction information contained in muck images.
Furthermore, the method extracts important construction information in the earth pressure balance shield tunneling process according to the muck image, so that the information to be extracted needs to be determined based on construction requirements, and the types of all tasks are determined. And then collecting muck data, and selecting and processing the data to obtain a multi-task recognition data set for balancing the soil pressure to obtain muck. And then constructing and training according to the recognition tasks to be completed to obtain the muck multitask recognition model. And finally, inputting a muck monitoring video during the construction of the earth pressure balance shield to realize the real-time identification of muck and obtain various construction information required by constructors.
Further, the earth pressure balance shield muck information extraction method based on deep multitask learning comprises the following steps:
step 1, determining a residue soil image information extraction task;
step 2, establishing a multitask muck image database;
and 3, constructing a multi-task muck identification model.
Further, the step 1 comprises the following steps:
1.1 obtaining construction information from the muck image, wherein the construction information comprises the front soil quality type, muck improvement condition and whether slag discharge abnormity occurs.
1.2 determining a muck identification task based on the information in the step 1.1, and determining the types of the tasks; the task types include: image classification, target detection, semantic segmentation and instance segmentation.
1.3 determining the labeling rules of the task identified in step 1.2.
Further, the step 2 comprises the following steps:
2.1 collecting a muck monitoring video of an application area; the muck monitoring camera is arranged right above or obliquely above the belt conveyor and is aligned with the muck outlet of the spiral muck discharging machine, so that the complete muck discharging process and muck form are guaranteed to be shot. And a lighting device is arranged above the soil outlet.
2.2 the data is preprocessed after the video data is collected. And intercepting the effective video of the stable unearthing stage.
2.3 decimate the video frames at a frequency of one per 30 frames.
2.4 image annotation for different types of recognition tasks.
The method comprises an image classification task, wherein image categories are marked in a mode of naming image classifications or dividing folders; the method comprises a target detection task, wherein the target detection task is used for manually marking the muck image, framing a target to be identified in each image by using a rectangular frame, and inputting the category of each target frame; the method comprises semantic segmentation and instance segmentation tasks, wherein the tasks are used for manually marking the muck images, marking the boundary of a target to be segmented in each image and inputting the category of each region. Therefore, each image must be labeled, and the information recorded therein is called the true value (GT), which is provided for the 3.3 training process.
And 2.5, the obtained muck picture file and the corresponding task label files are data sets for muck multi-task identification, and a training set, a verification set and a test set are divided.
Further, the step 3 includes the following steps:
step 3, constructing a multi-task muck identification model
3.1 custom data set
3.1.1 uploading the image data of the dregs and the label information of the corresponding tasks from the data set storage path, and respectively storing the image data of the dregs and the label information of the tasks in a list.
3.1.2 processing and converting the image data and the label data.
The image data is converted from PIL format to tensor format in the shape of (C, H, W), where C represents the number of image channels, H represents the picture height, W represents the picture width, and all pixel values are normalized to between 0-1 by dividing 255.
3.1.3 setting random image enhancement operation on the image before inputting into the network.
3.1.4, calculating the mean value and the standard deviation of the three-channel pixel values of the picture in the whole data set, carrying out standardization processing on the picture, and providing the picture for the step 3.3.
3.2, building a multitask network, wherein the network is divided into two parts: the multitasking shared network and each task specific network are provided to step 3.3.
3.3 construction of the training Process
Firstly, a user-defined data set and a multi-task network are imported, and a data loader is constructed in a mode of inputting user-defined pictures and tag data into a network.
The loss function of the network is designed as shown in equation 1.
During network training, hyper-parameters need to be set, including an initial learning rate value and a change mode of the initial learning rate value along with the increase of the number of training cycles, parameters of an optimizer and the optimizer, the batch size of input pictures, the number of training cycles and the weight of each task loss item of the most important loss function in the method.
The training can be started after the loss function and the hyperparameter are set. Each group of image input network obtains a classification or regression result of each task, and the predicted value of each task and the real value (GT) in the label provided by step 2.4 corresponding to each task are input into the total loss function, so that the current total loss value, that is, the distance between the predicted value and the real value, can be calculated. And calculating the derivatives of the total loss value to all the network parameters, and optimizing and updating the network parameters by using an optimizer, namely a round of training iteration. When all pictures in the training set are input into the network for training in turn, a training cycle is obtained. After finishing a training cycle, inputting the pictures of the verification set into the network in turn to calculate the network precision, and observing the network training condition according to the network precision and using the network training condition as a basis for adjusting the hyper-parameters in the next network training. And when the loss is reduced and converged to a certain stable value, finishing the training, and storing the trained network parameters to obtain the multi-task muck recognition model.
Loss＝α 1 Loss 1 +α 2 Loss 2 +α 3 Loss 3 +…+αnLoss n (1)
3.4 model testing and evaluating indexes;
and loading the trained model and parameters, and inputting the muck picture to be tested to obtain the recognition result of each task.
3.5 visualization of residue soil recognition result
And visualizing the recognition result obtained by each task output end in the network on the picture according to the requirement.
Advantageous effects
1. The invention provides a method for acquiring construction information of an earth pressure balance shield, which is to acquire various types of construction information by identifying muck images based on deep learning. The method can be used for model training and real-time identification during application by utilizing the monitoring video data, and has the characteristics of accuracy, rapidness and convenience in application.
2. The invention provides a method capable of simultaneously completing a plurality of muck information extraction tasks, namely, a multi-task muck identification model consisting of a task sharing network and a task specific network is designed and trained based on the idea of multi-task learning. The method can obtain the recognition results of a plurality of tasks through the joint training of a network, and has the characteristics of strong generalization, high speed and low cost.
THE ADVANTAGES OF THE PRESENT INVENTION
1. The method for extracting the muck information can realize synchronous extraction of various types of information in the muck image by establishing a multi-task learning network, saves a large amount of computational power consumption and memory occupation compared with the method for establishing an image recognition model for each task, and has remarkable advantages compared with the method for simultaneously reasoning by a plurality of recognition models.
2. The multi-task learning network established by the invention consists of a task sharing layer and a task specific layer, and as the tasks are jointly trained, the characteristics of mutual sharing, complementation and extraction enhancement can be realized in the task sharing layer, and the positive mutual influence can ensure that the generalization and the accuracy of the recognition of each task are better than those of the learning of a single task.
3. The method provided by the invention only needs to utilize the original muck monitoring video data on site in the data collection and practical application processes, and does not need to additionally install a sensor or other equipment. The self-reasoning speed of the adopted algorithm is higher than the frame rate of the monitoring video, so the method has the characteristics of real time and low cost.
Drawings
FIG. 1 illustrates the differences between multi-task learning and single-task learning.
FIG. 2 is a flow chart of the method of the present invention.
FIG. 3 is a process for determining the residue image information extraction task according to the method of the present invention
FIG. 4 is a process for establishing a multitasking muck image database by the method of the invention
FIG. 5 is a schematic diagram of the image data acquisition of the residue soil in this embodiment
FIG. 6 is a process for constructing a multi-task muck recognition model by the method of the invention
Fig. 7 is a schematic diagram of a multitasking network structure according to an embodiment.
Detailed Description
The core of the invention is to develop a muck image information extraction method which can accurately and synchronously complete a plurality of tasks in real time based on the muck image shot by a monitoring camera of a shield machine. By using the method, the realization of a plurality of information extraction tasks can be integrated in a convolutional neural network, and a model obtained after the network is trained can realize the muck identification target of a plurality of tasks, so that the multi-aspect information of the shield muck is obtained.
Fig. 1 shows the difference between the single-task learning and the multi-task learning, and fig. 1(a) shows that a model needs to be established for each task by using the conventional single-task learning method, whereas the multi-task learning method shown in fig. 1(b) can complete a plurality of tasks by establishing only one model. The method is characterized in that: the tunnel is multi-task, low in cost, automatic and suitable for earth pressure balance shield tunneling in various earth strata.
The technical scheme of the detailed embodiment is given below.
The overall procedure of the method of the inventionAs shown in fig. 2：
A method for extracting residue soil information of a soil pressure balance shield is characterized in that image recognition is carried out on residue soil based on a deep multitask learning algorithm, and multi-type construction information contained in a residue soil image is extracted.
According to the method, important construction information in the earth pressure balance shield tunneling process is extracted according to the muck image, so that the information to be extracted needs to be determined based on construction requirements, and the types of all tasks are determined. And then collecting muck data, and selecting and processing the data to obtain a multi-task recognition data set for balancing the soil pressure to obtain muck. And then constructing and training according to the recognition tasks to be completed to obtain the muck multitask recognition model. And finally, inputting a muck monitoring video during the construction of the earth pressure balance shield to realize the real-time identification of muck and obtain various construction information required by constructors.
Step 1, determining a residue soil image information extraction task
The determination flow of the muck image information extraction task is shown in fig. 3.
1.1 firstly, determining construction information needing to be obtained from the muck image according to the specific conditions of the tunnel engineering project, such as the front soil quality type, the muck improvement condition, whether abnormal slag discharge occurs and the like.
1.2 determining the muck recognition task based on the information to be extracted, and determining the types of the tasks. The field of image recognition includes four task types: image classification, target detection, semantic segmentation and instance segmentation.
1.3 further determining the labeling rules of each task, such as dividing the image classification task into several classes and how to divide the classes; how the target detection task and the segmentation task divide the target category, how the calibration target is selected, how the target calibration range is defined, and the like.
Step 2, establishing a multi-task muck image database
Establishment process of multitask muck image recognition databaseAs shown in fig. 4。
2.1 the method is applied to the area of the residue soil monitoring video is firstly collected widely, the resolution of the monitoring video is 1920 x 1080, and preferably not less than 1280 x 720. The image data acquisition schematic diagram of the slag soil is shown in fig. 5, and the slag soil monitoring camera is arranged right above or obliquely above the belt conveyor and is aligned with the soil outlet of the spiral soil discharging machine, so that the complete slag discharging process and the slag soil form are ensured to be shot. And the lighting equipment is arranged above the soil outlet, and the surface of the lens is kept clean, so that the shot image is bright and clear and is easy to identify.
2.2 the data is preprocessed after the video data is collected. The shield tunnel construction is carried out through two steps of continuous and cyclic excavation and tunneling and lining installation, and dregs are continuously discharged through a screw conveyor in the excavation and tunneling process. Therefore, the effective video in the stable unearthing stage is firstly intercepted, and the available video data is sorted for further processing.
2.3 extracting video frames at the frequency of one piece per 30 frames (the specific frame extracting frequency can be adjusted according to the advancing speed of the shield tunneling machine). And selecting images according to task requirements, selecting images with various muck forms and image backgrounds as much as possible when soil types need to be identified, and removing images shot under lens fouling and mosaic bad images and redundant images caused by video blockage.
2.4 the image classification task may tag image categories by image classification naming or by dividing folders. The target detection task can manually label the muck image by adopting image labeling software such as LabelImg, frame out a target to be identified in each image by using a rectangular frame, and input the category of each target frame. The semantic segmentation and instance segmentation tasks can be realized by manually labeling the muck images by adopting image labeling software such as Labelme, marking the boundary of a target to be segmented in each image, and inputting the category of each region. In the training and validation sets, each image must be tagged, and the information recorded therein is called the true value (GT), which is provided for the 3.3 training process.
2.5 the final acquired muck image file and corresponding task label files are a muck multitask identification data set, and the data set is divided into 8 parts: 1: 1 into a training set, a verification set and a test set, and paying attention to the distribution uniformity of different types of samples in each task on the training set, the verification set and the test set, especially aiming at the tasks with serious unbalanced sample types.
Step 3, constructing a multi-task muck identification model
Construction process of multi-task muck identification modelAs shown in fig. 6And also the constituent parts of the code and the writing order. The invention adopts a convolutional neural network to establish a multi-task muck recognition model, and the whole convolutional neural network comprises a multi-task sharing part and specific parts of each task. The model is built on a deep learning frame Pyorch and is realized by adopting a python language.
3.1 in custom data set Module
3.1.1 uploading the image data of the dregs and the label information of the corresponding tasks from the data set storage path, and respectively storing the image data of the dregs and the label information of the tasks in a list.
3.1.2 certain processing and conversion is performed on the image data and the label data. The image data is converted from PIL format to tensor format in the shape of (C, H, W), where C represents the number of image channels, H represents the picture height, W represents the picture width, and all pixel values are normalized to between 0-1 by dividing 255.
3.1.3 set up before the image input network to carry out certain random image enhancement operation, such as data enhancement of horizontal turning, grid mask, random cutting, color dithering, blurring, etc. to the image with a probability of 0.5, improve the diversity of database pictures, enhance the robustness of the model, note that at this moment, corresponding conversion needs to be carried out to the label file of the enhanced picture at the same time.
3.1.4, calculating the mean value and the standard deviation of the three-channel pixel values of the picture in the whole data set, standardizing the picture so as to facilitate the learning of the model, and providing the standard deviation and the mean value for the step 3.3.
3.2 in the multitask network building module, the network is mainly divided into two parts: multitask shared network and task specific network，And (3) providing to step 3.3.
The multi-task learning is an induction migration mechanism, the generalization capability of the model is improved through information transmission and sharing in the related task training process, and the specific implementation method is parallel training and sharing expression. In the method, the parallel training sharing expression is that the information extraction networks at the earlier stage of each task are the same network, and the weight parameters are shared. Because the task of extracting various information of the muck image needs to learn the characteristics of the muck in the image at first, most of the convolutional layers at the front end of the network are used for realizing the function of extracting the basic characteristics of the muck. From the simple corner and color features extracted from the beginning of the network to the advanced features reflecting the shape of the muck at the back. As most of the convolutional layers at the front ends of different task networks have consistent functions, the same shared network is adopted to extract the basic characteristics of the muck required by each task. And on the shared network layer, different output networks are built according to different tasks. And the output network of each task carries out respective training according to the basic characteristics of the muck obtained by the shared network, thereby realizing different information extraction functions.
Fig. 7 is a schematic structural diagram of the multitask network according to the embodiment, and two boxes "convolution combination" and "residual connection" in the lower right corner of the diagram are used to explain the basic components of the two structures "convolution combination" and "residual connection" in the main graph. The embodiment comprises two tasks, wherein the task 1 is a classification task for judging whether slag tapping is normal, and the task 2 is a target detection task for positioning the slag and classifying soil quality. The identification objects of the two tasks are muck, so that the same muck features need to be extracted, and then the next training learning is respectively carried out according to different task targets based on the muck features to obtain the identification results of the two tasks. Thus, the multitasking network consists of three parts, a multitasking shared network, a task 1 specific network and a task 2 specific network. The composition and relationship of the three network parts is described in detail below.
Input image first enters into multiTask sharing networkThe network is mainly formed by convolution combination and residual connection stacking and is used for extracting image features. The convolution combination is formed by combining a convolution layer, a batch of normalization layers and an activation layer and can be regarded as a function F, and a predicted value y is obtained after the input value x is subjected to convolution combination. The residual connection is that a channel for directly transmitting an input value is added into a network structure, and a predicted value of the structure with the residual connection is the sum of a convolution combination prediction result and an original input value, namely y ═ F (x) + x. The part f (x) learned by the convolution combination is the residual, i.e. the difference between the predicted value y and the observed value x. Therefore, the next layer connected by the residual errors not only contains the information of the previous layer after nonlinear change (convolution combination), but also contains the original information of the previous layer, so that the information can only increase gradually layer by processing, the performance of the model cannot be reduced due to the increase of the network depth, and the network degradation phenomenon which can occur during deep neural network training is solved. And respectively inputting the feature graphs output by the shared network into a specific network of the two tasks to train different tasks.
Classification task 1 is relatively simple, itTask-specific networkBasically a conventional classifier. And converting the feature graph output by the shared network into a one-dimensional vector through global tie pooling, inputting the one-dimensional vector into a full connection layer, converting the regression result into probability through a softmax layer, and obtaining a classification result of whether slag tapping is normal or not.
The object detection task 2 is relatively complex, fromShared networkTherein extract threeFeature fusion is performed on feature maps of different levels from top to bottom, so that the feature map with the lowest level has deep semantic information and basic information such as shallow texture and color, the included features have completeness and diversity, and the final prediction effect is improved. And respectively inputting the feature maps of the three levels into different convolution combinations to perform regression prediction of final results after fusion enhancement. The three feature maps with different sizes respectively contain features with different scales, and objects with different sizes are correspondingly predicted.Prediction output 3The characteristic diagram is large, contains the most detail information, and is responsible for predicting the small target object;prediction output 1The characteristic graph is small, overall information is easier to distinguish, and the characteristic graph is responsible for predicting a large target object. The three prediction output results are combined together to form a target detection result of the whole network on the picture, and the position and the type of the muck in the whole picture can be identified.
3.3 in building modules of the training Process
Firstly, a user-defined data set module and a multi-task network building module are imported, a muck data set and a multi-task muck identification network are instantiated, and a data loader is built to input a network in a mode of user-defined pictures and tag data. The loss function of the network is designed, and the multitask loss function used by the method is composed of weighted sum of the loss function of each task, as shown in formula 1. Because the difficulty degrees of different tasks are different and the loss reduction speeds are inconsistent, the loss values of different tasks need to be weighted and summed to obtain a final loss value so as to balance the training speeds of different tasks and obtain an optimal training result of each task. During network training, a plurality of hyper-parameters need to be set, including an initial learning rate value and a change mode of the initial learning rate value along with the increase of the number of training cycles, parameters of an optimizer and the optimizer, the size of an input image batch, the number of training cycles, the weight of each task loss item of the most important loss function in the method and the like. The training can be started after the loss function and the hyperparameter are set. Each image input network will obtain the classification or regression result of each task, and the predicted value of each task and the real value (GT) in the label provided by step 2.4 corresponding to each task are input into the total loss function, so as to calculate the current total loss value, that is, the distance between the predicted value and the real value. And calculating the derivatives of the total loss value to all the network parameters, and optimizing and updating the network parameters by using an optimizer, namely a round of training iteration. When all pictures in the training set are input into the network for training in turn, a training cycle is obtained. After finishing a training cycle, inputting the pictures of the verification set into the network in turn to calculate the network precision, and observing the network training condition according to the network precision and using the network training condition as a basis for adjusting the hyper-parameters in the next network training. And when the loss is reduced and converged to a certain stable value, finishing the training, and storing the trained network parameters to obtain the multi-task muck recognition model.
Loss＝α 1 Loss 1 +α 2 Loss 2 +α 3 Loss 3 +…+α n Loss n (1)
3.4 inModel testing and evaluation index moduleAnd loading the trained model and parameters, and inputting the muck picture to be tested to obtain the recognition result of each task. And compiling a calculation code of each task evaluation index, and quantitatively evaluating the identification result of each task in the test set, so that the effect of the multi-task muck identification model can be evaluated.
3.5 inMuck recognition result visualization moduleAnd visualizing the recognition results obtained by the task output ends in the network on a picture according to the needs. For example, an object detection frame or a division result to be displayed on a picture is drawn on an original image, and a recognition result or a secondary calculation result to be expressed in text is written at a certain position of the picture or recorded in a result file.
In summary,
the invention discloses a method for acquiring construction information of an earth pressure balance shield, which is to acquire various types of construction information by carrying out muck image identification based on deep learning. The method can be used for model training and real-time identification during application by utilizing the monitoring video data, and has the characteristics of accuracy, rapidness and convenience in application. The invention can simultaneously complete a plurality of muck information extraction tasks, namely, a multi-task muck identification model consisting of a task sharing network and a task specific network is designed and trained based on the idea of multi-task learning. The method can obtain the recognition results of a plurality of tasks through the joint training of a network, and has the characteristics of strong generalization, high speed and low cost.
The method can obtain the muck identification model with different and various task combinations according to the requirements of different tunnel projects, and the obtained model can be updated along with the continuous expansion of the muck database, so that the multi-task muck identification model with wider application range is obtained. Therefore, the method provided by the invention is theoretically suitable for multi-task information extraction of the earth pressure balance shield muck in all regions.
Attached: interpretation of terms:
soil texture: refers to the structure and properties of the soil. Also refers to the quality and structure of the soil properties.
Tunneling: an engineering building buried in the ground is a form in which people utilize underground space.
Subway: the rail transit which is built in cities and has the advantages of high speed, large traffic volume and electric traction is built in tunnels.
A shield method: a tunnel construction method is characterized in that a shield machine is used for excavating stratums and splicing tunnel segments.
The shield machine: a construction machine comprises a shell, a cutter head, pushing equipment, assembling equipment and other matched equipment, wherein the shell is a cylinder and plays a role in protection, and the other equipment is arranged inside the shell.
Earth pressure balance shield: the shield machine is characterized in that soil body cut by a front-end cutter head in a rotating mode when a shield is pushed is used for filling a soil cabin, and the passive soil pressure of the soil cabin is basically balanced with the soil pressure and the water pressure on a digging surface, so that the digging surface and the shield surface are in a balanced state.
Earth pressure balance shield muck: digging waste soil when the earth pressure balance shield is pushed.
Spiral unearthing machine: the machine is used for conveying the soil body cut by the cutter head to the belt conveyor through the rotation of the spiral member, and the rotation speed of the spiral member determines the unearthing speed.
A belt conveyor: and a belt device for conveying the soil from the spiral soil discharging machine to the residue soil vehicle.
Shield construction parameters: various parameters required to be set in the construction process of the shield machine, such as the rotating speed of a cutter head and the like, and whether the parameter setting is reasonable or not determines the safety and the quality of the shield construction.
Artificial intelligence: a new technology science for researching and developing theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.
Machine learning: a multi-field cross discipline relates to a plurality of disciplines such as probability theory, statistics, approximation theory, algorithm complexity theory and the like, is the core of artificial intelligence, and is a fundamental approach for enabling a computer to have intelligence.
Deep learning: the method is a new research direction in the field of machine learning, the intrinsic rules and the expression levels of sample data are learned, and the obtained information is greatly helpful for explaining data such as characters, images and sound.
Multi-task learning: multitask learning is an induction migration mechanism, and the basic aim is to improve the generalization performance of each task. The multi-task learning improves the generalization ability through the domain related information in the related task training signals, and adopts a parallel training shared representation method to learn a plurality of tasks.
Image classification: image classification is an important application of deep learning, namely, the type of an object in a picture is to be distinguished.
Target detection: target detection is an important application of deep learning, namely, objects in pictures are identified, and the positions of the objects are marked.
Semantic segmentation: semantic segmentation is an important application of deep learning, namely, pixel-level classification is carried out on an image, the class to which each pixel belongs is predicted, and individuals are not distinguished.
Example segmentation: example segmentation is an important application of deep learning, namely, each object in a graph is located, and pixel-level labeling is performed to distinguish different individuals.
Sample preparation: the basic unit of the data used in the machine learning method may be a one-dimensional matrix or a high-dimensional matrix.
Labeling: the actual information of the data.
LabelImg: and the target detection task is a tool for labeling the data set.
Labelme: semantic segmentation task tools that label data sets.
Data set: the collection of training, validation, and test sets is referred to as a data set, which includes data and labels.
Training set: is a data sample of model fitting for debugging a neural network.
And (4) verification set: the method is a sample set which is set aside in the model training process and can be used for adjusting the hyper-parameters of the model and carrying out preliminary evaluation on the capability of the model so as to check the training effect.
And (3) test set: used to evaluate the generalization ability of the final model. But not as a basis for algorithm-related selection of parameters, selection features, and the like. It is used to test the actual learning capabilities of the network.
A convolutional neural network: the method is a feedforward neural network which comprises convolution calculation and has a deep structure, and is one of representative algorithms of deep learning.
An image channel: in the RGB color mode, three channels of red, green and blue are indicated.
Image enhancement: the image is matched to the visual response characteristics by attaching some information or transformation data to the original image by some means to selectively highlight features of interest in the image or to suppress (mask) some unwanted features in the image.
Grid mask: and generating a grid with the same resolution as the original image, wherein the gray area value is 1, the black area value is 0, and multiplying the grid with the original image to obtain an enhanced image, so that the information deletion of a specific area is realized, and the method can be essentially understood as a regularization method.
Color dithering: the new image is generated by randomly adjusting the saturation, brightness and contrast of the original image.
Robustness: the system can still maintain certain performance characteristics under the condition of disturbance or uncertainty.
And (3) rolling layers: the convolutional layer is composed of a plurality of convolution units, and the parameters of each convolution unit are optimized through a back propagation algorithm. The convolution operation aims to extract different input features, the convolution layer of the first layer can only extract some low-level features such as edges, lines, corners and other levels, and more layers of networks can iteratively extract more complex features from the low-level features.
Batch standardization layer: batch Normalization (BN) is a technique used to improve the performance and stability of artificial neural networks. It can normalize the input of any layer in the neural network, fixing the mean and variance of the input signal of each layer.
An active layer: and performing activation operation, namely nonlinear transformation on input data, mapping the features to a high-dimensional nonlinear interval for interpretation, and solving the problem which cannot be solved by a linear model.
Global average pooling layer: and averaging the pixel values of the two-dimensional images of each channel of the feature map, so that each channel becomes an average value, and the feature map becomes a one-dimensional vector.
Characteristic diagram: and obtaining a layer after the image is subjected to feature extraction of the convolution layer.
Loss function: and the method is used for evaluating the degree of difference between the predicted value and the actual value of the model.
And (3) hyper-parameter: the hyper-parameter is a parameter that is set before the learning process is started, and is not parameter data obtained by training. In general, the hyper-parameters need to be optimized, and a set of optimal hyper-parameters is selected for the model, so as to improve the learning performance and effect.
Learning rate: is the step size at which the model parameters are updated each time.
Batch size: number of samples input to the network per training.
Claims (5)
1. The invention discloses an earth pressure balance shield muck information extraction method based on deep multi-task learning, which is characterized in that the method extracts important construction information in the earth pressure balance shield tunneling process according to a muck image, so that information to be extracted is determined based on construction requirements, and the types of tasks are determined; collecting muck data, selecting and processing the data to obtain a multi-task recognition data set of muck with balanced soil pressure; then constructing and training according to the recognition tasks to be completed to obtain a muck multi-task recognition model; and finally, inputting a muck monitoring video during the construction of the earth pressure balance shield to realize the real-time identification of muck and obtain various construction information required by constructors.
2. The earth pressure balance shield muck information extraction method based on deep multitask learning according to claim 1, characterized by comprising the following steps:
step 1, determining a residue soil image information extraction task;
step 2, establishing a multitask muck image database;
and 3, constructing a multi-task muck identification model.
3. The earth pressure balance shield muck information extraction method based on deep multitask learning according to claim 2, wherein the step 1 comprises the following steps:
1.1 obtaining construction information from the muck image, wherein the construction information comprises the front soil quality type, muck improvement condition and whether slag discharge abnormity occurs or not;
1.2 determining a muck identification task based on the information in the step 1.1, and determining the types of the tasks; the task types include: image classification, target detection, semantic segmentation and instance segmentation;
1.3 determining the labeling rules of the task identified in step 1.2.
4. The earth pressure balance shield muck information extraction method based on deep multitask learning according to claim 2, characterized in that the step 2 comprises the following steps:
2.1 collecting a muck monitoring video of an application area; the muck monitoring camera is arranged right above or obliquely above the belt conveyor and is aligned with the muck outlet of the spiral muck remover, so that the complete muck removing process and muck form are ensured to be shot; installing lighting equipment above the soil outlet;
2.2, preprocessing the data after collecting the video data, and intercepting the effective video in the stable unearthing stage;
2.3 extracting video frames at a frequency of one per 30 frames;
2.4, image annotation is carried out according to the types of different recognition tasks;
the method comprises an image classification task, wherein image categories are marked in a mode of naming image classifications or dividing folders; the method comprises a target detection task, wherein the target detection task is used for manually marking the muck image, framing a target to be identified in each image by using a rectangular frame, and inputting the category of each target frame; the method comprises semantic segmentation and instance segmentation tasks, wherein the semantic segmentation and instance segmentation tasks are used for manually marking the muck images, marking the boundary of a target to be segmented in each image and inputting the category of each region; therefore, each image must be labeled, and the information recorded therein is called the true value, which is used to provide step 3.3 for the training process;
and 2.5, the obtained muck picture file and the corresponding task label files are data sets for muck multi-task identification, and a training set, a verification set and a test set are divided.
5. The earth pressure balance shield muck information extraction method based on deep multitask learning according to claim 2, characterized in that the step 3 comprises the following steps:
3.1 custom datasets
3.1.1 uploading the image data of the dregs and the corresponding label information of a plurality of tasks from the data set storage path, respectively storing in a list
3.1.2 processing and converting image data and tag data
Converting the image data from a PIL format into a tensor format with the shape of (C, H, W), wherein C represents the number of image channels, H represents the height of the picture, W represents the width of the picture, and dividing all pixel values by 255 to be normalized to be between 0 and 1;
3.1.3 setting random image enhancement operation on the image before inputting into the network;
3.1.4 calculating the mean value and the standard deviation of three-channel pixel values of the picture in the whole data set, carrying out standardization processing on the picture, and providing the picture for the step 3.3;
3.2, building a multitask network, wherein the network is divided into two parts: the multitask shared network and each task specific network are provided to step 3.3
3.3 construction of the training Process
Firstly, importing a user-defined data set and a multi-task network building, and constructing a data loader to input a user-defined picture and tag data into a network;
designing a loss function of the network, as shown in formula 1;
setting hyper-parameters including initial values of learning rate and the changing mode of the initial values along with the increase of training cycle times, parameters of an optimizer and the optimizer, input picture batch size, training cycle times and the weight of each task loss item of the most important loss function in the method when training a network;
training can be started after the loss function and the hyper-parameter are set; each image input network will obtain the classification or regression result of each task, and the predicted value of each task and the real value (GT) in the label provided by step 2.4 corresponding to each task are input into the total loss function to calculate the current total loss value, that is, the distance between the predicted value and the real value; calculating the derivatives of the total loss value to all network parameters, and performing optimization updating on the network parameters by using an optimizer, namely performing one round of training iteration; when all pictures in the training set are input into the network for training in turn, a training cycle is formed; after finishing a training cycle, inputting the pictures of the verification set into the network in turn to calculate the network precision, observing the network training condition according to the network precision, and taking the observed network training condition as a basis for adjusting the hyper-parameters in the next network training; after the loss is reduced and converged to a certain stable value, the training can be finished, and the trained network parameters are stored, so that the multi-task residue soil recognition model is obtained;
Loss＝α 1 Loss 1 +α 2 Loss 2 +α 3 Loss 3 +…+α n Loss n (1)
3.4 model testing and evaluating indexes;
loading the trained model and parameters, and inputting a muck picture to be tested to obtain the recognition result of each task;
3.5 visualization of residue soil recognition result
And visualizing the recognition result obtained by each task output end in the network on the picture according to the requirement.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210484070.3A CN114973067A (en) | 2022-05-06 | 2022-05-06 | Earth pressure balance shield muck information extraction method based on deep multi-task learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210484070.3A CN114973067A (en) | 2022-05-06 | 2022-05-06 | Earth pressure balance shield muck information extraction method based on deep multi-task learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114973067A true CN114973067A (en) | 2022-08-30 |
Family
ID=82981543
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210484070.3A Pending CN114973067A (en) | 2022-05-06 | 2022-05-06 | Earth pressure balance shield muck information extraction method based on deep multi-task learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114973067A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118067006A (en) * | 2024-04-19 | 2024-05-24 | 中交隧道工程局有限公司 | Dynamic detection method for deslagging volume of slurry shield |
-
2022
- 2022-05-06 CN CN202210484070.3A patent/CN114973067A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118067006A (en) * | 2024-04-19 | 2024-05-24 | 中交隧道工程局有限公司 | Dynamic detection method for deslagging volume of slurry shield |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108776772B (en) | Cross-time building change detection modeling method, detection device, method and storage medium | |
| Tong et al. | A supervised and fuzzy-based approach to determine optimal multi-resolution image segmentation parameters | |
| Jianya et al. | A review of multi-temporal remote sensing data change detection algorithms | |
| CN101364259B (en) | Method for extracting road various information of multi-level knowledge driven panchromatic remote sensing image | |
| CN112990227B (en) | Face geology detection method | |
| CN103049763B (en) | Context-constraint-based target identification method | |
| CN111275688A (en) | Small target detection method based on context feature fusion screening of attention mechanism | |
| CN110490100A (en) | Ground automatic identification based on deep learning names method and system | |
| CN112347895A (en) | Ship remote sensing target detection method based on boundary optimization neural network | |
| KR102346676B1 (en) | Method for creating damage figure using the deep learning-based damage image classification of facility | |
| US20220315243A1 (en) | Method for identification and recognition of aircraft take-off and landing runway based on pspnet network | |
| CN112613350A (en) | High-resolution optical remote sensing image airplane target detection method based on deep neural network | |
| CN109886147A (en) | A kind of more attribute detection methods of vehicle based on the study of single network multiple-task | |
| CN112967255A (en) | Shield segment defect type identification and positioning system and method based on deep learning | |
| CN113971764B (en) | Remote sensing image small target detection method based on improvement YOLOv3 | |
| CN112560675A (en) | Bird visual target detection method combining YOLO and rotation-fusion strategy | |
| CN114140448A (en) | Tunnel face surrounding rock intelligent grading method and device based on deep learning | |
| CN114973067A (en) | Earth pressure balance shield muck information extraction method based on deep multi-task learning | |
| Li et al. | 3DCentripetalNet: Building height retrieval from monocular remote sensing imagery | |
| CN117409327A (en) | Dynamic quantitative remote sensing monitoring method for surface mine | |
| CN112785629A (en) | Aurora motion characterization method based on unsupervised deep optical flow network | |
| CN114972906A (en) | Soil quality type identification method for excavation surface of soil pressure balance shield | |
| Weidner et al. | The influence of training data variability on a supervised machine learning classifier for Structure from Motion (SfM) point clouds of rock slopes | |
| CN116758421A (en) | Remote sensing image directed target detection method based on weak supervised learning | |
| CN116206155A (en) | Waste steel classification and identification method based on YOLOv5 network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |