CN116503606B - Road surface wet and slippery region segmentation method and device based on sub-graph feature fusion - Google Patents

Road surface wet and slippery region segmentation method and device based on sub-graph feature fusion Download PDF

Info

Publication number
CN116503606B
CN116503606B CN202310764266.2A CN202310764266A CN116503606B CN 116503606 B CN116503606 B CN 116503606B CN 202310764266 A CN202310764266 A CN 202310764266A CN 116503606 B CN116503606 B CN 116503606B
Authority
CN
China
Prior art keywords
sub
graph
features
feature
road surface
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202310764266.2A
Other languages
Chinese (zh)
Other versions
CN116503606A (en
Inventor
王建强
郭宇昂
余贵珍
曲小波
王云鹏
许庆
徐少兵
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tsinghua University
Beihang University
Original Assignee
Tsinghua University
Beihang University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tsinghua University, Beihang University filed Critical Tsinghua University
Priority to CN202310764266.2A priority Critical patent/CN116503606B/en
Publication of CN116503606A publication Critical patent/CN116503606A/en
Application granted granted Critical
Publication of CN116503606B publication Critical patent/CN116503606B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/16—Image acquisition using multiple overlapping images; Image stitching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)
  • Traffic Control Systems (AREA)

Abstract

The application relates to the technical field of unmanned operation, in particular to a road surface wet and slippery region segmentation method and device based on sub-graph feature fusion, wherein the method comprises the following steps: obtaining a training set and a test set of the wet-skid road surface image, cutting the training set image to obtain a first sub-image for neural network training, constructing a sub-image feature quality evaluation model, using the feature training of the first sub-image, cutting the test set to obtain a second sub-image, extracting high-level features to calculate features to be fused, sending the features of the second sub-image into the trained sub-image feature quality evaluation model to obtain sub-image feature fusion weights, and finally obtaining a road surface wet-skid region segmentation result by utilizing feature fusion calculation among the sub-images. According to the embodiment of the application, the context information of each sub-graph can be enriched by realizing the fusion of the high-level characteristics of each sub-graph, so that the false detection of a dry road surface and a wet road surface is reduced, the segmentation result of a wet region is more accurate, and the safety of vehicle driving is further improved.

Description

Road surface wet and slippery region segmentation method and device based on sub-graph feature fusion
Technical Field
The application relates to the technical field of unmanned operation, in particular to a road surface wet and slippery region segmentation method and device based on sub-graph feature fusion.
Background
In the driving process, the wet and slippery road surface state can bring potential risks to driving safety, and detecting the wet and slippery area on the road surface can provide environment information for a driver or an automatic driving automobile, so that the driver or the automatic driving automobile can adopt safer driving strategies.
In the related art, when an image is segmented, the whole image is cut into a plurality of subgraphs with smaller sizes, each subgraph is segmented, and the results are spliced, so that the original image is segmented, and finally the road surface and the obstacle on a wet road are segmented.
However, in the related art, the receptive field of the network is reduced by simply splitting and splicing the subgraphs, the network cannot be ensured to obtain sufficient context information, and high-level characteristics in each subgraph are difficult to embody, so that the false detection rate of a wet road surface is increased, the accuracy of a road surface wet and slippery region splitting result is reduced, the driving safety of a user is influenced, and the problem is to be solved.
Disclosure of Invention
The application provides a road surface wet and slippery region segmentation method and device based on sub-graph feature fusion, which are used for solving the problems that in the related technology, the receptive field of a network is reduced aiming at simple segmentation and splicing of sub-graphs, the network cannot be ensured to obtain sufficient context information, high-level features in each sub-graph are difficult to embody, the false detection rate aiming at a dry and wet road surface is increased, the accuracy of road surface wet and slippery region segmentation results is reduced, the driving safety of a user is influenced and the like.
An embodiment of a first aspect of the present application provides a road surface wet-skid region segmentation method based on sub-graph feature fusion, including the following steps: collecting wet road surface image data or acquiring a data set disclosed by a network to generate a training set and a testing set; cutting each image of the training set into a plurality of first subgraphs which have overlapping areas and are consistent in size, and sending each first subgraph into a preset neural network segmentation algorithm based on a coding and decoding structure for training to obtain a trained network; constructing a sub-graph feature quality evaluation model, and extracting IoU (Intersection over Union, cross ratio) of a prediction result of each first sub-graph, HSV (Hue, saturation, value) features and high-level features of the sub-graph for model training; cutting each image of the test set into a plurality of second sub-images which have overlapping areas and are consistent in size, sending the second sub-images into the trained network for operation, extracting high-level characteristics in the operation process, and calculating characteristics to be fused for the overlapping areas of each second sub-image based on a cross attention mechanism; calculating the weight of the features to be fused based on the sub-graph feature quality evaluation model, and carrying out feature fusion of the mutually overlapped regions among sub-graphs according to the weight to obtain features of which the feature fusion is completed in the overlapped regions; expanding the features integrated in the overlapping region to the whole subgraph to obtain expanded subgraph features, and sending the expanded subgraph features into a decoder of the trained network to obtain a road surface wet and slippery region segmentation result.
Optionally, in an embodiment of the present application, the extracting IoU of the prediction result of each first sub-graph, HSV features of the sub-graph and high-level features perform model training, including: HSV features and high-level features of a first sub-graph are subjected to global mean pooling and then serve as input of a model, ioU of a predicted result of the first sub-graph serves as a true value of an output result of the model, so that the sub-graph feature quality evaluation model is trained, and the trained model can output sub-graph feature quality evaluation scores.
Optionally, in an embodiment of the present application, the feature of merging the overlapping area extends to an entire sub-graph, including: treating high-level features of sub-graphs without feature fusion as in the attention mechanismMatrix sum->Matrix and regarding the high-level characteristics of the sub-graph after the overlapping area has the fusion characteristics as +.>A matrix; and calculating a new feature map based on an attention mechanism to serve as a sub-graph feature after feature expansion.
Optionally, in an embodiment of the present application, the calculation formula of the weight is:
wherein ,is->Zhang Zitu weight in feature fusion, < ->For the number of subgraphs involved in feature fusion, +.>Is->Zhang Zitu the quality evaluation score is an exponentially amplified value.
Optionally, in one embodiment of the present application, the learning rate update formula of the sub-graph feature quality evaluation model is:
wherein ,for the total number of iterations +.>The current iteration number.
An embodiment of a second aspect of the present application provides a road surface wet-skid region segmentation apparatus based on sub-graph feature fusion, including: the acquisition module is used for acquiring the image data of the wet road surface or acquiring a data set disclosed by a network and generating a training set and a testing set; the training module is used for cutting each image of the training set into a plurality of first subgraphs which have overlapping areas and are consistent in size, and sending each first subgraph into a preset neural network segmentation algorithm based on a coding and decoding structure for training to obtain a trained network; the construction module is used for constructing a sub-graph feature quality evaluation model, extracting IoU of the prediction result of each first sub-graph, HSV features and high-grade features of the sub-graph and using the HSV features and the high-grade features for model training; the computing module is used for cutting each image of the test set into a plurality of second sub-images which have overlapping areas and are consistent in size, sending the second sub-images into the trained network for operation, extracting high-level characteristics in the operation process, and computing characteristics to be fused for the overlapping areas of each second sub-image based on a cross attention mechanism; the fusion module is used for calculating the weight of the features to be fused based on the sub-image feature quality evaluation model, and carrying out feature fusion of the mutually overlapped areas among the sub-images according to the weight to obtain features of which the feature fusion is completed in the overlapped areas; the segmentation module is used for expanding the features fused in the overlapping region to the whole subgraph to obtain expanded subgraph features, and sending the expanded subgraph features into a decoder of the trained network to obtain a road surface wet and slippery region segmentation result.
Optionally, in an embodiment of the present application, the building module is specifically configured to pool HSV features and high-level features of the first sub-graph through global average as input of a model, and use IoU of a predicted result of the first sub-graph as a true value of an output result of the model, so as to train the sub-graph feature quality evaluation model, and the trained model may output a sub-graph feature quality evaluation score.
Optionally, in one embodiment of the present application, the segmentation module includes: a classification unit for considering the high-level features of the subgraph without feature fusion as in the attention mechanismMatrix sum->Matrix and regarding the high-level characteristics of the sub-graph after the overlapping area has the fusion characteristics as +.>A matrix; and the calculating unit is used for calculating a new feature map based on the attention mechanism and taking the new feature map as a sub-graph feature after feature expansion.
Optionally, in an embodiment of the present application, the calculation formula of the weight is:
wherein ,is->Zhang Zitu weight in feature fusion, < ->For the number of subgraphs involved in feature fusion, +.>Is->Zhang Zitu the quality evaluation score is an exponentially amplified value.
Optionally, in one embodiment of the present application, the learning rate update formula of the sub-graph feature quality evaluation model is:
wherein ,for the total number of iterations +.>The current iteration number.
An embodiment of a third aspect of the present application provides an electronic device, including: the road surface wet and slippery region segmentation method based on the sub-graph feature fusion comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the road surface wet and slippery region segmentation method based on the sub-graph feature fusion.
An embodiment of the fourth aspect of the present application provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the road surface wet skid region segmentation method based on sub-graph feature fusion as above.
According to the embodiment of the application, the contextual information of each sub-graph can be enriched by fusing the high-level characteristics of each sub-graph, so that the false detection of a dry road surface and a wet road surface is reduced, the segmentation result of a wet region is more accurate, and the safety of vehicle driving is further improved. Therefore, the problems that in the related technology, the receptive field of a network is reduced by simply dividing and splicing the subgraphs, the network cannot be guaranteed to obtain sufficient context information, high-level characteristics in each subgraph are difficult to embody, the false detection rate of a wet road surface is increased, the accuracy of a road surface wet and slippery region dividing result is reduced, the driving safety of a user is influenced and the like are solved.
Additional aspects and advantages of the application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the application.
Drawings
The foregoing and/or additional aspects and advantages of the application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:
fig. 1 is a flowchart of a road surface wet and slippery region segmentation method based on sub-graph feature fusion according to an embodiment of the present application;
FIG. 2 is a logic diagram of a road surface wet skid region segmentation method based on subgraph feature fusion according to an embodiment of the present application;
FIG. 3 is a schematic diagram of the overall structure of a road surface wet and slippery region segmentation method based on sub-graph feature fusion according to an embodiment of the present application;
FIG. 4 is a diagram illustrating image cropping results according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a sub-graph feature quality assessment model according to an embodiment of the present application;
FIG. 6 is a schematic diagram of feature computation and feature fusion to be fused in an overlapping region according to an embodiment of the present application;
FIG. 7 is a feature extension schematic of an embodiment of the application;
fig. 8 is a schematic structural diagram of a road surface wet and slippery region segmentation device based on sub-graph feature fusion according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative and intended to explain the present application and should not be construed as limiting the application.
The following describes a road surface wet-skid region segmentation method and device based on sub-graph feature fusion according to an embodiment of the application with reference to the accompanying drawings. Aiming at the problems that in the related technology mentioned in the background technology, the receptive field of a network is reduced by aiming at simple segmentation and splicing of subgraphs, the network cannot be guaranteed to obtain sufficient context information, high-level characteristics in each subgraph are difficult to embody, the false detection rate of a wet road surface is increased, the accuracy of a road surface wet-skid region segmentation result is reduced, and the driving safety of a user is influenced. Therefore, the problems that in the related technology, the receptive field of a network is reduced by simply dividing and splicing the subgraphs, the network cannot be guaranteed to obtain sufficient context information, high-level characteristics in each subgraph are difficult to embody, the false detection rate of a wet road surface is increased, the accuracy of a road surface wet and slippery region dividing result is reduced, the driving safety of a user is influenced and the like are solved.
Specifically, fig. 1 is a schematic flow chart of a road surface wet and slippery region segmentation method based on sub-graph feature fusion provided by an embodiment of the present application.
As shown in fig. 1, the road surface wet and slippery region segmentation method based on sub-graph feature fusion comprises the following steps:
in step S101, wet road surface image data is acquired or a data set disclosed by a network is acquired, and a training set and a test set are generated.
In the actual implementation process, high-resolution wet road surface image data or a data set of the wet road surface image disclosed by using a network can be acquired, and the obtained data set is further divided into a training set and a testing set for model training in the following steps.
In step S102, each image of the training set is cut into a plurality of first subgraphs having overlapping areas and consistent in size, and each first subgraph is sent to a preset neural network segmentation algorithm based on a codec structure to perform training, so as to obtain a trained network.
It can be understood that, in the embodiment of the present application, the first sub-graph may be obtained by clipping each image in the training set obtained in the above step, and used for training by sending the first sub-graph into a preset neural network segmentation algorithm based on a codec structure.
It should be noted that, the preset neural network segmentation algorithm based on the codec structure may be set by those skilled in the art according to the actual situation, which is not limited herein.
In step S103, a sub-graph feature quality evaluation model is constructed, and IoU of the prediction result of each first sub-graph, HSV features and high-level features of the sub-graph are extracted for model training.
In the actual execution process, an artificial neural network can be used for constructing a sub-graph feature quality evaluation model, the number of input layer nodes of the network is set to 259, hidden layer nodes are set to 500, the number of output layer nodes is set to 1, a prediction result of each first sub-graph after the pre-set neural network training based on the coding and decoding structure in the steps is extracted, and the cross ratio IoU in the prediction result, the HSV features and the high-level features of the sub-graph are used for model training.
Optionally, in one embodiment of the present application, extracting IoU of the prediction result of each first sub-graph, HSV features and high-level features of the sub-graph for model training includes: HSV features and high-level features of the first sub-graph are subjected to global mean pooling and then serve as input of a model, ioU of a predicted result of the first sub-graph serves as a true value of an output result of the model, so that the sub-graph feature quality evaluation model is trained, and the trained model can output sub-graph feature quality evaluation scores.
Specifically, the HSV features and the high-level features of the first sub-graph in the steps can be subjected to global averaging to obtain the input of the sub-graph feature quality evaluation model, and IoU of the predicted result of the first sub-graph in the steps is used as the true value of the model output result, so that training of the sub-graph feature quality evaluation model is realized.
Optionally, in one embodiment of the present application, the learning rate update formula of the sub-graph feature quality assessment model is:
wherein ,for the total number of iterations +.>The current iteration number.
For example, the sub-graph feature quality evaluation model can be trained using the above model, and the number of training iterations is set4 ten thousand times, wherein the training process takes mean square error as a loss function, and the initial learning rate is 0.5.
In step S104, each image of the test set is cut into a plurality of second sub-images having overlapping areas and consistent in size, and sent to the trained network for operation, high-level features in the operation process are extracted, and features to be fused are calculated for the overlapping areas of each second sub-image based on a cross attention mechanism.
Specifically, the test set image can be cut to obtain a second sub-image, and the second sub-image is sent to the trained sub-image feature quality evaluation model obtained in the step to be operated, and for each sub-image, the high-level features output by the network encoder are extracted. For overlapping regions between subgraphs, cross-attention mechanisms may be used to calculate features to be fused.
In step S105, based on the sub-graph feature quality evaluation model, the weights of the features to be fused are calculated, and feature fusion of the mutually overlapping regions between sub-graphs is performed according to the weights, so as to obtain features of which feature fusion is completed in the overlapping regions.
In the actual execution process, HSV features and high-grade features of each second sub-graph in the steps are extracted, the HSV features and the high-grade features are input into the sub-graph feature quality evaluation model obtained in the steps after global averaging, the result output by the model is sub-graph feature quality evaluation score, the numerical value of the score is amplified through an exponential function, and the score numerical value of each sub-graph is normalized to be used as the weight for sub-graph overlapping region feature fusion.
Optionally, in an embodiment of the present application, the calculation formula of the weight is:
wherein ,is->Zhang Zitu weight in feature fusion, < ->For the number of subgraphs involved in feature fusion, +.>Is->Zhang Zitu the quality evaluation score is an exponentially amplified value.
Specifically, the firstZhang Zitu the value of the characteristic quality evaluation score amplified by an exponential function +.>The method comprises the following steps:
wherein ,is->Zhang Zitu characteristic quality assessment score, ->Is->Zhang Zitu the numerical value of the index function amplified score is further normalized according to the numerical value of the index function amplified score, and the numerical value of the score of each sub-graph is used as the weight of the sub-graph overlapping region feature fusion.
In step S106, the features fused in the overlapping area are expanded to the whole sub-graph, the expanded sub-graph features are obtained, and the expanded sub-graph features are sent to a decoder of the trained network, so as to obtain the road surface wet-skid area segmentation result.
It can be understood that in the embodiment of the application, the expanded sub-image advanced features can be sent to a trained neural network segmentation algorithm decoder based on a coding and decoding structure to finish subsequent operation, the output of the decoder is the road surface wet and slippery region segmentation result of the sub-image, and the segmentation result of each sub-image can be spliced to obtain the segmentation result of the original image.
In the actual execution process, each image shot in real time by using a camera can be regarded as a test set image, the feature calculation to be fused in the overlapping area can be carried out after the subgraph is segmented, the feature fusion is carried out on the overlapping area of the high-level features of the subgraph according to the weight, the features fused in the overlapping area are expanded to the whole subgraph, and the subgraph features after the feature expansion are input into a decoder, so that the segmentation of the road wet-sliding area is realized.
Optionally, in one embodiment of the present application, expanding the feature fused in by the overlapping region to the whole sub-graph includes: treating high-level features of sub-graphs without feature fusion as in the attention mechanismMatrix sum->Matrix and regarding the high-level characteristics of the sub-graph after the overlapping area has the fusion characteristics as +.>A matrix;and calculating a new feature map based on an attention mechanism to serve as a sub-graph feature after feature expansion.
It will be appreciated that in embodiments of the present application, the new feature map obtained by the attention mechanism calculation may be a sub-graph high-level feature that implements feature expansion.
The following describes the working of the embodiment of the present application in detail with a specific embodiment, and specific steps are shown in fig. 2, where the encoder and decoder of deep 3plus algorithm may be selected, and a sub-graph feature fusion portion is inserted in the middle as shown in fig. 3, specifically:
firstly, self-picking high-resolution wet road surface image data or using a data set disclosed by a network, setting two thirds of all data as a training set and setting the rest data as a test set. As shown in fig. 4, each image of the training set is cut into multiple sub-images with identical sizes, and each sub-image has a third of overlapping area with four adjacent sub-images, namely, the top, bottom, left and right sub-images, so that the encoder and the decoder in fig. 3 are trained by using the sub-images.
Further, as shown in fig. 5, an artificial neural network is used to construct a sub-graph feature quality evaluation model, the number of input nodes of the network is 259, the number of hidden nodes is 500, the number of output nodes is 1, and IoU of each sub-graph prediction result, HSV features and high-level features of the sub-graph in the training process are extracted. HSV features and high-grade features of the subgraph are input into a subgraph feature quality evaluation model after global mean pooling, and IoU of a subgraph prediction result is taken as a true value of a model output result. Training the sub-graph feature quality evaluation model, wherein the training iteration number is 4 ten thousand times. In the training process, the mean square error is taken as a loss function, the initial learning rate is 0.5, and the learning rate updating formula is as follows:
wherein ,for a total number of iterations, 40000,/>the current iteration number.
Secondly, the test set image is cut again by the same method as the training set and is sent to an encoder of the deep labv3plus network after training is finished for operation. And extracting high-level features output by the deep 3plus network encoder for each sub-graph, and calculating features to be fused by using a cross attention mechanism for overlapping areas among the high-level features of the sub-graph. As shown in fig. 6, the calculation of the feature to be fused in the overlapping area of the center sub-graph and the upper sub-graph is taken as an example here:
wherein ,is center subgraph->And upper Fang Zitu->Features of the overlapping region, ">Is->And->The characteristics of the overlap region. />And the feature to be fused is required to be fused into the overlapping area of the center sub-graph and the upper sub-graph.
And extracting HSV features and high-grade features of each sub-graph obtained by cutting again, inputting the HSV features and the high-grade features into a trained sub-graph feature quality evaluation model after global averaging, and outputting a result which is a sub-graph feature quality evaluation score by the model. Amplifying the score value through an exponential function, normalizing the score value of each sub-graph to be used as the weight of the feature fusion of the overlapping region of the sub-graph:
wherein ,is->Zhang Zitu characteristic quality assessment score, ->For the number of subgraphs involved in feature fusion, +.>Is the firstZhang Zitu weight in feature fusion. And realizing feature fusion on the overlapping area of the high-level features of the subgraph according to the weight. Taking the feature fusion of the overlapping region of the center sub-graph and the upper sub-graph of fig. 6 as an example:
wherein ,，/>the weight of the central sub-graph in the feature fusion and the weight of the upper sub-graph in the feature fusion are respectively +.>To realize the characteristic of the overlapping area above the fused center subgraph, the original +.>Features.
Finally, the sub-graph high-level features without feature fusion are regarded as being in the attention mechanismMatrix sum->Matrix, regarding the high-level characteristics of the sub-graph with the fusion characteristics in the overlapped area as +.>Matrix:
as shown in fig. 7, a new feature map is calculated based on the attention mechanismThe feature map is the sub-graph high-level feature which realizes feature expansion. The obtained characteristic diagram->And sending the result to a decoder of the trained network to complete subsequent operation. The output of the decoder is the road surface wet and slippery region segmentation result of the subgraph, and the segmentation results of the subgraphs are spliced to obtain the segmentation result of the original image.
According to the road surface wet-skid region segmentation method based on the sub-graph feature fusion, which is provided by the embodiment of the application, the context information of each sub-graph can be enriched by fusing the high-level features of each sub-graph, so that the false detection of a dry-wet road surface is reduced, the segmentation result of the wet-skid region is more accurate, and the safety of vehicle driving is further improved. Therefore, the problems that in the related technology, the receptive field of a network is reduced by simply dividing and splicing the subgraphs, the network cannot be guaranteed to obtain sufficient context information, high-level characteristics in each subgraph are difficult to embody, the false detection rate of a wet road surface is increased, the accuracy of a road surface wet and slippery region dividing result is reduced, the driving safety of a user is influenced and the like are solved.
The road surface wet and slippery region segmentation device based on sub-graph feature fusion according to the embodiment of the application is described with reference to the accompanying drawings.
Fig. 8 is a schematic structural diagram of a road surface wet and slippery region segmentation device based on sub-graph feature fusion according to an embodiment of the present application.
As shown in fig. 8, the road surface wet region dividing apparatus 10 based on the sub-graph feature fusion includes: acquisition module 100, training module 200, construction module 300, calculation module 400, fusion module 500, and segmentation module 600.
The acquisition module 100 is configured to acquire image data of a wet road surface or acquire a data set disclosed by a network, and generate a training set and a testing set.
The training module 200 is configured to cut each image of the training set into a plurality of first subgraphs having overlapping areas and consistent sizes, and send each first subgraph to a preset neural network segmentation algorithm based on a codec structure for training, so as to obtain a trained network.
The construction module 300 is configured to construct a sub-graph feature quality evaluation model, and extract IoU of the prediction result of each first sub-graph, HSV features and high-level features of the sub-graph for model training.
The computing module 400 is configured to cut each image of the test set into a plurality of second sub-graphs having overlapping regions and consistent sizes, send the second sub-graphs to the trained network for operation, extract high-level features in the operation process, and compute features to be fused for the overlapping regions of each second sub-graph based on a cross attention mechanism.
The fusion module 500 is configured to calculate weights of features to be fused based on the sub-graph feature quality evaluation model, and perform feature fusion of mutually overlapping regions between sub-graphs according to the weights, so as to obtain features of which feature fusion is completed in the overlapping regions.
The segmentation module 600 is configured to expand the features fused in the overlapping region to the whole sub-graph, obtain the expanded sub-graph features, and send the expanded sub-graph features to a decoder of the trained network, so as to obtain the road surface wet and slippery region segmentation result.
Optionally, in one embodiment of the present application, the building module 300 is specifically configured to pool HSV features and high-level features of the first sub-graph through global averaging, and use IoU of a predicted result of the first sub-graph as a true value of an output result of the model, so as to train the sub-graph feature quality evaluation model, and the trained model may output the sub-graph feature quality evaluation score.
Optionally, in one embodiment of the present application, the segmentation module 600 includes: a classification unit and a calculation unit.
Wherein the classifying unit is used for regarding the high-level features of the subgraph without feature fusion as the attention mechanismMatrix sum->Matrix and regarding the high-level characteristics of the sub-graph after the overlapped area has the fusion characteristics as the attention mechanismA matrix.
And the calculating unit is used for calculating a new feature map based on the attention mechanism and taking the new feature map as a sub-graph feature after feature expansion.
Optionally, in an embodiment of the present application, the calculation formula of the weight is:
wherein ,is->Zhang Zitu weight in feature fusion, < ->For the number of subgraphs involved in feature fusion, +.>Is->Zhang Zitu the quality evaluation score is an exponentially amplified value.
Optionally, in one embodiment of the present application, the learning rate update formula of the sub-graph feature quality assessment model is:
wherein ,for the total number of iterations +.>The current iteration number.
It should be noted that the foregoing explanation of the embodiment of the road surface wet-skid region segmentation method based on the sub-graph feature fusion is also applicable to the road surface wet-skid region segmentation device based on the sub-graph feature fusion of this embodiment, and will not be repeated here.
According to the road surface wet-skid area segmentation device based on the sub-graph feature fusion, which is provided by the embodiment of the application, the context information of each sub-graph can be enriched by fusing the high-level features of each sub-graph, so that the false detection of a dry-wet road surface is reduced, the segmentation result of the wet-skid area is more accurate, and the safety of vehicle driving is further improved. Therefore, the problems that in the related technology, the receptive field of a network is reduced by simply dividing and splicing the subgraphs, the network cannot be guaranteed to obtain sufficient context information, high-level characteristics in each subgraph are difficult to embody, the false detection rate of a wet road surface is increased, the accuracy of a road surface wet and slippery region dividing result is reduced, the driving safety of a user is influenced and the like are solved.
Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device may include:
memory 901, processor 902, and a computer program stored on memory 901 and executable on processor 902.
The processor 902 implements the road surface wet-skid region segmentation method based on the sub-graph feature fusion provided in the above embodiment when executing the program.
Further, the electronic device further includes:
a communication interface 903 for communication between the memory 901 and the processor 902.
Memory 901 for storing a computer program executable on processor 902.
Memory 901 may comprise high-speed RAM memory or may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
If the memory 901, the processor 902, and the communication interface 903 are implemented independently, the communication interface 903, the memory 901, and the processor 902 may be connected to each other through a bus and perform communication with each other. The bus may be an industry standard architecture (Industry Standard Architecture, abbreviated ISA) bus, an external device interconnect (Peripheral Component, abbreviated PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated EISA) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, only one thick line is shown in fig. 9, but not only one bus or one type of bus.
Alternatively, in a specific implementation, if the memory 901, the processor 902, and the communication interface 903 are integrated on a chip, the memory 901, the processor 902, and the communication interface 903 may communicate with each other through internal interfaces.
The processor 902 may be a central processing unit (Central Processing Unit, abbreviated as CPU) or an application specific integrated circuit (Application Specific Integrated Circuit, abbreviated as ASIC) or one or more integrated circuits configured to implement embodiments of the present application.
The present embodiment also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the road surface wet skid region segmentation method based on sub-graph feature fusion as above.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or N embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present application, "N" means at least two, for example, two, three, etc., unless specifically defined otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and additional implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order from that shown or discussed, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the embodiments of the present application.
Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or N wires, a portable computer cartridge (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It is to be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the N steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As with the other embodiments, if implemented in hardware, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
Those of ordinary skill in the art will appreciate that all or a portion of the steps carried out in the method of the above-described embodiments may be implemented by a program to instruct related hardware, where the program may be stored in a computer readable storage medium, and where the program, when executed, includes one or a combination of the steps of the method embodiments.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing module, or each unit may exist alone physically, or two or more units may be integrated in one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules may also be stored in a computer readable storage medium if implemented in the form of software functional modules and sold or used as a stand-alone product.
The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, or the like. While embodiments of the present application have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the application, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the application.

Claims (8)

1. The road surface wet and slippery region segmentation method based on the subgraph feature fusion is characterized by comprising the following steps of:
collecting wet road surface image data or acquiring a data set disclosed by a network to generate a training set and a testing set;
cutting each image of the training set into a plurality of first subgraphs which have overlapping areas and are consistent in size, and sending each first subgraph into a preset neural network segmentation algorithm based on a coding and decoding structure for training to obtain a trained network;
constructing a subgraph feature quality evaluation model, and extracting IoU of a prediction result of each first subgraph, HSV features and high-grade features of the subgraph for model training;
cutting each image of the test set into a plurality of second sub-images which have overlapping areas and are consistent in size, sending the second sub-images into the trained network for operation, extracting high-level characteristics in the operation process, and calculating characteristics to be fused for the overlapping areas of each second sub-image based on a cross attention mechanism;
calculating the weight of the features to be fused based on the sub-graph feature quality evaluation model, and carrying out feature fusion of the mutually overlapped regions among sub-graphs according to the weight to obtain features of which the feature fusion is completed in the overlapped regions; and
expanding the features integrated in the overlapping region to a whole sub-graph to obtain expanded sub-graph features, and sending the expanded sub-graph features into a decoder of the trained network to obtain a road surface wet and slippery region segmentation result, wherein the method comprises the steps ofExpanding the features fused in the overlapping region to an entire sub-graph, including: treating high-level features of sub-graphs without feature fusion as in the attention mechanismMatrix sum->Matrix and regarding the high-level characteristics of the sub-graph after the overlapping area has the fusion characteristics as +.>A matrix;
the extracting IoU of the predicted result of each first sub-graph, HSV features and high-grade features of the sub-graph for model training comprises the following steps: HSV features and high-level features of a first sub-graph are subjected to global mean pooling and then serve as input of a model, ioU of a predicted result of the first sub-graph serves as a true value of an output result of the model, so that the sub-graph feature quality evaluation model is trained, and the trained model can output sub-graph feature quality evaluation scores.
2. The method of claim 1, wherein the weight is calculated as:
wherein ,is->Zhang Zitu weight in feature fusion, < ->For the number of subgraphs involved in feature fusion, +.>Is->Zhang Zitu the quality evaluation score is an exponentially amplified value.
3. The method of claim 1, wherein the learning rate update formula of the sub-graph feature quality assessment model is:
wherein ,for the total number of iterations +.>The current iteration number.
4. The utility model provides a road surface wet and slippery region segmentation device based on subgraph feature fuses which characterized in that includes:
the acquisition module is used for acquiring the image data of the wet road surface or acquiring a data set disclosed by a network and generating a training set and a testing set;
the training module is used for cutting each image of the training set into a plurality of first subgraphs which have overlapping areas and are consistent in size, and sending each first subgraph into a preset neural network segmentation algorithm based on a coding and decoding structure for training to obtain a trained network;
the construction module is used for constructing a sub-graph feature quality evaluation model, extracting IoU of the prediction result of each first sub-graph, HSV features and high-grade features of the sub-graph and using the HSV features and the high-grade features for model training;
the computing module is used for cutting each image of the test set into a plurality of second sub-images which have overlapping areas and are consistent in size, sending the second sub-images into the trained network for operation, extracting high-level characteristics in the operation process, and computing characteristics to be fused for the overlapping areas of each second sub-image based on a cross attention mechanism;
the fusion module is used for calculating the weight of the features to be fused based on the sub-image feature quality evaluation model, and carrying out feature fusion of the mutually overlapped areas among the sub-images according to the weight to obtain features of which the feature fusion is completed in the overlapped areas; and
the segmentation module is used for expanding the features fused in the overlapping region to the whole subgraph to obtain expanded subgraph features, sending the expanded subgraph features into a decoder of the trained network to obtain a road surface wet and slippery region segmentation result, and expanding the features fused in the overlapping region to the whole subgraph, and comprises the following steps: treating high-level features of sub-graphs without feature fusion as in the attention mechanismMatrix sum->Matrix and regarding the high-level characteristics of the sub-graph after the overlapping area has the fusion characteristics as +.>A matrix; calculating a new feature map based on an attention mechanism to serve as a sub-graph feature after feature expansion;
the extracting IoU of the predicted result of each first sub-graph, HSV features and high-grade features of the sub-graph for model training comprises the following steps: HSV features and high-level features of a first sub-graph are subjected to global mean pooling and then serve as input of a model, ioU of a predicted result of the first sub-graph serves as a true value of an output result of the model, so that the sub-graph feature quality evaluation model is trained, and the trained model can output sub-graph feature quality evaluation scores.
5. The apparatus of claim 4, wherein the weight is calculated by the formula:
wherein ,is->Zhang Zitu weight in feature fusion, < ->For the number of subgraphs involved in feature fusion, +.>Is->Zhang Zitu the quality evaluation score is an exponentially amplified value.
6. The apparatus of claim 4, wherein the learning rate update formula of the sub-graph feature quality assessment model is:
wherein ,for the total number of iterations +.>The current iteration number.
7. An electronic device, comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the road surface wet skid region segmentation method based on sub-graph feature fusion as claimed in any one of claims 1 to 3.
8. A computer-readable storage medium having stored thereon a computer program, characterized in that the program is executed by a processor for implementing the road surface wet skid region segmentation method based on sub-graph feature fusion as claimed in any one of claims 1 to 3.
CN202310764266.2A 2023-06-27 2023-06-27 Road surface wet and slippery region segmentation method and device based on sub-graph feature fusion Active CN116503606B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310764266.2A CN116503606B (en) 2023-06-27 2023-06-27 Road surface wet and slippery region segmentation method and device based on sub-graph feature fusion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310764266.2A CN116503606B (en) 2023-06-27 2023-06-27 Road surface wet and slippery region segmentation method and device based on sub-graph feature fusion

Publications (2)

Publication Number Publication Date
CN116503606A CN116503606A (en) 2023-07-28
CN116503606B true CN116503606B (en) 2023-08-29

Family

ID=87320590

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310764266.2A Active CN116503606B (en) 2023-06-27 2023-06-27 Road surface wet and slippery region segmentation method and device based on sub-graph feature fusion

Country Status (1)

Country Link
CN (1) CN116503606B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113065578A (en) * 2021-03-10 2021-07-02 合肥市正茂科技有限公司 Image visual semantic segmentation method based on double-path region attention coding and decoding
US11222217B1 (en) * 2020-08-14 2022-01-11 Tsinghua University Detection method using fusion network based on attention mechanism, and terminal device
CN114937033A (en) * 2022-06-27 2022-08-23 辽宁工程技术大学 Rural highway pavement disease intelligent detection method based on deep convolutional neural network
CN115393695A (en) * 2021-05-20 2022-11-25 中移动信息技术有限公司 Method and device for evaluating quality of face image, electronic equipment and storage medium

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11222217B1 (en) * 2020-08-14 2022-01-11 Tsinghua University Detection method using fusion network based on attention mechanism, and terminal device
CN113065578A (en) * 2021-03-10 2021-07-02 合肥市正茂科技有限公司 Image visual semantic segmentation method based on double-path region attention coding and decoding
CN115393695A (en) * 2021-05-20 2022-11-25 中移动信息技术有限公司 Method and device for evaluating quality of face image, electronic equipment and storage medium
CN114937033A (en) * 2022-06-27 2022-08-23 辽宁工程技术大学 Rural highway pavement disease intelligent detection method based on deep convolutional neural network

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于特征融合的实时语义分割算法;蔡雨;黄学功;张志安;朱新年;马祥;;激光与光电子学进展;57(2);全文 *

Also Published As

Publication number Publication date
CN116503606A (en) 2023-07-28

Similar Documents

Publication Publication Date Title
KR102373456B1 (en) Learning method and learning device, and testing method and testing device for detecting parking spaces by using point regression results and relationship between points to thereby provide an auto-parking system
CN106570453B (en) Method, device and system for pedestrian detection
CN108230731B (en) Parking lot navigation system and method
CN110796007B (en) Scene recognition method and computing device
CN113033604B (en) Vehicle detection method, system and storage medium based on SF-YOLOv4 network model
KR20200100810A (en) Own vehicle location prediction method and autonomous driving device for autonomous driving
JP2022505759A (en) Methods and equipment for testing driver assistance systems
JP6846070B2 (en) Learning methods and devices that provide functional safety by alerting drivers about potential risk situations using explainable artificial intelligence that verifies the detection process of autonomous driving networks, and using them. Sting method and testing apparatus {LEARNING mETHOD aND LEARNING dEVICE FOR PROVIDING FUNCTIONAL SAFETY BY WARNING DRIVER ABOUT POTENTIAL DANGEROUS SITUATION BY USING EXPLAINABLE AI WHICH VERIFIES DETECTION PROCESSES OF AUTONOMOUS DRIVING NETWORK, aND tESTING mETHOD aND tESTING dEVICE USING THE SAME}
CN113343461A (en) Simulation method and device for automatic driving vehicle, electronic equipment and storage medium
WO2020007589A1 (en) Training a deep convolutional neural network for individual routes
CN111694917A (en) Vehicle abnormal track detection and model training method and device
CN113935143A (en) Estimating collision probability by increasing severity level of autonomous vehicle
CN111062405A (en) Method and device for training image recognition model and image recognition method and device
JP2023540613A (en) Method and system for testing driver assistance systems
CN114495060B (en) Road traffic marking recognition method and device
CN116503606B (en) Road surface wet and slippery region segmentation method and device based on sub-graph feature fusion
CN111507151B (en) Support method and apparatus for enabling evaluation of automatic travel object detection process
JP7207227B2 (en) DRIVING ACTION EVALUATION DEVICE, DRIVING ACTION EVALUATION METHOD, AND DRIVING ACTION EVALUATION PROGRAM
CN111143424B (en) Feature scene data mining method, device and terminal
CN116343148A (en) Lane line detection method, device, vehicle and storage medium
Bougharriou et al. Vehicles distance estimation using detection of vanishing point
US20240037296A1 (en) Comparison of digital representations of driving situations of a vehicle
CN116129380A (en) Method and system for judging driving lane of automatic driving vehicle, vehicle and storage medium
CN115171066A (en) Method, device and equipment for determining perception risk and storage medium
CN111126271B (en) Bayonet snap image vehicle detection method, computer storage medium and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant