CN114120289B - Method and system for identifying driving area and lane line - Google Patents
Method and system for identifying driving area and lane line Download PDFInfo
- Publication number
- CN114120289B CN114120289B CN202210083455.9A CN202210083455A CN114120289B CN 114120289 B CN114120289 B CN 114120289B CN 202210083455 A CN202210083455 A CN 202210083455A CN 114120289 B CN114120289 B CN 114120289B
- Authority
- CN
- China
- Prior art keywords
- semantic
- sample
- lane line
- branch network
- features
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Abstract
The invention provides a method and a system for identifying a driving area and a lane line, wherein the method comprises the following steps: acquiring a structured road image to be identified; inputting a structural road image to be recognized into a road recognition model to obtain global semantic features output by a basic network in the road recognition model; extracting the driving region in the global semantic features through a first branch network of the road recognition model to obtain the semantic features of the driving region; extracting the lane lines in the global semantic features through a second branch network of the road recognition model to obtain lane line semantic features; and classifying and predicting the fusion features through a prediction layer in the road recognition model to obtain a driving area and a lane line in the structural road image to be recognized, wherein the fusion features are obtained by fusing global semantic features, the driving area semantic features and the lane line semantic features. The invention effectively improves the identification precision of the driving area and the lane line.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a method and a system for identifying a driving area and a lane line.
Background
Structured roads refer to driving roads with relatively regular edges, flat surfaces, distinct lane lines and other artificial markings, such as: highways, urban arterial roads, etc. The road has clear road marking lines, the background environment of the road is single, and the geometric characteristics of the road are obvious. Therefore, the detection of lane lines or driving areas can be simplified for the structured road detection problem.
Identification of driving areas and lane lines of a structured road is one of the subjects of intense research in the field of intelligent transportation, and is also an important research direction for pattern identification application based on computer vision. The task needs to divide a driving area and a lane line in a structured road image, classify the driving area and the lane line at a pixel level, and provide an upstream technical basis for other downstream tasks, such as an advanced driving assistance system, intelligent traffic violation detection and the like.
Because the structured road scene presents various different forms, and the driving areas and the lane lines in different scenes have similarity, the existing method for identifying the lane lines and the driving areas still has more missed detection and false detection, and the identification precision is lower. Therefore, a method and a system for identifying a driving area and a lane line are needed to solve the above problems.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a method and a system for identifying a driving area and a lane line.
The invention provides a method for identifying a driving area and a lane line, which comprises the following steps:
acquiring a structured road image to be identified;
inputting the structural road image to be recognized into a road recognition model to obtain global semantic features output by a basic network in the road recognition model;
extracting the driving area in the global semantic features through a first branch network of the road recognition model to obtain the semantic features of the driving area; extracting the lane lines in the global semantic features through a second branch network of the road recognition model to obtain lane line semantic features;
classifying and predicting fusion characteristics through a prediction layer in the road recognition model to obtain a driving area and a lane line in the structural road image to be recognized, wherein the fusion characteristics are obtained by fusing the global semantic characteristics, the driving area semantic characteristics and the lane line semantic characteristics;
wherein the base network, the first branch network and the second branch network are constructed by a convolutional neural network.
According to the method for identifying the driving area and the lane line provided by the invention, after the fusion features are classified and predicted through the prediction layer in the road recognition model to obtain the driving area and the lane line in the structural road image to be recognized, the method further comprises the following steps:
and performing edge extraction processing and mean value correction processing on the prediction results of the driving area and the lane line output by the road recognition model to obtain the processed prediction results.
According to the method for identifying the driving area and the lane line, provided by the invention, the algorithm of the edge extraction processing is a Canny edge detection algorithm.
According to the method for identifying the driving area and the lane line, provided by the invention, the road identification model is obtained by training the following steps:
constructing a training sample set through a structured road sample image marked with a driving area label and a lane line label;
inputting the sample images in the training sample set into a first convolutional neural network for training to obtain a basic network and global semantic sample characteristics output by the basic network;
inputting the global semantic sample characteristics into a second convolutional neural network to perform extraction training of driving area pixel characteristics to obtain a first branch network and driving area semantic sample characteristics output by the first branch network;
inputting the global semantic sample characteristics into a third convolutional neural network for training so as to extract and train lane line pixel characteristics, and obtaining a second branch network and lane line semantic sample characteristics output by the second branch network;
performing interactive training on the first branch network and the second branch network based on the driving area semantic sample characteristics and the lane line semantic sample characteristics to obtain a trained first branch network and a trained second branch network, and outputting local semantic enhancement sample characteristics;
fine-tuning the pre-trained prediction layer through the global semantic sample characteristics and the local semantic enhancement sample characteristics to obtain a trained prediction layer;
and constructing a road recognition model according to the basic network, the trained first branch network, the trained second branch network and the trained prediction layer.
According to the method for identifying the driving area and the lane line, provided by the invention, the first convolution neural network is an HRNetV2 model.
According to the method for identifying the driving area and the lane line provided by the invention, the interactive training is carried out on the first branch network and the second branch network based on the semantic sample characteristics of the driving area and the semantic sample characteristics of the lane line to obtain the trained first branch network and the trained second branch network, and the local semantic enhancement sample characteristics are output, and the method comprises the following steps:
semantic feature alignment processing is carried out on the driving area semantic sample features corresponding to the same recognition area in the sample image of the first branch network based on the lane line semantic sample features, and first local semantic enhancement sample features are obtained;
on the basis of the semantic sample features of the driving area, semantic feature alignment processing is carried out on the lane line semantic sample features corresponding to the same identification area in the sample image of the second branch network, and second local semantic enhancement sample features are obtained;
the fine tuning of the pre-trained prediction layer through the global semantic sample features and the local semantic enhancement sample features to obtain the trained prediction layer includes:
and fine-tuning the pre-trained prediction layer through the global semantic sample characteristic, the first local semantic enhancement sample characteristic and the second local semantic enhancement sample characteristic to obtain the trained prediction layer.
According to the method for identifying the driving area and the lane line provided by the invention, before the pre-trained prediction layer is finely adjusted through the global semantic sample feature, the first local semantic enhancement sample feature and the second local semantic enhancement sample feature to obtain the trained prediction layer, the method further comprises the following steps:
splicing the global semantic sample feature, the first local semantic enhancement sample feature and the second local semantic enhancement sample feature to obtain a spliced sample feature;
inputting the spliced sample characteristics into a layer of 1 × 1 convolution to obtain sample fusion characteristics, and finely adjusting the pre-trained prediction layer through the sample fusion characteristics to obtain the trained prediction layer.
The invention also provides a system for identifying the driving area and the lane line, which comprises:
the road image acquisition module is used for acquiring a structured road image to be identified;
the first processing module is used for inputting the structural road image to be recognized into a road recognition model to obtain global semantic features output by a basic network in the road recognition model;
the second processing module is used for extracting the driving area in the global semantic features through the first branch network of the road recognition model to obtain the semantic features of the driving area; extracting the lane lines in the global semantic features through a second branch network of the road recognition model to obtain lane line semantic features;
the driving region and lane line identification module is used for classifying and predicting fusion characteristics through a prediction layer in the road identification model to obtain a driving region and a lane line in the structural road image to be identified, wherein the fusion characteristics are obtained by fusing the global semantic characteristics, the driving region semantic characteristics and the lane line semantic characteristics;
wherein the basic network, the first branch network and the second branch network are constructed by a convolutional neural network.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor executes the program to realize the steps of any one of the driving area and lane line identification methods.
The invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method for identifying driving areas and lane lines as described in any of the above.
According to the method and the system for identifying the driving area and the lane line, the road identification model is constructed based on the convolutional neural network, and the basic network and the branch network in the model are jointly trained, so that the global semantic feature representation and the local semantic feature representation are realized, the robustness and the generalization of the model to the identification of the driving area and the lane line are improved, the undetected rate and the false detection rate of the road identification are reduced, and the identification precision of the driving area and the lane line is effectively improved.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic flow chart of a method for identifying a driving area and a lane line according to the present invention;
FIG. 2 is a schematic diagram illustrating post-processing of a driving area and lane line recognition result of a structured road according to the present invention;
FIG. 3 is a schematic diagram of an overall structure of a driving area and lane line identification model of a structured road provided in the present invention;
FIG. 4 is a schematic structural diagram of a driving area and lane line identification system according to the present invention;
fig. 5 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Because the structured road scene presents various different forms, and the driving areas in different scenes have similarity with the lane lines, the characteristic representation of the driving areas and the lane lines needs higher expression capability and discrimination capability to complete the task. Deep learning is a popular research direction in the field of machine learning in recent years, has obtained great success in the fields of computer vision, natural language processing and the like, and particularly, a deep convolutional neural network can extract features with rich semantic information and strong discrimination from a two-dimensional image by virtue of three structural characteristics of local receptive field, shared weight and feature downsampling, and has excellent performance in large-scale image classification and object positioning, detection and segmentation tasks. The invention designs a deep convolutional neural network, and through the joint training of a basic network and a branch interactive network, the robustness and the generalization of a road recognition model on the recognition of a driving area and a lane line can be improved through a feature representation with global and local semantics, so that the recognition accuracy of the driving area and the lane line is effectively improved.
Fig. 1 is a schematic flow chart of a method for identifying a driving area and a lane line provided by the present invention, and as shown in fig. 1, the present invention provides a method for identifying a driving area and a lane line, including:
In the invention, the road in front of the target vehicle is shot by the forward camera arranged on the vehicle, so that the structured road image to be identified is acquired, and the functions of driving assistance or automatic driving and the like are realized through the subsequent road identification process. It should be noted that, in the present invention, the structured road image to be identified may also be collected at a fixed point by a wayside camera arranged on a structured road such as an expressway or an urban arterial road, and the collected image is identified, so that the present invention may be used for detecting driving violations of vehicles.
In the invention, the basic model is obtained by training the HRNetV2 model. The HRNetV2 model takes a high-resolution convolution flow as a first stage, gradually adds the high-resolution convolution flow to a low-resolution convolution flow one by one to form a new stage, and connects the multi-resolution convolution flow in parallel to enable the resolution of the parallel flow in the later stage to be composed of the resolution of the previous stage and the lower resolution, and finally extracts the pixel feature representation retaining the image detail information, namely the basic model extracts the global semantic features comprising the driving area and the corresponding lane lines from the structural road image to be identified.
103, extracting a driving area in the global semantic features through a first branch network of the road identification model to obtain the semantic features of the driving area; and extracting the lane lines in the global semantic features through a second branch network of the road recognition model to obtain lane line semantic features.
In the invention, two branch networks respectively extract global semantic feature representations of respective corresponding categories to represent each pixel feature in an image, namely a first branch network extracts feature maps of all pixels in a driving area, and a second branch network extracts feature maps of all pixels in a lane line. Preferably, in the present invention, the pixel features (i.e. the driving area semantic features and the lane line semantic features) extracted by each of the two branch networks may assist feature extraction of the other branch network, that is, the first branch network transmits information of a driving area to the second branch network, and the second branch network transmits information of a lane line to the first branch network, so that the two branch networks extract enhanced local semantic feature representations.
104, classifying and predicting fusion features through a prediction layer in the road recognition model to obtain a driving area and a lane line in the structural road image to be recognized, wherein the fusion features are obtained by fusing the global semantic features, the driving area semantic features and the lane line semantic features;
wherein the base network, the first branch network and the second branch network are constructed by a convolutional neural network.
In the invention, the global semantic features, the driving area semantic features and the lane line semantic features obtained in the embodiment are subjected to channel fusion, and the pixel level classification prediction is carried out. Preferably, the enhanced local semantic feature representations output by the two branch networks in the above embodiment are spliced with the pixel feature representation (i.e., global semantic feature) output by the base network according to the channel dimension, and the number of channels is changed by a layer of 1 × 1 convolution, so that the input feature dimension is 512 and the output feature dimension is 256, and thus, pixel prediction is performed through a prediction layer in the road recognition model, and finally, the recognition results of the driving area and the lane line area are output respectively.
According to the method for identifying the driving area and the lane line, the road identification model is constructed based on the convolutional neural network, and the basic network and the branch network in the model are jointly trained, so that the global and local semantic feature representation is realized, the robustness and the generalization of the model to the identification of the driving area and the lane line are improved, the undetected rate and the false undetected rate of the road identification are reduced, and the identification precision of the driving area and the lane line is effectively improved.
On the basis of the above embodiment, after the fusion features are classified and predicted by the prediction layer in the road recognition model to obtain the driving area and the lane line in the structural road image to be recognized, the method further includes:
and performing edge extraction processing and mean value correction processing on the prediction results of the driving area and the lane line output by the road recognition model to obtain the processed prediction results.
In the invention, in order to keep the robustness of the prediction of a driving area and a lane line in a structured road image and prevent the sudden change of the curvature in an application scene in a short time, edge extraction and mean value correction are used for post-processing the prediction result of the deep convolutional neural network, wherein the algorithm of the edge extraction processing is a Canny edge detection algorithm, so that the accuracy and the robustness of the prediction result are enhanced; meanwhile, the change gradual process of the recognition target in the application scene is reserved. Fig. 2 is a schematic diagram of post-processing of the driving area and lane line identification result of the structured road provided by the present invention, which can refer to fig. 2, and for the driving area obtained by prediction, firstly, edge extraction is performed, and then a final driving area prediction result is obtained through mean value correction processing; and for the predicted lane line, carrying out mean value correction on the prediction result to obtain a final lane line prediction result.
On the basis of the above embodiment, the road recognition model is obtained by training through the following steps:
and constructing a training sample set through the structured road sample image marked with the driving area label and the lane line label.
In the invention, in order to obtain the labeled pixel level label, for each structured road sample image, a driving area in the image and lane line types (such as white dotted lines or yellow solid lines) on two sides of the driving area are labeled. After the labeling is finished, the structured road category of each pixel on the original sample image is obtained according to the labeling result, and therefore a training sample set for segmenting the driving area and the lane line is constructed and obtained.
And inputting the sample images in the training sample set into a first convolutional neural network for training to obtain a basic network and global semantic sample characteristics output by the basic network.
In the invention, the first convolution neural network is an HRNetV2 model, the labeled structured road sample image is input into an HRNetV2 model for training, and the global semantic features of the structured road sample image are extracted. And guiding learning by using a training sample set segmented by a driving area and a lane line through the labeled pixel level label, training to obtain a basic network after meeting the preset training times, and outputting the global semantic sample characteristic representation in the structured road sample image.
In the invention, in order to keep the detail information of the structured road image, a high-resolution basic network is adopted to extract the characteristics. Specifically, the basic network is obtained by training an HRNetV2 model, and the model is obtained by gradually adding sub-networks of low-resolution feature maps in parallel to a main network of a high-resolution feature map, realizing multi-scale fusion and feature extraction on different networks, and finally extracting pixel feature representations retaining image detail information.
Inputting the global semantic sample characteristics into a second convolutional neural network to perform extraction training of driving area pixel characteristics to obtain a first branch network and driving area semantic sample characteristics output by the first branch network;
inputting the global semantic sample characteristics into a third convolutional neural network for training so as to extract and train lane line pixel characteristics, and obtaining a second branch network and lane line semantic sample characteristics output by the second branch network;
and performing interactive training on the first branch network and the second branch network based on the driving area semantic sample characteristics and the lane line semantic sample characteristics to obtain a trained first branch network and a trained second branch network, and outputting local semantic enhancement sample characteristics.
In the invention, in order to obtain the characteristic representation of the driving area and the position corresponding to the lane line, different convolution kernels are respectively adopted in the two branch networks to extract the lane line characteristic and the driving area characteristic. Specifically, the global semantic sample features are respectively input into two branch networks, feature representations of the corresponding positions of each class are extracted, and the feature representations are respectively extracted and output.
Specifically, in order to obtain reliable class feature representation, labeled pixel labels are used to guide the generation of semantic feature and position feature representation of each class. In the invention, a characteristic diagram corresponding to a pixel label is used to perform matrix multiplication operation with convolution kernels in a branch network, firstly, characteristic diagrams of all pixels belonging to a driving area or a lane line of the type on the image are extracted, and then, corresponding weights wi of convolution kernels Pi of main lines of each branch network are updated by continuously approaching to the minimum value of a loss function, wherein i represents the ith convolution kernel. Preferably, the category semantic features extracted by the two branch networks respectively are correspondingly interacted to complete convolution calculation of the convolution kernel between the categories, namely the features of the two branch networks are extracted, the convolution interaction is carried out through splicing, and then the features after the convolution calculation (the respective enhanced local semantic features are obtained) are output to be used as the features input by the two branches in the next training stage.
Fine-tuning the pre-trained prediction layer through the global semantic sample characteristics and the local semantic enhancement sample characteristics to obtain a trained prediction layer;
and constructing a road recognition model according to the basic network, the trained first branch network, the trained second branch network and the trained prediction layer.
In the invention, the global semantic sample characteristics, the driving area semantic sample characteristics and the lane line semantic sample characteristics are subjected to channel fusion, and the road identification model is finally obtained by performing pixel-level classification training on a pre-trained prediction layer.
On the basis of the above embodiment, the interactively training the first branch network and the second branch network based on the driving area semantic sample features and the lane line semantic sample features to obtain a trained first branch network and a trained second branch network, and outputting local semantic enhancement sample features includes:
semantic feature alignment processing is carried out on the driving area semantic sample features corresponding to the same recognition area in the sample image of the first branch network based on the lane line semantic sample features, and first local semantic enhancement sample features are obtained;
on the basis of the semantic sample features of the driving area, semantic feature alignment processing is carried out on the lane line semantic sample features corresponding to the same identification area in the sample image of the second branch network, and second local semantic enhancement sample features are obtained;
the fine tuning of the pre-trained prediction layer through the global semantic sample features and the local semantic enhancement sample features to obtain the trained prediction layer includes:
and fine-tuning the pre-trained prediction layer through the global semantic sample characteristic, the first local semantic enhancement sample characteristic and the second local semantic enhancement sample characteristic to obtain the trained prediction layer.
In the invention, the feature information of the two branch networks is interacted, and the interaction process depends on the convolution kernel between the lateral classes to carry out semantic alignment of the features, thereby obtaining the enhanced local semantic features.
Fig. 3 is a schematic diagram of an overall structure of a driving area and lane line identification model of a structured road provided by the present invention, which can refer to fig. 3, and correspondingly transmits a feature map obtained by one of the branch networks at different sampling stages to a corresponding hierarchy of the other branch network, that is, features obtained in the same area identified by the two branch networks are fused, so as to mutually output semantic features in the two branch networks. Specifically, semantic feature interaction is carried out by utilizing lateral inter-class convolution kernels of the branch networks, the semantic features of a driving area or a lane line output by one branch network are convoluted with the corresponding inter-class convolution kernels of the same identification area in the other branch network, corresponding constraint information is extracted, the driving area alpha is matched with the lane line beta, information of the driving area is transmitted to the lane line branch, and information of the lane line is transmitted to the driving area branch. The method comprises the steps that a second branch network outputs lane line semantic sample characteristics, and after convolution operation is conducted on the lane line semantic sample characteristics and an inter-class convolution kernel Qi of the first branch network at the level, first local semantic enhancement sample characteristics are obtained; correspondingly, the first branch network outputs the lane line semantic sample characteristics, and after the lane line semantic sample characteristics are convolved with the inter-class convolution kernel Qi of the level of the second branch network, second local semantic enhancement sample characteristics are obtained. And updating the corresponding weight ti of the inter-class convolution kernels Qi of the two branch network main lines by continuously approaching the minimum value of the preset loss function.
The method adopts the lane line characteristics and the driving area characteristics which are respectively extracted by the two branch networks to carry out interactive transmission so as to train the two branch networks to learn the area characteristic representation of different structured road categories, exclude other interference information, generate enhanced local characteristic representation containing the position information and the semantic information of the target category, improve the discrimination and the reliability of the characteristics and further improve the prediction accuracy of different traffic areas.
In the present invention, the corresponding weight wi of the convolution kernel Pi and the corresponding weight ti of the inter-class convolution kernel Qi are iteratively updated and calculated only during model training. In the model testing stage, the method directly adopts the finally updated convolution kernel after the channel fusion to carry out feature representation.
On the basis of the above embodiment, before the pre-trained prediction layer is fine-tuned through the global semantic sample feature, the first local semantic enhancement sample feature, and the second local semantic enhancement sample feature to obtain a trained prediction layer, the method further includes:
splicing the global semantic sample feature, the first local semantic enhancement sample feature and the second local semantic enhancement sample feature to obtain a spliced sample feature;
inputting the spliced sample characteristics into a layer of 1 × 1 convolution to obtain sample fusion characteristics, and finely adjusting the pre-trained prediction layer through the sample fusion characteristics to obtain the trained prediction layer.
In the invention, in order to combine the enhanced pixel feature representation (namely, the first local semantic enhancement sample feature and the second local semantic enhancement sample feature) with the pixel feature representation (namely, the global semantic sample feature) of the basic network, the pixel feature representation output by the basic network and the enhanced feature representation are spliced together according to the channel dimension, the number of channels is changed through a layer of 1 × 1 convolution, and then pixel prediction is carried out, so that the feature after channel fusion has both local and global semantic features, thereby increasing the discrimination of the pixel feature.
In an embodiment, the driving area and lane line identification method provided by the present invention is described in its entirety. The invention provides a method for identifying a driving area and a lane line based on coupled learning, which is a serial solution of a deep convolutional neural network and characteristic post-processing aiming at the problem of identifying the driving lane and the lane line in a structured road. The HRNeTV2 network is trained by the basic network, so that the trained basic network is used for extracting semantic-rich pixel feature representations corresponding to the driving area and the lane line; the two branch networks respectively extract global semantic feature representations of respective corresponding categories to represent each pixel feature in the structured image, and simultaneously, the pixel features output by the two branch networks are subjected to coupling learning to assist the other branch network in judging.
Furthermore, in the training process of the two branch networks, updating the feature representation of different driving areas and lane lines, firstly, under the guidance of a data set adopting labeled pixel level labels, learning the feature representation of the driving lane and lane line type areas of the whole data set by a preposed basic network to obtain a global semantic feature representation method, and inputting the feature into the two branch networks; the two branch networks respectively extract the characteristics corresponding to the lane line identification and the driving area identification, and on the basis, the characteristic transmission process of interaction of the lane line characteristics and the driving area characteristics is adopted to guide the two branch networks to learn the area characteristic representation of different structured road types, so that the interference of irrelevant information is effectively reduced, the discrimination and reliability of the characteristics are enhanced, the characteristic representation simultaneously containing global information and local information is obtained, and the prediction accuracy of different traffic areas is further improved.
In the actual application stage of the road identification model, each pixel feature is represented by using the updated feature representation, the similarity between each pixel feature and each structured road category region feature is calculated, and the feature of each pixel is represented by using the global region feature, so that the discrimination of the pixel feature is enhanced, and the identification performance is effectively improved. And finally, post-processing the prediction result output by the road recognition model, wherein the post-processing consists of two parts of edge extraction and mean value correction, so that the precision of the structured road traffic area recognition can be effectively improved, the missing detection and the false detection of the recognition are reduced, and the improvement on the recognition of a road area sensitive to global information is more remarkable.
The following describes the driving area and lane line identification system provided by the present invention, and the driving area and lane line identification system described below and the driving area and lane line identification method described above may be referred to in correspondence with each other.
Fig. 4 is a schematic structural diagram of a driving area and lane line identification system provided by the present invention, and as shown in fig. 4, the present invention provides a driving area and lane line identification system, which includes a road image acquisition module 401, a first processing module 402, a second processing module 403, and a driving area and lane line identification module 404, where the road image acquisition module 401 is configured to acquire a structural road image to be identified; the first processing module 402 is configured to input the to-be-identified structured road image into a road identification model, so as to obtain a global semantic feature output by a basic network in the road identification model; the second processing module 403 is configured to extract a driving area in the global semantic features through a first branch network of the road identification model, so as to obtain driving area semantic features; extracting the lane lines in the global semantic features through a second branch network of the road recognition model to obtain lane line semantic features; the driving area and lane line identification module 404 is configured to perform classified prediction on fusion features through a prediction layer in the road identification model to obtain a driving area and a lane line in the structural road image to be identified, where the fusion features are obtained by fusing the global semantic features, the driving area semantic features, and the lane line semantic features; wherein the basic network, the first branch network and the second branch network are constructed by a convolutional neural network.
The driving area and lane line identification system provided by the invention constructs a road identification model based on the convolutional neural network, and has global and local semantic feature representation through the joint training of the basic network and the branch network in the model, so that the robustness and the generalization of the model to the driving area and lane line identification are improved, the undetected rate and the false undetected rate of the road identification are reduced, and the accuracy of the driving area and lane line identification is effectively improved.
The system provided by the present invention is used for executing the above method embodiments, and for the specific processes and details, reference is made to the above embodiments, which are not described herein again.
Fig. 5 is a schematic structural diagram of an electronic device provided in the present invention, and as shown in fig. 5, the electronic device may include: a Processor (Processor) 501, a communication Interface (Communications Interface) 502, a Memory (Memory) 503, and a communication bus 504, wherein the Processor 501, the communication Interface 502, and the Memory 503 are configured to communicate with each other via the communication bus 504. The processor 501 may call logic instructions in the memory 503 to perform a driving area and lane line identification method, the method comprising: acquiring a structured road image to be identified; inputting the structural road image to be recognized into a road recognition model to obtain global semantic features output by a basic network in the road recognition model; extracting the driving region in the global semantic features through a first branch network of the road recognition model to obtain the semantic features of the driving region; extracting the lane lines in the global semantic features through a second branch network of the road recognition model to obtain lane line semantic features; classifying and predicting fusion characteristics through a prediction layer in the road recognition model to obtain a driving area and a lane line in the structural road image to be recognized, wherein the fusion characteristics are obtained by fusing the global semantic characteristics, the driving area semantic characteristics and the lane line semantic characteristics; wherein the base network, the first branch network and the second branch network are constructed by a convolutional neural network.
In addition, the logic instructions in the memory 503 may be implemented in the form of software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, which when executed by a computer, enable the computer to perform the driving area and lane line identification method provided by the above methods, the method comprising: acquiring a structured road image to be identified; inputting the structural road image to be recognized into a road recognition model to obtain global semantic features output by a basic network in the road recognition model; extracting the driving region in the global semantic features through a first branch network of the road recognition model to obtain the semantic features of the driving region; extracting the lane lines in the global semantic features through a second branch network of the road recognition model to obtain lane line semantic features; classifying and predicting fusion characteristics through a prediction layer in the road recognition model to obtain a driving area and a lane line in the structural road image to be recognized, wherein the fusion characteristics are obtained by fusing the global semantic characteristics, the driving area semantic characteristics and the lane line semantic characteristics; wherein the base network, the first branch network and the second branch network are constructed by a convolutional neural network.
In yet another aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, the computer program being implemented by a processor to perform the method for identifying a driving area and a lane line provided in the foregoing embodiments, the method including: acquiring a structured road image to be identified; inputting the structural road image to be recognized into a road recognition model to obtain global semantic features output by a basic network in the road recognition model; extracting the driving region in the global semantic features through a first branch network of the road recognition model to obtain the semantic features of the driving region; extracting the lane lines in the global semantic features through a second branch network of the road recognition model to obtain lane line semantic features; classifying and predicting fusion characteristics through a prediction layer in the road recognition model to obtain a driving area and a lane line in the structural road image to be recognized, wherein the fusion characteristics are obtained by fusing the global semantic characteristics, the driving area semantic characteristics and the lane line semantic characteristics; wherein the base network, the first branch network and the second branch network are constructed by a convolutional neural network.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (8)
1. A method for identifying a driving area and a lane line is characterized by comprising the following steps:
acquiring a structured road image to be identified;
inputting the structural road image to be recognized into a road recognition model to obtain global semantic features output by a basic network in the road recognition model;
extracting the driving region in the global semantic features through a first branch network of the road recognition model to obtain the semantic features of the driving region; extracting the lane lines in the global semantic features through a second branch network of the road recognition model to obtain lane line semantic features;
classifying and predicting fusion characteristics through a prediction layer in the road recognition model to obtain a driving area and a lane line in the structural road image to be recognized, wherein the fusion characteristics are obtained by fusing the global semantic characteristics, the driving area semantic characteristics and the lane line semantic characteristics;
wherein the basic network, the first branch network and the second branch network are constructed by a convolutional neural network;
the road recognition model is obtained by training through the following steps:
constructing a training sample set through a structured road sample image marked with a driving area label and a lane line label;
inputting the sample images in the training sample set into a first convolutional neural network for training to obtain a basic network and global semantic sample characteristics output by the basic network;
inputting the global semantic sample characteristics into a second convolutional neural network to perform extraction training of driving area pixel characteristics to obtain a first branch network and driving area semantic sample characteristics output by the first branch network;
inputting the global semantic sample characteristics into a third convolutional neural network for training so as to perform extraction training of lane line pixel characteristics, and obtaining a second branch network and lane line semantic sample characteristics output by the second branch network;
performing interactive training on the first branch network and the second branch network based on the driving area semantic sample characteristics and the lane line semantic sample characteristics to obtain a trained first branch network and a trained second branch network, and outputting local semantic enhancement sample characteristics;
fine-tuning the pre-trained prediction layer through the global semantic sample characteristics and the local semantic enhancement sample characteristics to obtain a trained prediction layer;
constructing a road recognition model according to the basic network, the trained first branch network, the trained second branch network and the trained prediction layer;
the interactive training is performed on the first branch network and the second branch network based on the driving area semantic sample characteristics and the lane line semantic sample characteristics to obtain a trained first branch network and a trained second branch network, and local semantic enhancement sample characteristics are output, and the method comprises the following steps:
semantic feature alignment processing is carried out on the driving area semantic sample features corresponding to the same recognition area in the sample image of the first branch network based on the lane line semantic sample features, and first local semantic enhancement sample features are obtained;
on the basis of the semantic sample features of the driving area, semantic feature alignment processing is carried out on the lane line semantic sample features corresponding to the same identification area in the sample image of the second branch network, and second local semantic enhancement sample features are obtained;
the fine tuning of the pre-trained prediction layer through the global semantic sample features and the local semantic enhancement sample features to obtain the trained prediction layer includes:
and fine-tuning the pre-trained prediction layer through the global semantic sample characteristic, the first local semantic enhancement sample characteristic and the second local semantic enhancement sample characteristic to obtain the trained prediction layer.
2. The method for identifying driving regions and lane lines according to claim 1, wherein after the classification and prediction of the fusion features by the prediction layer in the road recognition model is performed to obtain the driving regions and the lane lines in the structured road image to be identified, the method further comprises:
and performing edge extraction processing and mean value correction processing on the prediction results of the driving area and the lane line output by the road recognition model to obtain the processed prediction results.
3. The method according to claim 2, wherein the algorithm of the edge extraction process is a Canny edge detection algorithm.
4. The method for identifying driving regions and lane lines according to claim 1, wherein the first convolutional neural network is a HRNetV2 model.
5. The method for identifying driving regions and lane lines according to claim 1, wherein before the pre-trained prediction layer is fine-tuned through the global semantic sample feature, the first local semantic enhancement sample feature and the second local semantic enhancement sample feature to obtain a trained prediction layer, the method further comprises:
splicing the global semantic sample feature, the first local semantic enhancement sample feature and the second local semantic enhancement sample feature to obtain a spliced sample feature;
inputting the spliced sample characteristics into a layer of 1 × 1 convolution to obtain sample fusion characteristics, and finely adjusting the pre-trained prediction layer through the sample fusion characteristics to obtain the trained prediction layer.
6. The utility model provides a driving region and lane line identification system which characterized in that includes:
the road image acquisition module is used for acquiring a structured road image to be identified;
the first processing module is used for inputting the structural road image to be recognized into a road recognition model to obtain global semantic features output by a basic network in the road recognition model;
the second processing module is used for extracting the driving region in the global semantic features through the first branch network of the road recognition model to obtain the semantic features of the driving region; extracting the lane lines in the global semantic features through a second branch network of the road recognition model to obtain lane line semantic features;
the driving region and lane line identification module is used for classifying and predicting fusion characteristics through a prediction layer in the road identification model to obtain a driving region and a lane line in the structural road image to be identified, wherein the fusion characteristics are obtained by fusing the global semantic characteristics, the driving region semantic characteristics and the lane line semantic characteristics;
wherein the basic network, the first branch network and the second branch network are constructed by a convolutional neural network;
the road recognition model is obtained by training through the following steps:
constructing a training sample set through a structured road sample image marked with a driving area label and a lane line label;
inputting the sample images in the training sample set into a first convolutional neural network for training to obtain a basic network and global semantic sample characteristics output by the basic network;
inputting the global semantic sample characteristics into a second convolutional neural network to perform extraction training of driving area pixel characteristics to obtain a first branch network and driving area semantic sample characteristics output by the first branch network;
inputting the global semantic sample characteristics into a third convolutional neural network for training so as to perform extraction training of lane line pixel characteristics, and obtaining a second branch network and lane line semantic sample characteristics output by the second branch network;
performing interactive training on the first branch network and the second branch network based on the driving area semantic sample characteristics and the lane line semantic sample characteristics to obtain a trained first branch network and a trained second branch network, and outputting local semantic enhancement sample characteristics;
fine-tuning the pre-trained prediction layer through the global semantic sample characteristics and the local semantic enhancement sample characteristics to obtain a trained prediction layer;
constructing a road recognition model according to the basic network, the trained first branch network, the trained second branch network and the trained prediction layer;
the interactive training is performed on the first branch network and the second branch network based on the driving area semantic sample characteristics and the lane line semantic sample characteristics to obtain a trained first branch network and a trained second branch network, and local semantic enhancement sample characteristics are output, and the method comprises the following steps:
semantic feature alignment processing is carried out on the driving area semantic sample features corresponding to the same recognition area in the sample image of the first branch network based on the lane line semantic sample features, and first local semantic enhancement sample features are obtained;
on the basis of the semantic sample features of the driving area, semantic feature alignment processing is carried out on the lane line semantic sample features corresponding to the same identification area in the sample image of the second branch network, and second local semantic enhancement sample features are obtained;
the fine tuning of the pre-trained prediction layer through the global semantic sample features and the local semantic enhancement sample features to obtain the trained prediction layer includes:
and fine-tuning the pre-trained prediction layer through the global semantic sample characteristic, the first local semantic enhancement sample characteristic and the second local semantic enhancement sample characteristic to obtain the trained prediction layer.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the driving area and lane line identification method according to any one of claims 1 to 5 when executing the computer program.
8. A non-transitory computer readable storage medium, on which a computer program is stored, wherein the computer program, when being executed by a processor, implements the steps of the driving area and lane line identification method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210083455.9A CN114120289B (en) | 2022-01-25 | 2022-01-25 | Method and system for identifying driving area and lane line |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210083455.9A CN114120289B (en) | 2022-01-25 | 2022-01-25 | Method and system for identifying driving area and lane line |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114120289A CN114120289A (en) | 2022-03-01 |
| CN114120289B true CN114120289B (en) | 2022-05-03 |
Family
ID=80360878
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210083455.9A Active CN114120289B (en) | 2022-01-25 | 2022-01-25 | Method and system for identifying driving area and lane line |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114120289B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114581806B (en) * | 2022-03-18 | 2024-03-19 | 重庆科技学院 | Industrial part empty rate calculation method based on trunk edge feature extraction |
| CN114821531B (en) * | 2022-04-25 | 2023-03-28 | 广州优创电子有限公司 | Lane line recognition image display system based on electronic exterior rearview mirror ADAS |
| CN115147801B (en) | 2022-08-29 | 2022-12-23 | 北京百度网讯科技有限公司 | Lane line recognition method and device, electronic device and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107169527A (en) * | 2017-06-06 | 2017-09-15 | 西北工业大学 | Classification method of medical image based on collaboration deep learning |
| CN109086871A (en) * | 2018-07-27 | 2018-12-25 | 北京迈格威科技有限公司 | Training method, device, electronic equipment and the computer-readable medium of neural network |
| WO2021007812A1 (en) * | 2019-07-17 | 2021-01-21 | 深圳大学 | Deep neural network hyperparameter optimization method, electronic device and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111460921B (en) * | 2020-03-13 | 2023-05-26 | 华南理工大学 | Lane line detection method based on multitasking semantic segmentation |
| CN111144388B (en) * | 2020-04-03 | 2020-07-14 | 速度时空信息科技股份有限公司 | Monocular image-based road sign line updating method |
| CN113392812B (en) * | 2021-07-08 | 2022-06-07 | 湖南大学 | Road lane line detection method and system based on deep neural network |
| CN113688836A (en) * | 2021-09-28 | 2021-11-23 | 四川大学 | Real-time road image semantic segmentation method and system based on deep learning |
-
2022
- 2022-01-25 CN CN202210083455.9A patent/CN114120289B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107169527A (en) * | 2017-06-06 | 2017-09-15 | 西北工业大学 | Classification method of medical image based on collaboration deep learning |
| CN109086871A (en) * | 2018-07-27 | 2018-12-25 | 北京迈格威科技有限公司 | Training method, device, electronic equipment and the computer-readable medium of neural network |
| WO2021007812A1 (en) * | 2019-07-17 | 2021-01-21 | 深圳大学 | Deep neural network hyperparameter optimization method, electronic device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114120289A (en) | 2022-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114120289B (en) | Method and system for identifying driving area and lane line | |
| Sirohi et al. | Convolutional neural networks for 5G-enabled intelligent transportation system: A systematic review | |
| CN111666921B (en) | Vehicle control method, apparatus, computer device, and computer-readable storage medium | |
| CN111598095A (en) | Deep learning-based urban road scene semantic segmentation method | |
| CN112380921A (en) | Road detection method based on Internet of vehicles | |
| CN110263786B (en) | Road multi-target identification system and method based on feature dimension fusion | |
| Kanagaraj et al. | Deep learning using computer vision in self driving cars for lane and traffic sign detection | |
| Ojha et al. | Vehicle detection through instance segmentation using mask R-CNN for intelligent vehicle system | |
| CN111008632B (en) | License plate character segmentation method based on deep learning | |
| Rateke et al. | Passive vision region-based road detection: A literature review | |
| Cao et al. | An end-to-end neural network for multi-line license plate recognition | |
| Huang et al. | Measuring the absolute distance of a front vehicle from an in-car camera based on monocular vision and instance segmentation | |
| Xing et al. | The Improved Framework for Traffic Sign Recognition Using Guided Image Filtering | |
| CN114973199A (en) | Rail transit train obstacle detection method based on convolutional neural network | |
| CN115100469A (en) | Target attribute identification method, training method and device based on segmentation algorithm | |
| CN109241893B (en) | Road selection method and device based on artificial intelligence technology and readable storage medium | |
| Yu et al. | SignHRNet: Street-level traffic signs recognition with an attentive semi-anchoring guided high-resolution network | |
| CN112288701A (en) | Intelligent traffic image detection method | |
| CN116883650A (en) | Image-level weak supervision semantic segmentation method based on attention and local stitching | |
| Saravanarajan et al. | Improving semantic segmentation under hazy weather for autonomous vehicles using explainable artificial intelligence and adaptive dehazing approach | |
| Pan et al. | A Hybrid Deep Learning Algorithm for the License Plate Detection and Recognition in Vehicle-to-Vehicle Communications | |
| Luo et al. | AD-RoadNet: an auxiliary-decoding road extraction network improving connectivity while preserving multiscale road details | |
| CN114495050A (en) | Multitask integrated detection method for automatic driving forward vision detection | |
| CN114359572A (en) | Training method and device of multi-task detection model and terminal equipment | |
| Tumuluru et al. | SMS: SIGNS MAY SAVE–Traffic Sign Recognition and Detection using CNN |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |