WO2021249071A1 - 一种车道线的检测方法及相关设备 - Google Patents

一种车道线的检测方法及相关设备 Download PDF

Info

Publication number
WO2021249071A1
WO2021249071A1 PCT/CN2021/092070 CN2021092070W WO2021249071A1 WO 2021249071 A1 WO2021249071 A1 WO 2021249071A1 CN 2021092070 W CN2021092070 W CN 2021092070W WO 2021249071 A1 WO2021249071 A1 WO 2021249071A1
Authority
WO
WIPO (PCT)
Prior art keywords
lane line
grid
predicted
feature
grids
Prior art date
Application number
PCT/CN2021/092070
Other languages
English (en)
French (fr)
Inventor
蔡信岳
徐航
张维
杨臻
李震国
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP21821650.5A priority Critical patent/EP4152204A4/en
Publication of WO2021249071A1 publication Critical patent/WO2021249071A1/zh
Priority to US18/064,016 priority patent/US20230144209A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/771—Feature selection, e.g. selecting representative features from a multi-dimensional feature space
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/803—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of input or preprocessed data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle
    • G06T2207/30256—Lane; Road marking

Definitions

  • This application relates to the field of artificial intelligence, and in particular to a method for detecting lane lines and related equipment.
  • Intelligent driving such as automatic driving, assisted driving, etc.
  • technologies rely on artificial intelligence, visual computing, radar, monitoring devices and global positioning systems to cooperate, so that vehicles can achieve automatic driving without the need for human active operation.
  • the first step is the collection and processing of environmental information, and the lane line, as one of the most important indication information of the road, can effectively guide the intelligent vehicle to drive in the restricted road area. Therefore, how to detect the lane lines of the road in real time and accurately is an important link in the design of intelligent vehicle-related systems, which can assist in route planning, road deviation warning and other functions, and can provide a reference for accurate navigation.
  • the commonly used lane line detection method is lane line detection based on convolution features.
  • the image to be detected is input into a neural network for feature extraction, and then the extracted features (each feature map is divided into multiple in advance) Grid) is decoded by the prediction head model to generate dense line clusters (that is, multiple predicted lane lines), and finally, according to the confidence of each predicted lane line (also called the confidence of the grid, the confidence response is Whether there is a lane line passing through the grid and the probability of passing through the grid, a grid with a confidence level greater than a preset value is used to predict the lane line, and a grid with a confidence level lower than the preset value is used to predict the lane line.
  • Sort the line clusters by taking the value of the predicted lane line with the highest confidence value as the baseline, and grouping the lane lines into a group on the condition that the distance between other predicted lane lines and the baseline is less than the threshold value , Divide the line clusters into several groups in a similar way, and take the baseline in each group as the final output of the detection result of a real lane line in this group.
  • each grid can accurately predict an entire lane line passing through this grid, but the actual lane line span is too long, and the part of the lane line far away from the center of the grid is difficult. Effective prediction.
  • the embodiment of the application provides a method for detecting lane lines and related equipment, which are used to perform feature fusion on the first feature maps output by different layers of the neural network after training, and the second feature map obtained is based on the second feature
  • the multiple lane line prediction results obtained from the map are integrated to improve the accuracy of lane line detection.
  • the embodiments of the present application first provide a method for detecting lane lines, which can be used in the field of image processing.
  • the method includes: first, the detection device performs feature fusion on the first feature maps output by different layers of the trained neural network , One or more second feature maps are obtained, and the first one or more second feature maps are the merged feature maps. This is because the performance of feature maps extracted from different layers of the neural network is different.
  • the low-level feature maps have higher resolution and contain more position and detailed information, but because there are fewer convolutions, they have lower semantics and more noise; high-level features Feature maps have stronger semantic information, but have low resolution and poor perception of details.
  • the second feature map obtained has multi-level features.
  • it can be divided into multiple grids. If there are grids with a confidence value greater than a preset threshold (for example, the preset threshold is 0.6) in the multiple grids, then The confidence level greater than the preset threshold may be referred to as the first confidence level, and the grid corresponding to the first confidence level may be referred to as the first grid.
  • a predicted lane line corresponding to each first grid (that is, the first predicted lane line) can be further obtained through the trained prediction head model.
  • n first grids corresponding to the n first grids are obtained according to the prediction head model. After the first predicted lane line, it is necessary to divide the predicted n first predicted lane lines into m groups, where each group includes at least one predicted lane line.
  • the q first confidence levels corresponding to the q first predicted lane lines and the corresponding q first grids are paired with the q
  • a predicted lane line is integrated to obtain a second predicted lane line, and the second predicted lane line is output as a detection result of a real lane line in the second feature map.
  • the output second feature map merges the multiple levels of features of the first feature map output by different layers of the neural network, and the second feature map has both low-level features.
  • the relevant features of the receptive field make it conducive to the accurate regression of the lane line, and the relevant features of the high-level receptive field make it easy to judge whether the lane line exists;
  • group the output predicted lane line set in each group Due to the limitation of the receptive field, each predicted lane line has its own optimal prediction interval.
  • the embodiment of this application selects the optimal prediction interval of each line to integrate the lane lines, and finally combines the advantages of different predicted lane lines. Form the final output predicted lane line.
  • the q first predicted lane lines can be occupied in the second feature map.
  • the area is divided into a plurality of sub-areas, and each sub-area includes the center point position (anchor point position) of at least one of the q first grids, if the first sub-area of the plurality of sub-areas includes at least two
  • the anchor point positions of the first grids one first grid is selected as the second grid from the at least two first grids according to the first confidence levels corresponding to the at least two first grids, for example, From the at least two first grids, the first grid with the largest first confidence value is selected as the second grid, and the portion of the first predicted lane line corresponding to the second grid in the first sub-region is selected as the second grid.
  • the first part of the second predicted lane line, or the weighted operation is performed on the part of the first predicted lane line corresponding to the at least two first grids in the first subregion, and the obtained calculation result is used as the second predicted lane line If the second sub-region of the multiple sub-regions only includes the anchor point position of the first grid, then the first predicted lane line corresponding to the first grid included in the second sub-region is located in the first The part of the two sub-regions is used as the second part of the second predicted lane line; finally, the first part and the second part are integrated to obtain the final output second predicted lane line.
  • selecting a first grid from the at least two first grids as the second grid according to the first confidence levels respectively corresponding to the at least two first grids may specifically be: At least two first grids select the first grid with the largest first confidence level as the second grid.
  • the first predicted lane line in the first group is directly used as the comparison to the second feature map.
  • the detection result of a real lane line is output.
  • the network structure of the neural network can have multiple choices, and there are also multiple feature fusion models built by the embodiment of this application, so How to choose a neural network (which can be called a target neural network) and a feature fusion model (which can be called a target feature fusion model) from a variety of neural networks and multiple feature fusion models is very important.
  • a neural network which can be called a target neural network
  • a feature fusion model which can be called a target feature fusion model
  • it can be obtained by, but not limited to, the following methods: search for a set of neural network + feature fusion model from a search space composed of a variety of existing neural networks and a variety of feature fusion models constructed by this application The combination.
  • each search process includes sampling from the search space, sampling Then you can get a string representing "neural network + feature fusion model” (ie a code combination), and generate a specific instance of "neural network + feature fusion model” based on this string, that is, the specific structure of a lane line detection network ; Afterwards, this instance is trained through the training data in the training set to obtain the trained lane line detection network, and after the training is completed, the performance evaluation of the trained lane line detection network is performed to obtain the network performance.
  • a string representing "neural network + feature fusion model” ie a code combination
  • this instance is trained through the training data in the training set to obtain the trained lane line detection network, and after the training is completed, the performance evaluation of the trained lane line detection network is performed to obtain the network performance.
  • the detection network includes the corresponding target neural network and target feature fusion model.
  • the target neural network in the target lane line detection network can be used to perform feature extraction on the image to be detected, and the first feature map output by the different layers of the target neural network can be obtained, and then the first feature map output by the different layers can be used as the target The input of the feature fusion model, and the output of the second feature map by the target feature fusion model.
  • the target neural network and the target feature fusion model meeting the requirements can be obtained by searching.
  • This search method can merge from a large number of neural networks and features
  • the relevant structure of the target neural network and the target feature fusion model that best match the detection task is selected in the model, which indirectly improves the detection performance of the model.
  • the target feature fusion model may be selected by searching from multiple feature fusion models constructed in advance in this application.
  • the embodiment of the present application first constructs multiple feature fusion models according to multiple preset fusion rules, and each constructed feature fusion model corresponds to a fusion rule.
  • One of the fusion rules is illustrated below. : Pass at least two high-resolution first feature maps through at least one first convolution kernel respectively to obtain at least two third feature maps, and then process the resolutions of the obtained at least two third feature maps , Obtain at least two fourth feature maps with the same resolution as the low-resolution first feature map, and finally, merge the at least two fourth feature maps with the first feature map with the first resolution in a preset combination , Get at least one second feature map.
  • the feature fusion model is constructed by the embodiment of this application through preset fusion rules (ie, preset rules), the fusion rules are different, and the constructed feature fusion models are also different. This description of the fusion rules facilitates the understanding of the fusion rules described in this application.
  • processing the resolutions of the obtained at least two third feature maps may specifically be: performing a down-sampling operation, a pooling operation, or a convolution operation on the obtained at least two third feature maps. Etc., it is not limited here.
  • a lane line detection network is selected from a plurality of sampled lane line detection networks according to the preset conditions as the final target lane line detection network
  • the method may specifically include, but is not limited to, the following method: selecting a lane line detection network performance that is optimal among the performance of each lane line detection network corresponding to all sampling points as the target lane line detection network.
  • the initial confidence level of the first grid can be obtained first, and then the initial confidence level of each first grid is adjusted according to a preset manner, so as to obtain the first confidence level of each first grid. A degree of confidence.
  • the first confidence of the first grid is obtained by adjusting the initial confidence of the first grid.
  • the purpose of adjustment is to make the prediction for the far end of the lane line more accurate during the prediction process.
  • a statistical model is first established based on the distribution of the raster anchor point positions at the far end of the real lane line in each image in the training set.
  • the statistical model has at least one parameter to be determined, and then based on each image in the training set.
  • the statistical results of the grid anchor point positions at the far end of the real lane line estimate the parameters of the statistical model to obtain the parameter estimates, and the lane line detection network is trained in the above process (such as training in the search process, In full training, etc.), the estimation of the parameter is adjusted to obtain the determined value of the final statistical model parameter, and then the anchor point position of the first grid is calculated through the statistical model that determines the determined value of the parameter to obtain the first grid
  • the weight value of the anchor point position of the grid, and finally, the initial confidence level and weight value of the first grid are calculated to obtain the first confidence level of the first grid, for example, the initial confidence level of each first grid and
  • the weight values of the anchor point positions of each first grid are multiplied to obtain the first confidence of each first grid.
  • a method for obtaining the first confidence degree from the initial confidence degree based on a statistical method is provided, which is operability.
  • the statistical model includes: Gaussian function.
  • the statistical model can be modeled as a Gaussian function.
  • the second aspect of the embodiments of the present application also provides a lane line detection framework, which may specifically include a neural network, a feature fusion model, a confidence adjustment model, a prediction head model, and a prediction lane line integration model.
  • the neural network is used to extract features of the input image to be detected after training;
  • the feature fusion model is used to perform feature fusion on the first feature maps output by different layers of the neural network after training to obtain the first feature map Two feature maps;
  • the confidence adjustment model is used to divide the second feature map into multiple grids, and obtain the first confidence level of the first grid in the multiple grids, and the first confidence level is used for Represents the first probability that the real lane line in the second feature map crosses the first grid and the first probability exceeds the first preset threshold;
  • the predictive head model is used to obtain the relationship between the first grid and the first grid after training.
  • the first predicted lane line corresponding to the grid; the predicted lane line integration model is used to divide the n first predicted lane lines into m groups when the first grid has n, m ⁇ n and n ⁇ 2; the The predicted lane line integration model is also used for when there are q first predicted lane lines in the first group in the m group, according to the q first predicted lane lines corresponding respectively to the q first confidence levels and the corresponding respectively.
  • the q first grids integrate the q first predicted lane lines to obtain a second predicted lane line, and the second predicted lane line is output as the detection result of a real lane line in the second feature map, q ⁇ 2.
  • the predicted lane line integration model is specifically used to: divide the area occupied by the q first predicted lane lines in the second feature map into multiple sub-areas, and each sub-area includes the The position of the center point of at least one first grid in the q first grids; when the first sub-areas of the plurality of sub-regions include the center point positions of at least two first grids, then according to the at least two A grid corresponds to a first confidence level, a first grid is selected from the at least two first grids as the second grid, and the first predicted lane line corresponding to the second grid is located in the first sub The part of the area is used as the first part of the second predicted lane line, or the part of the first predicted lane line corresponding to the at least two first grids in the first sub-area is weighted to obtain the calculation result As the first part of the second prediction lane line; when the second subregion of the plurality of subregions only includes the center point position of the first grid, the first prediction corresponding to the first grid
  • the predicted lane line integration model is specifically further used for: selecting the first confidence from the at least two first grids according to the first confidence levels respectively corresponding to the at least two first grids The first grid with the largest degree value is used as the second grid.
  • the predicted lane line integration model is also used for: when there is only one first predicted lane line in the first group in the m group, then the first predicted lane line in the first group It is output as a detection result of a real lane line in the second feature map.
  • the feature fusion model is specifically used to: construct a search space, which is composed of the codes of multiple neural networks and the codes of multiple feature fusion models, and the multiple feature fusion models are based on A variety of preset rules are constructed; sampling is performed in the search space to obtain a first sampling point, and the first sampling point is a first coding combination composed of the coding of the first neural network and the coding of the first feature fusion model; according to the The first code combination generates the first lane line detection network, and trains the first lane line detection network through the training set; performs performance evaluation according to the trained first lane line detection network to obtain the trained first lane line Detection network performance; when the sampling reaches the preset value, and the performance of the first lane line detection network after training meets the preset conditions, the first neural network in the first lane line detection network after training is performed on the image to be detected Feature extraction to obtain first feature maps output by different layers of the first neural network; the first feature maps output from different layers of the first neural network are used
  • the feature fusion model is also specifically used to: pass at least two high-resolution first feature maps through at least one first convolution kernel respectively to obtain at least two third feature maps;
  • the resolution of the at least two third feature maps is processed to obtain at least two fourth feature maps with the same resolution as the low-resolution first feature map; the at least two fourth feature maps are compared with the first feature map.
  • the first feature maps of the resolution are merged in a preset combination manner to obtain at least one second feature map.
  • the feature fusion model is specifically used to perform down-sampling operation, pooling operation or convolution operation on the at least two third feature maps.
  • the performance of the first lane line detection network after the training meets the preset condition includes: the performance of the first lane line detection network after the training is the best among the performance of each lane line detection network corresponding to all sampling points. excellent.
  • the confidence adjustment model is specifically used to: divide the second feature map into multiple grids, and obtain the initial confidence of the first grid in the multiple grids, and the initial confidence The confidence is used to indicate the initial probability that the real lane line in the second feature map crosses the first grid and the initial probability exceeds the initial preset threshold; the initial confidence is adjusted in a preset manner to obtain the first The first confidence level of the raster.
  • the confidence adjustment model is also specifically used to: establish a statistical model based on the distribution of the grid center point position at the far end of the true lane line in each image in the training set, and the statistical model has at least one to be determined According to the statistical results of the grid center point position at the far end of the real lane line in each image in the training set, estimate the parameter to obtain the estimate of the parameter; adjust the estimate of the parameter during the training process, Obtain the determined value of the parameter; calculate the center point position of the first grid through the statistical model that determines the determined value of the parameter to obtain the weight value of the center point position of the first grid; The initial confidence of the grid is calculated with the weight value to obtain the first confidence of the first grid.
  • the statistical model includes: Gaussian function.
  • the trained framework is deployed on the detection device.
  • the detection equipment includes: wheeled mobile equipment, for example, for example, wheeled construction equipment, autonomous driving vehicles, auxiliary driving vehicles, etc., as long as it is equipped with wheeled mobile equipment, it is called This is the wheeled mobile device for this application.
  • wheeled mobile device is an example of an autonomous driving vehicle.
  • the autonomous driving vehicle may be a car, a truck, a motorcycle, a bus, a boat, an airplane, a helicopter, Lawn mowers, recreational vehicles, playground vehicles, construction equipment, trams, golf carts, trains, and trolleys, etc., are not particularly limited in the embodiments of the present application.
  • a third aspect of the embodiments of the present application provides a detection device, which has the function of implementing the foregoing first aspect or any one of the possible implementation methods of the first aspect.
  • This function can be realized by hardware, or by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above-mentioned functions.
  • the fourth aspect of the embodiments of the present application provides a detection device, which may include a memory, a processor, and a bus system.
  • the memory is used to store a program
  • the processor is used to call the program stored in the memory to execute the first aspect of the embodiment of the present application. Or any one of the possible implementation methods of the first aspect.
  • the fifth aspect of the present application provides a computer-readable storage medium that stores instructions in the computer-readable storage medium.
  • the computer can execute the first aspect or any one of the first aspects. Way of realization.
  • the sixth aspect of the embodiments of the present application provides a computer program or computer program product.
  • the computer program or computer program product runs on a computer, the computer executes the first aspect or any one of the possible implementation manners of the first aspect. method.
  • Figure 1 is a schematic diagram of the comparison of the two methods of feature fusion
  • Fig. 2 is a schematic diagram of two images of a driving road taken by a camera mounted on the own vehicle;
  • Fig. 3 is a schematic diagram of the visualization effect of the lane line predicted by the lane line detection algorithm
  • Figure 4 is a schematic diagram of different long tail situations in lane line detection
  • Figure 5 is a schematic diagram of a typical lane line detection architecture
  • Figure 6 is a schematic diagram of the working principle of the prediction head model
  • Figure 7 is a schematic diagram of the predicted lane line obtained through Line-NMS processing
  • FIG. 8 is a schematic diagram of a lane line detection framework after training constructed by an embodiment of the application.
  • FIG. 9 is a schematic diagram of an image processing system architecture provided by an embodiment of the application.
  • FIG. 10 is a schematic diagram of the deployment position of the lane line detection framework after training provided on an autonomous vehicle according to an embodiment of the application;
  • FIG. 11 is a schematic flowchart of a method for detecting lane lines according to an embodiment of the application.
  • FIG. 12 is a schematic diagram of a preset fusion rule provided by an embodiment of the application.
  • FIG. 13 is a schematic diagram of coding rules provided by an embodiment of this application.
  • FIG. 14 is a schematic diagram of a target lane line detection network obtained through search according to an embodiment of the application.
  • FIG. 15 is a schematic diagram of dividing the basic neural network into 5 stages according to an embodiment of the application.
  • FIG. 16 is a schematic diagram of full training provided by an embodiment of the application.
  • FIG. 17 is a schematic diagram of dividing an image to be detected into multiple grids and determining the first grid from the multiple grids according to an embodiment of the application;
  • FIG. 18 is a schematic diagram of the first confidence degree obtained by calculating the initial confidence degree according to an embodiment of the application.
  • FIG. 19 is a schematic diagram of obtaining a first confidence degree from an initial confidence degree based on a statistical method provided by an embodiment of the application;
  • 20 is a schematic diagram of a frequency heat map and a Gaussian function of the grid center point at the far end of each lane line provided by an embodiment of the application;
  • FIG. 21 is a schematic diagram of the first predicted lane line having different levels of features obtained through the trained prediction head model provided by an embodiment of the application;
  • FIG. 22 is a flowchart of obtaining a second predicted lane line provided by an embodiment of the application.
  • FIG. 23 is a schematic diagram of a method for obtaining a second predicted lane line based on integration of multiple first predicted lane lines in the same group according to an embodiment of the application;
  • 24 is another schematic diagram of a method for obtaining a second predicted lane line based on integration of multiple first predicted lane lines in the same group according to an embodiment of the application;
  • FIG. 25 is a schematic diagram of an application scenario provided by an embodiment of the application.
  • FIG. 26 is a schematic diagram of a detection device provided by an embodiment of the application.
  • FIG. 27 is another schematic diagram of a detection device provided by an embodiment of the application.
  • FIG. 28 is a schematic diagram of a structure of a chip provided by an embodiment of the application.
  • the embodiment of the application provides a method for detecting lane lines and related equipment, which are used to perform feature fusion on the first feature maps output by different layers of the neural network after training, and the second feature map obtained is based on the second feature
  • the multiple lane line prediction results obtained from the map are integrated to improve the accuracy of lane line detection.
  • the embodiments of the present application involve a lot of related knowledge about lane line detection and neural networks.
  • a neural network can be composed of neural units. Specifically, it can be understood as a neural network with an input layer, a hidden layer, and an output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the number of layers in the middle is all It is the hidden layer. Among them, a neural network with many hidden layers is called a deep neural network (DNN).
  • the work of each layer in the neural network can be expressed in mathematical expressions To describe, from the physical level, the work of each layer in the neural network can be understood as through five operations on the input space (the set of input vectors) to complete the transformation from the input space to the output space (that is, the row space of the matrix to the column Space), these five operations include: 1. Dimension Up/Down; 2.
  • the purpose of training the neural network is to finally get the weight matrix of all layers of the trained neural network. Therefore, the training process of the neural network is essentially the way of learning to control the space transformation, and more specifically, the learning of the weight matrix.
  • the training process of the neural network is essentially the way of learning to control the space transformation, and more specifically, the learning of the weight matrix.
  • To update the weight matrix of each layer of neural network (of course, there is usually an initialization process before the first update, that is, to pre-configure parameters for each layer in the neural network), for example, if the predicted value of the network is high, then Adjust the weight matrix to make its prediction lower, and keep adjusting until the neural network can predict the really desired target value.
  • the loss function (loss function) or objective function (objective function)
  • object function objective function
  • Feature fusion generally has two methods: concat and add.
  • concat is a series of feature fusion method, that is, directly connect two features, if the dimensions of the two input features x and y are p and q, the dimension of the output feature z is p+q; add is a kind
  • the parallel fusion strategy is to combine two feature vectors to obtain a new feature z with the number of channels unchanged for the input features x and y. The details are shown in Figure 1.
  • add means that the amount of information under the characteristics of the description image has increased, but the dimension of the description image itself has not increased, but the amount of information under each dimension is increasing; and concat is the number of channels The merging of, that is to say, the features describing the image itself have increased, but the information under each feature has not increased.
  • wheeled mobile robot It is a comprehensive system integrating environment perception, dynamic decision-making and planning, behavior control and execution, etc. It can also be called a wheeled mobile robot or a wheeled intelligent body.
  • wheeled construction equipment self-driving vehicles, Driving assisted vehicles, etc., as long as they are equipped with wheeled movable equipment are referred to as wheeled mobile equipment as described in the present application.
  • wheeled mobile device is an example of an autonomous driving vehicle.
  • the autonomous driving vehicle may be a car, a truck, a motorcycle, a bus, a boat, an airplane, a helicopter, Lawn mowers, recreational vehicles, playground vehicles, construction equipment, trams, golf carts, trains, and trolleys, etc., are not particularly limited in the embodiments of the present application.
  • perception can have a variety of modules, such as laser perception module, visual perception module, millimeter wave perception module, etc., and the visual perception module is one of the key modules , Is widely used in advanced driver assistance systems (ADAS) and autonomous driving systems (ADS), which can provide wheeled mobile devices (such as autonomous vehicles) with the system Provides relevant information about the surrounding environment (for example, lane lines on the road). The relevant information can also be referred to as perception information, which provides a solid basis for reasonable decision-making in regulation and control.
  • ADAS advanced driver assistance systems
  • ADS autonomous driving systems
  • the perceptual information used is related videos (images to be extracted later) or images taken by a visual perception module (for example, a camera installed on a wheeled mobile device) on the road surface of the vehicle, as shown in Figure 2. Shown are two images (image 1 and image 2) of the road surface captured by the camera mounted on the own vehicle. The obtained relevant image is used as the image to be detected, which is used to detect the lane line of the driving road.
  • Figure 3 uses the lane line detection algorithm to detect the image to be detected in real time input by the visual perception module to predict the lane line
  • the location and its visualization results are shown in Figure 3.
  • the four acquired images to be detected are detected and output respectively 4, 2, 5, and 3 predicted lane lines.
  • Target detection also called target extraction, is a kind of image segmentation based on target geometric and statistical characteristics. It focuses on a specific object in the image and requires the category information and location information of this specific object to be obtained at the same time. Its accuracy and real-time sexuality is an important capability of the entire system. Target detection is applied in many scenarios, such as driverless security systems, intelligent transportation systems, and intelligent monitoring systems.
  • Lane line detection can be regarded as a special kind of target detection. Since general target detection and lane line detection have large differences in target features, target objects in general target detection (such as cats, dogs, human faces, etc.) The area occupied by has a certain aspect ratio, which is easy to be recognized, while the lane line in the lane line detection is slender, with a long span and narrow width, and it is difficult to accumulate enough robust features to describe The entire lane line is not easy to be recognized, which places harsh requirements on the neural network, so lane line detection is more challenging than general target detection. In addition, driving scenarios are usually quite complex.
  • lane line detection is very important for the accurate positioning of autonomous vehicles and the impact on driving decisions.
  • the lane line can effectively guide the intelligent driving vehicle to drive in the restricted road area.
  • Real-time detection of the road lane line is an important link in the ADAS or ADS of the intelligent driving vehicle. It is helpful to assist in path planning, carry out road deviation warning and other functions, and can provide a reference for precise navigation.
  • the existing lane detection methods are roughly divided into the following two categories:
  • Lane line detection based on manual features.
  • Lane line detection methods based on manual features rely on handmade features (including color, edge information, etc.). These features combine Hough transform and Kalman filtering to detect lane lines. In most cases It has robust detection performance. However, because it cannot adapt well to the complex and changeable lane lines in real life, this manual feature-based lane line detection method cannot effectively provide reliable lane line detection for the downstream modules of autonomous driving (such as regulatory systems). As a result, it has been rarely mentioned in autonomous driving research.
  • Lane line detection based on convolution features b.
  • a typical lane line detection architecture is shown in Figure 5.
  • the main process of the lane line detection architecture is: First, Input the image to be detected into a neural network (for example, convolutional neural network) for feature extraction to obtain the feature map of the last layer of the neural network, and decode the extracted feature map through the predictive head model to generate dense lines Clusters (that is, multiple predicted lane lines), and finally, by performing line non-maximum suppression (Line-NMS) processing on the line clusters, the final prediction result of the lane line in the image to be detected is output.
  • a neural network for example, convolutional neural network
  • Line-NMS line non-maximum suppression
  • the above prediction head model works as shown in Figure 6.
  • the feature map output by the neural network is divided into multiple grids of size w ⁇ h, one of which can be denoted as ⁇ G ij ⁇ , the center point of each grid G ij on the feature map is called the anchor point. If the center point of G ij is close to a certain real lane line on the ground, the G ij is responsible for detecting the lane line.
  • the grid G ij S ij reflects the degree of confidence is the probability by which a lane line of the grid G ij, the prediction
  • the grid G ij S ij reflects the degree of confidence is the probability by which a lane line of the grid G ij, the prediction
  • the probability of a lane line passing through the grid G ij is the lane line on the grid.
  • the prediction head model will predict a set of offsets ⁇ x ijz and an end position (that is, the position where the far end of the lane line disappears on the feature map), where ⁇ x ijz is the real lane line on the ground and the passing anchor.
  • the horizontal distance between the vertical lines of the points, as shown in FIG. 6, is a set of offsets ⁇ x ijz as indicated by four double-headed arrows, where the black dot position is the anchor point.
  • each lane line in the image to be detected can be recovered from the ⁇ x ijz and end position predicted by the model.
  • a real lane line on the image to be detected often causes multiple grid responses. Therefore, in the prediction stage, the prediction result corresponding to a real lane line will have many predicted lane lines, which need to be processed with the help of Line-NMS. So as to generate the final prediction result.
  • the process of processing through Line-NMS is specifically: after obtaining the line cluster (ie multiple predicted lane lines) output by the prediction head model, sort the predicted lane lines according to the confidence value, and then take the maximum confidence value
  • the predicted lane line of is the baseline and the distance between other predicted lane lines and the baseline is less than the threshold value.
  • the lane lines are divided into one group, and the line clusters are divided into several groups in a similar manner, and the line clusters in each group are respectively taken
  • the baseline is the final output of the detection result of a real lane line in this group.
  • the predicted lane line output after the Line-NMS processing is shown in the right part of Figure 7, and finally 3 predicted lane lines are output.
  • the left part of Figure 7 is based on Confidence value is an indication of grouping all predicted lane lines in the line cluster (divided into 3 groups).
  • the above-mentioned method of lane line detection based on convolution features is relatively primitive.
  • the entire algorithm design has a premise that it is assumed that each grid can accurately predict an entire lane line passing through this grid, but the actual lane line spans Long, the part of the lane line far from the center of the grid is difficult to effectively predict; and the use of a straight-tube network backbone cannot effectively expand the network's receptive field, which limits the network's expressive ability.
  • an embodiment of the present application provides a lane line detection method, which is used to perform feature fusion on the first feature maps output by different layers of the trained neural network, and the obtained first feature maps are obtained. Two feature maps, and multiple lane line prediction results obtained based on the second feature map are integrated, thereby improving the accuracy of lane line detection.
  • the framework for realizing the lane line detection function of the embodiment of the present application is introduced.
  • the framework can be called a lane line detection framework (also called a lane line detection algorithm).
  • the lane line detection framework constructed by this application achieves the detection
  • Figure 8 is a trained lane line detection framework constructed by this application.
  • the lane line detection framework consists of the following parts: neural network 101, feature fusion Model 102, prediction head model 103, prediction lane line integration module 104, wherein the neural network 101 is used to extract features from the input image to be detected, thereby obtaining multiple feature maps of different layers of the neural network 101, and different feature maps are extracted to The features of the image to be detected are also different.
  • Some feature maps have high-level receptive fields, and some feature maps have low-level receptive fields.
  • the neural network 101 can set the search parameters to determine from the various nerves that match the search parameters.
  • the search space formed by the network is searched; the feature fusion model 102 is used to perform feature fusion on the feature maps of different layers extracted by the neural network 101 to obtain the fused feature maps.
  • the fused feature maps have low-level receptive fields.
  • the feature fusion model 102 is constructed according to a variety of different preset rules in this embodiment of the application to obtain multiple ones.
  • the neural network 101 can be searched in a separately set search space, or it can be searched in a search space formed with multiple feature fusion models.
  • the feature fusion model 102 can be searched in a separately set search space. Obtained, it can also be searched in a search space formed with multiple neural networks, and it is not limited here.
  • the prediction head model 103 is a bridge between the feature map and the real lane line. It is used to decode and generate the lane line corresponding to the original image according to the feature map with different levels after fusion. Specifically, it is used for the grid division of the fusion feature map.
  • the lane line prediction is performed in the grid, and each grid with a confidence level higher than the preset threshold can predict a lane line correspondingly; the predicted lane line integration module 104 groups all the predicted lane lines, in each group because Feeling the limitations of the field, each lane line has its own optimal prediction interval.
  • the prediction lane line integration module 104 selects the optimal prediction interval of each lane line to integrate the lane lines, and finally combines the prediction advantages of different lane lines. Form the final lane line.
  • the constructed lane line detection framework may also have a confidence adjustment model 105.
  • the confidence adjustment model 105 is used to adjust the initial confidence of the grid.
  • the lane line detection framework described in Figure 8 above is a lane line detection framework that has been trained. After the lane line detection framework is constructed, the entire lane line detection framework needs to be trained first, and the trained lane The line detection framework can be applied to specific lane line detection tasks. Specifically, the entire training and detection process of the lane line detection framework can be introduced based on the image processing system 200 shown in FIG. 9.
  • the image processing system 200 includes a detection device 210, a training device 220, a database 230, The client device 240, the data storage system 250, and the data collection device 260.
  • the detection device 210 includes a calculation module 211, which is essentially the lane line detection framework 201 after training provided by the embodiment of the application.
  • the data acquisition device 260 (for example, a camera mounted on a wheeled mobile device) is used to acquire the open source large-scale data set (ie training set) required by the user, and store the data set in the database 230, and the training device 220 Based on the data set maintained in the database 230, each model in the lane line detection framework 201 is trained (for example, the neural network 101, the feature fusion model 102, etc. shown in FIG. 8 are trained).
  • the detection device 210 can call data, codes, etc. in the data storage system 250, and can also store data, instructions, etc. in the data storage system 250.
  • the data storage system 250 may be placed in the detection device 210, or the data storage system 250 may be an external memory relative to the detection device 210.
  • the lane line detection framework 201 trained by the training device 220 can be applied to different systems or devices (ie, the detection device 210).
  • the detection device 210 can be various wheeled mobile devices (wheeled construction equipment, autonomous vehicles, Auxiliary driving vehicles, etc.), self-driving vehicles can also be cars, trucks, motorcycles, buses, boats, airplanes, helicopters, lawn mowers, recreational vehicles, playground vehicles, construction equipment, trams, golf carts, Trains, and trolleys, etc.
  • the detection device 210 is configured with an I/O interface 212 to perform data interaction with external devices, and the "user" can input data to the I/O interface 212 through the client device 240.
  • the client device 240 may be a camera device mounted on an autonomous driving vehicle.
  • the image taken by the camera device is input as input data to the calculation module 211 of the detection device 210, and the lane line detection frame 201 in the calculation module 211 compares the input After the image is detected, the detection result is obtained, and then the detection result is output to the camera device or directly displayed on the display interface (if any) of the detection device 210; in addition, in some embodiments of the present application, the client device 240 may also Integrate in the detection device 210, for example, when the detection device 210 is an autonomous driving vehicle, it can directly capture images through the camera of the autonomous vehicle or receive images sent by other devices (such as mobile phones), and then the autonomous driving The calculation module 211 in the vehicle detects the image to obtain a detection result, and directly presents the detection result on the display interface of the mobile phone.
  • the product form of the detection device 210 and the client device 240 is not limited here.
  • FIG. 9 is only a schematic diagram of an image processing system architecture provided by an embodiment of the present application, and the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation.
  • the data storage system 250 is an external memory relative to the detection device 210. In other cases, the data storage system 250 can also be placed in the detection device 210; in FIG. 9, the client device 240 is an external device relative to the detection device 210. In other cases Next, the client device 240 may also be integrated in the detection device 210.
  • the lane line detection frame 201 may also be split into multiple sub-modules/sub-units to jointly implement the solutions provided in the embodiments of the present application, and the details are not limited here.
  • the training of the lane line detection framework 201 described in the above embodiment can be implemented on the cloud side.
  • it can be implemented by a training device 220 on the cloud side (the training device 220 can be set on one or more servers). Or on a virtual machine)
  • Obtain the training set and train the lane line detection framework according to multiple sets of training data in the training set to obtain the trained lane line detection framework 201.
  • the trained lane line detection framework 201 is sent again Apply the detection device 210, for example, send it to the relevant system of the autonomous vehicle (such as ADAS, ADS, etc.) for lane line detection.
  • the relevant system of the autonomous vehicle such as ADAS, ADS, etc.
  • the training device 220 The lane line detection frame 201 performs overall training, and the trained lane line detection frame 201 is then sent to the detection device 210 for use; the training of the lane line detection frame 201 described in the foregoing embodiment can also be implemented on the terminal side, that is, training
  • the device 220 may be located on the terminal side.
  • a wheeled mobile device such as an autonomous driving vehicle, an assisted driving vehicle, etc.
  • the trained lane line detection frame 201 is obtained, and the trained lane line detection frame 201 can be used directly in the terminal device, or sent by the terminal device to other devices for use.
  • the embodiment of the present application does not limit on which device (cloud side or terminal side) the lane line detection framework 201 is trained or applied.
  • the trained lane line detection framework (or the detection device 210 described in FIG. 9) described in FIG. 8 and FIG. 9 constructed by the present application can be deployed in various wheeled mobile devices for monitoring
  • the relevant perceptual information (such as video or image) on the road surface captured by the camera on the wheeled mobile device is processed, and the predicted lane line is output.
  • the predicted lane line is input to the downstream module of the wheeled mobile device for further processing.
  • Figure 10 shows the top-down layering.
  • Type system architecture there can be defined interfaces between systems to transmit data between systems to ensure the real-time and integrity of the data. The following is a brief introduction to each system:
  • Environmental perception is the most basic part of autonomous vehicles. Whether it is making driving behavior decisions or global path planning, it needs to be built on the basis of environmental perception. Based on the real-time perception of the road traffic environment, corresponding judgments, Decision-making and planning to enable intelligent driving of vehicles.
  • the environmental perception system mainly uses various sensors to obtain relevant environmental information, thereby completing the construction of the environmental model and the knowledge expression of the traffic scene.
  • the sensors used include one or more cameras, single-line radar (SICK), and four-line radar. (IBEO), three-dimensional lidar (HDL-64E), etc., where the video or image collected by the camera is mainly responsible for traffic light detection, lane line detection, road sign detection, vehicle recognition, etc.; other lidar sensors are mainly responsible for moving ⁇ Static detection, recognition and tracking of key obstacles, as well as detection and extraction of non-key obstacles such as road boundaries, shrub belts, and surrounding buildings.
  • the road-related perception data captured by the camera is used, that is, the image to be detected on the road surface.
  • the image to be detected may be an image directly shot by the camera or extracted from a video shot by the camera).
  • the image to be detected is not limited here.
  • the result (that is, the predicted lane line), the prediction result can be directly used as the downstream module of the autonomous vehicle (such as the autonomous decision-making system) for further processing, or it can be input to other models in the environment perception system (such as the map construction model, Figure 10 Not shown), used to construct an occupancy grid map (OGM) that can express the road environment together with other perception information, and finally send the OGM to the autonomous decision-making system for further decision-making and planning.
  • OGM occupancy grid map
  • Autonomous decision-making system is a key component of autonomous vehicles.
  • the system is mainly divided into two core subsystems: behavior decision-making and motion planning.
  • the behavior decision-making subsystem is mainly to obtain the global optimal driving route by running the global planning layer.
  • the motion planning subsystem plans a feasible driving trajectory based on safety, stability and other indicators based on the received driving behavior instructions and the current local environment perception information, and sends it to the control system.
  • control system is divided into two parts: the control subsystem and the execution subsystem.
  • the control subsystem is used to convert the feasible driving trajectory generated by the autonomous decision-making system into the specific execution instructions of each execution module and pass it to the execution.
  • Subsystem; the execution subsystem receives the execution instructions from the control subsystem and sends them to each control object to perform reasonable control of the vehicle's steering, braking, throttle, gear position, etc., so that the vehicle can drive automatically to complete the corresponding Driving operation.
  • the accuracy of the driving operation of an autonomous vehicle mainly depends on whether the specific execution instructions generated by the control system for each execution module are accurate, and the accuracy depends on autonomous decision-making. Systems and autonomous decision-making systems are faced with uncertainty.
  • the uncertainty factors mainly include the following aspects: 1) The characteristics of each sensor in the environmental perception system and the uncertainty caused by calibration errors, the perception mechanism and perception of different sensors The range and the corresponding error mode are different, and the calibration error caused by the installation of the self-driving vehicle will eventually be reflected in the uncertainty of the perception information; 2) the uncertainty caused by the data processing delay of the environmental perception system This is because the road environment is complex and the amount of data information is huge, which makes the data processing of the environment perception system also require a large amount of calculation, and the entire environment is constantly changing, which will inevitably lead to the delay of data and information, thereby affecting autonomous decision-making The correct judgment of the system; 3) The different processing methods of perception information will also bring uncertainty.
  • FIG. 10 the overall architecture of the autonomous vehicle shown in FIG. 10 is only for illustration. In actual applications, it may contain more or fewer systems/subsystems or modules, and each system/subsystem or module It may include multiple components, which are not specifically limited here.
  • the lane line detection method provided by the embodiment of the application can be performed based on the trained lane line detection framework shown in Figures 8 and 9 above, and the lane line detection method provided by the embodiment of the application can be applied to each
  • a wheeled mobile device for intelligent driving such as unmanned driving, assisted driving, etc.
  • motion planning such as speed planning, driving behavior decision-making, global path planning, etc.
  • Figure 11 is for A schematic flow chart of the lane line detection method provided in the application embodiment specifically includes:
  • the detection device deployed with the aforementioned lane line detection framework first performs feature fusion on the first feature maps output by different layers of the trained neural network to obtain one or more second feature maps.
  • the feature map is the fused feature map. This is because the performance of feature maps extracted from different layers of the neural network is different.
  • the low-level feature maps have higher resolution and contain more position and detailed information, but because there are fewer convolutions, they have lower semantics and more noise; high-level features Feature maps have stronger semantic information, but have low resolution and poor perception of details. Therefore, feature fusion is performed on the feature maps extracted from different layers of the neural network, and the second feature map obtained has multi-level features.
  • this step 1101 can be calculated based on the trained feature fusion model 102 in the embodiment corresponding to FIG. 8. Specifically, based on the embodiment corresponding to FIG. 8, the image to be detected is first extracted to multiple feature maps (ie, the first feature map) of different layers through the trained neural network 101, and then features are performed through the trained feature fusion model 102 Fusion, output one or more fused feature maps (that is, the second feature map).
  • feature maps ie, the first feature map
  • the trained feature fusion model 102 Fusion output one or more fused feature maps (that is, the second feature map).
  • the feature fusion model 102 may be selected by searching from multiple feature fusion models constructed in advance in the present application. Specifically, the embodiment of the present application first constructs multiple feature fusion models according to multiple preset fusion rules, and each constructed feature fusion model corresponds to a fusion rule.
  • One of the fusion rules is illustrated below. , Assuming that each layer of the neural network is divided into x stages (for example, x is 4), each stage includes one or more layers of the neural network, and the feature map extracted from the last layer of each stage is used as the first feature map , Because different stages have different resolutions, different channel numbers, and different receptive fields.
  • G 1,...t represents the first feature map output at different stages of the neural network, and the spatial resolution from the feature map F 1 to the feature map F t is gradually reduced by a certain multiple (for example, a multiple of 2)
  • the feature fusion model to be constructed is set in advance to include y fusion layers ⁇ O i ⁇ , where the value of y is a hyperparameter.
  • the fusion layer refers to a fusion operation that performs feature fusion on multiple first feature maps. For each a fusion layer, and F 1, ... z a (e.g., 2) a first high-resolution features in FIG pick t as input, and then a resolution selected from F 1, ... t in The lower first feature map is used as a comparison.
  • the z first feature maps first pass at least one convolution kernel operation to output z third feature maps, and then the resolution of these z third feature maps is processed (for example, down-sampling operation, pooling operation, convolution operation, etc., which are not limited here), obtain z fourth feature maps with the same resolution as the selected first feature map with lower resolution, and then add The z fourth feature maps are merged again according to the preset combination manner to obtain at least one second feature map.
  • Different fusion rules will generate a code corresponding to the pre-set coding method. Through this code, the detection device can identify which fusion rule is, and then know which feature fusion model is.
  • Figure 12 For ease of understanding, the following takes Figure 12 as an example to illustrate one of the fusion rules and the corresponding coding:
  • the first feature maps feature maps can also be called feature layers
  • the feature map "0" and the feature map "1” are two feature maps with higher resolution
  • the feature map "2" is the lower resolution A feature map.
  • the above-mentioned fusion rule can be coded as "012-022" according to the preset coding rule. The meaning of each number is shown in Figure 13. According to the coding, the detection equipment The corresponding fusion rule can be identified.
  • the neural network and the feature fusion model are determined after training, the network structure of the neural network can have multiple choices.
  • feature fusion models built by the embodiment of this application. It is very important to select a neural network (which can be called a target neural network) and a feature fusion model (which can be called a target feature fusion model) from a variety of neural networks and multiple feature fusion models. In some embodiments, it can be obtained through, but not limited to, the following methods:
  • lane line detection is a special type of target detection
  • many existing neural networks for target detection such as residual networks (RestNet, ResNeXt, etc.), fast region-generating convolutional neural networks (faster region- Convolutional neural network, Faster R-CNN), single-shot multibox defender (SSD), etc.
  • RestNet residual networks
  • ResNeXt ResNeXt
  • fast region-generating convolutional neural networks faster region- Convolutional neural network
  • SSD single-shot multibox defender
  • this application analyzes, experiments, and evaluates many lane line detection models, and determines a basic model structure suitable for lane line detection tasks as the combined architecture of "neural network + feature fusion model", which can be called
  • the lane line detection network then encodes the lane line detection network composed of "neural network + feature fusion model", and all the corresponding codes of the lane line detection network composed of "neural network + feature fusion model” form the search space.
  • search rules for example, random sampling
  • the search space of "neural network + feature fusion model” is first constructed, that is, the code combination corresponding to each pair of “neural network + feature fusion model”.
  • the process of each search includes sampling from the search space, and after sampling A string representing "neural network + feature fusion model” (ie, a code combination) can be obtained, and a specific instance of "neural network + feature fusion model” is generated based on this string, that is, the specific structure of a lane line detection network; After that, this instance is trained through the training data in the training set to obtain the trained lane line detection network, and after the training is completed, the performance evaluation of the trained lane line detection network is performed to obtain the network performance.
  • a string representing "neural network + feature fusion model” ie, a code combination
  • this instance is trained through the training data in the training set to obtain the trained lane line detection network, and after the training is completed, the performance evaluation of the trained lane line detection network is performed to obtain the network performance.
  • each lane line detection network Perform the above operations on each sampling point to obtain the performance of the lane line detection network corresponding to each sampling point.
  • the sampling reaches the preset number of times, such as 400 times, there are 400 corresponding network structures of the lane line detection network obtained.
  • the performance of these 400 lane line detection networks has also been obtained through performance evaluation.
  • a lane line detection network is selected from the 400 lane line detection networks according to the preset conditions. The network serves as the final target lane line detection network.
  • the target lane line detection network includes the corresponding target neural network (e.g., the above-mentioned trained neural network 101) and the target feature fusion model (e.g., the above-mentioned trained feature fusion model) 102).
  • the target neural network in the target lane line detection network can be used to perform feature extraction on the image to be detected, and the first feature map output by the different layers of the target neural network can be obtained, and then the first feature map output by the different layers can be used as the target The input of the feature fusion model, and the output of the second feature map by the target feature fusion model.
  • a lane line detection network is selected from multiple sampled lane line detection networks according to the preset conditions set in advance in this application as the final
  • the methods of the target lane line detection network can specifically include but are not limited to the following methods: 1) Select a lane line detection network performance that is the best among the performance of each lane line detection network corresponding to all sampling points as the target lane line detection network; 2) choose a lane line detection network that has the largest computing power among all the lane line detection networks corresponding to all sampling points as the target lane line detection network.
  • FIG. 14 is one of the ways to obtain the target lane line detection network through search according to an embodiment of this application.
  • the search space of neural network + feature fusion model is the code combination corresponding to each pair of “neural network + feature fusion model”.
  • Each search process includes sampling from the search space. After sampling, a representative “neural network” can be obtained.
  • Network + feature fusion model" character string that is, a code combination, which is a combination of the first code of the neural network obtained by sampling and the second code of the feature fusion model obtained by sampling.
  • a coding combination composed of the first code of the neural network and the second code of the feature fusion model is generated, and then the code combination is decoded to generate a lane line detection network , And then use the training data in the training set to train the lane line detection network obtained by each sampling, and further input the output of the lane line detection network (that is, the resulting fused feature map) into the prediction shown in Figure 8.
  • the head model 102 and the predicted lane line integration module 104 are trained for the entire lane line detection framework, and finally the performance of the lane line detection network is evaluated based on the predicted lane line.
  • the specific evaluation methods can include but are not limited to the following methods: Each pair of code combinations generated by the search is input into the lane line detection framework shown in Figure 8, and the parameters of the prediction head model 103 and the parameters of the predicted lane line integration module 104 are input, and the lane line detection framework is initialized according to all the parameters, and then Input a total of M images in the test set in sequence to obtain the output of the lane line detection framework. Specifically, for each image in the test set, first obtain the fused feature map output by the feature fusion model 102 in the lane line detection network.
  • the integrated predicted lane line is output from the predicted lane line integration module 104, and then F1 is estimated according to the output predicted lane line and label, and the estimated value of F1 is output.
  • the estimated value of F1 is used as the The performance indicators of the lane line detection network, and finally adjust the parameters of the entire lane line detection framework (for example, the prediction head model 103, the prediction lane line integration module 104) according to the performance evaluation results of each lane line detection network, and the accumulation of enough
  • make a scatter plot composed of the delay represented by the sampling point and the estimated value of F1 take out the point above the Pareto (pareto) front, and then use this point as the basis for random inactivation , Random exchange and other operations (that is, the basic operation of genetic algorithm), as the new sampling point.
  • a preset value for example, 300
  • the best point of pareto is output as the final search result, that is, the final output lane line detection network.
  • the neural network is constructed by stacking ResNet/ResNeXt basic blocks (blocks), because ResNet/ResNeXt blocks have many (for example, 10 to 45 blocks). ).
  • the ResNet/ResNeXt block can be divided into multiple stages according to needs. The last block of different stages outputs a first feature map.
  • Figure 15 shows that all blocks are divided into 5 stages, namely stages 1 to In stage 5, the basic channels are 48, 64, 80, 96, 128, and each stage can choose to include a different number of blocks.
  • the number of blocks in the neural network is different, and the way to divide the stages is different, then the construction is obtained
  • the position of increasing the number of channels is obtained by searching.
  • the number of blocks included in each neural network, the number of divided stages, the number of blocks specifically included in each stage, the position of the number of channels, etc. can be obtained through advance
  • the set rules are coded (that is, the first code shown in FIG. 14), and each first code corresponds to one neural network described above.
  • the full amount of training data in the training set can be used to evaluate the performance of the lane line detection network obtained by the search. to evaluate.
  • the total search space composed of the neural network is about 5 ⁇ 10 12 cases, after each sampling, it is necessary to assign values (ie training) to the parameters in each lane line detection network initialized, due to the fully trained lane line
  • the detection network can represent the accuracy of the searched neural network (the accuracy drift of the model without fully training is too severe), and the complete training on the full training set is too time-consuming. Therefore, in some embodiments of this application, Sampling is performed on the training set, and then full training is performed on representative samples.
  • the training set obtained by graph sampling may be 100,000 sheets, and the test set may be 30,000 sheets.
  • the main purpose is to train a lane line detection framework that can be deployed on the device. .
  • the embodiment of the present application uses the full training set for training and performs performance evaluation.
  • the specific method can be shown in Figure 16, decode the code combination of the searched target "neural network + feature fusion model" to obtain the target lane line detection network, and train the target lane line detection network.
  • After the training is completed Perform performance evaluation, and adjust the collective parameters of the entire lane line detection framework of the vehicle according to the performance evaluation results. If the final performance evaluation result meets the performance requirements, output the corresponding lane line detection framework file for deployment.
  • the specific training process is similar to the training process in the search process, and will not be repeated here.
  • the target "neural network + feature fusion model" obtained in the search process is to ensure that the search time can be tolerated, so the strategy of sampling the training set is used, and the search and evaluation are used A subset of the training set.
  • the full training process is to train the searched target "neural network + feature fusion model” using the full training set (for example, 1 million images).
  • the parameters of the lane line detection framework are fine-tuned according to the evaluation results. Its main purpose is to train a lane line detection framework that can be deployed on the device.
  • the first search space composed of a variety of existing neural networks and the second search space composed of multiple feature fusion models constructed by this application are respectively searched, trained separately, and evaluated separately.
  • the search for neural networks in the first search space and the training and performance evaluation process of each neural network searched out, and the search for feature fusion models in the second search space and the training of each searched feature fusion model The performance evaluation process is similar to the above method (1) searching for the combined architecture of "neural network + feature fusion model" in the jointly combined search space, which will not be repeated here.
  • method (2) is to separate the two modules for searching, training, and performance evaluation, and then combine the trained target neural network and the trained target feature fusion model into the required The target lane line detection network.
  • At least one second feature map can be obtained. For each obtained second feature map, it can be divided into multiple grids. If the multiple grids have a confidence value greater than the preset A grid with a threshold (for example, a preset threshold of 0.6) is set, and the confidence level greater than the preset threshold may be referred to as the first confidence level, and the grid corresponding to the first confidence level may be referred to as the first grid.
  • a threshold for example, a preset threshold of 0.6
  • the purpose of selecting the grid corresponding to the first confidence level from the multiple grids is to remove those redundant grids that do not contribute to the prediction of the lane line, so as to reduce the amount of calculation.
  • the second feature map output by the lane line detection network is divided into multiple grids of size w ⁇ h, one of which can be Denoted as ⁇ G ij ⁇ , the center point of each grid G ij on the second feature map is called the anchor point. If the center point of G ij is close to a certain real lane line on the ground, the G ij is responsible for detecting the lane String.
  • the grid G ij S ij reflects the degree of confidence is the probability by which a lane line of the grid G ij, the training phase Since each of the training images on the lane line transactions are to be annotated, the probability of a lane line by a grid G ij only two cases: through, the grid G ij is the confidence value S ij is 1; It does not pass, then the grid G ij S ij is the confidence value of 0. In the detection stage, whether there is a lane line on the extracted second feature map and the presence of a lane line at a specific location is in the form of probability.
  • the probability of a lane line passing through the grid G ij is that the lane line is on the grid.
  • the first grid can be obtained first.
  • the initial confidence level of each grid is adjusted according to the preset method to obtain the first confidence level of each first grid.
  • the purpose of adjustment is to make the prediction
  • the prediction of the far end of the lane line can be more accurate. Specifically, including but not limited to the following methods:
  • the first confidence is obtained from the initial confidence based on the statistical method.
  • the statistical model has at least one parameter to be determined.
  • the statistical results of the location of the grid anchor points estimate the parameters of the statistical model to obtain the estimation of the parameters, and the parameters are used in the above-mentioned training process of the lane line detection network (eg, training in the search process, full training, etc.) Adjust the estimation of the parameters to obtain the determined values of the final statistical model parameters, and then calculate the anchor point positions of the first grid through the statistical model with the determined values of the parameters to obtain the weight value of the anchor point positions of the first grid
  • the initial confidence and weight value of the first grid are calculated to obtain the first confidence of the first grid.
  • the initial confidence of each first grid is compared with the The weight values of the anchor point positions of the grid are multiplied to obtain the first confidence of each first grid.
  • ⁇ 1 to ⁇ 4 are the 4 parameters of the statistical model, and X is the coordinate of the grid anchor point where the far end of the lane line is located.
  • the statistical model is determined to be the Gaussian function
  • statistics are performed according to the coordinates of the raster anchor point position at the far end of each lane line on the training set, and the parameter values of the above-mentioned Gaussian function are estimated as the confidence in the lane line detection framework in Figure 8.
  • the initialization parameters of the model 105 are adjusted to the degree.
  • the parameters of the statistical model included in the confidence adjustment model 105 need to be adjusted at the same time, and the parameters are further adjusted according to the adjusted parameters.
  • Value formula (1) Calculate the weight value of the anchor point position of each first grid.
  • X represents the value of each first grid.
  • the first confidence is obtained from the initial confidence based on the learning method.
  • the method (1) is based on the statistical results of the far-end distribution of the lane lines on a large number of training sets.
  • the first grid can be obtained from the initial confidence of the first grid by learning.
  • the first confidence of a grid specifically, a neural network can be constructed, such as convolutional neural networks (convolutional neural networks, CNN), and the initial confidence of the first grid in each training image in the training set is taken as the The input of the CNN, the output of the first confidence of the first grid, by constructing an appropriate loss function, then the trained CNN can output the first confidence of the first grid that meets the requirements of this application.
  • CNN convolutional neural networks
  • the trained prediction head model (for example, the prediction head model 103 in the lane line detection framework in Fig. 8) can be further used to obtain the corresponding data for each first grid.
  • a predicted lane line (that is, the first predicted lane line).
  • the first predicted lane line obtained from the first grid is obtained through the prediction head model, and the specific working principle of the prediction head model is as described in the embodiment corresponding to FIG. 6 above. Do not repeat it.
  • the second feature map has multi-level features, and the low-level features in the second feature map are distinguished
  • the rate is higher, containing more location and detail information, but due to less convolution, it has lower semantics and more noise; the high-level features in the second feature map have stronger semantic information, but the resolution Low, poor perception of details. Therefore, as shown in Figure 21, the first predicted lane line obtained by the trained prediction head model also has different levels of characteristics. Among them, the low-level receptive field is conducive to the accurate regression of the lane line, and the high-level feeling Wild, it is helpful to judge whether the lane line exists.
  • n first grids corresponding to the n first grids are obtained according to the prediction head model. After the first predicted lane line, it is necessary to divide the predicted n first predicted lane lines into m groups, where each group includes at least one predicted lane line.
  • the q first confidence levels corresponding to the q first predicted lane lines and the corresponding q first grids are paired with the q
  • a predicted lane line is integrated to obtain a second predicted lane line, and the second predicted lane line is output as a detection result of a real lane line in the second feature map.
  • FIG. 22 For ease of understanding, the following uses FIG. 22 as an example to illustrate how to divide the n first predicted lane lines into m groups, and obtain the final detection result.
  • S 1 ⁇ s 1 , s 2 ,..., s n , ⁇
  • L ⁇ l 1 , l 2 ,..., l n , ⁇
  • the first group is used to compare
  • any group in the m group must include at least one first predicted lane line. If there is a group or multiple groups in the m group, the first predicted lane line included is greater than 2. In the case of multiple lanes, the first predicted lane line in the group needs to be integrated to obtain a second predicted lane line.
  • first predicted lane line in the group needs to be integrated to obtain a second predicted lane line.
  • the first predicted lane line in the first group is directly output as the detection result of a real lane line in the second feature map .
  • the area occupied by the q first predicted lane lines in the second feature map can be divided into multiple sub-areas, each sub-areas
  • the area includes the anchor point positions of at least one of the q first grids. If the first sub-area of the plurality of sub-areas includes at least two anchor point positions of the first grid, it can pass but
  • the processing is not limited to the following methods:
  • Processing method 1 According to the first confidence degrees corresponding to the at least two first grids, one first grid is selected as the second grid from the at least two first grids, for example, from the at least two first grids.
  • the grid selects the first grid with the largest first confidence value as the second grid, and uses the part of the first predicted lane line corresponding to the second grid in the first sub-region as the first part of the second predicted lane line ; If the second sub-region of the multiple sub-regions includes only one anchor point position of the first grid, then the first predicted lane line corresponding to the first grid included in the second sub-region is located in the second sub-region The part is used as the second part of the second predicted lane line; finally, the first part and the second part are integrated to obtain the final output second predicted lane line.
  • FIG. 23 uses FIG. 23 as an example to illustrate how to obtain a second predicted lane line based on the integration of multiple first predicted lane lines in the same group.
  • the area occupied by the six first predicted lane lines in the feature map is divided into multiple sub-regions (assuming that it is divided into sub-region 1, sub-region 2 and sub-region 3).
  • sub-region 1, sub-region 2 and sub-region 3 respectively include the anchor point positions and numbers of the first grid as shown in Figure 23 (the black dots in Figure 23 are the anchor point positions of the first grid), from It can be seen in Figure 23 that there is only one anchor point position of the first grid in subregion 1.
  • the first predicted lane line corresponding to the first grid (with a first confidence of 0.81) is located
  • the part of the sub-region 1 is used as the part 1 that constitutes the second predicted lane line; there are three anchor point positions of the first grid in the sub-region 2, and the first confidence levels corresponding to the three first grids are respectively 0.76 , 0.83 and 0.90, the part of the first predicted lane line corresponding to the first grid with a confidence of 0.90 in the sub-region 2 is taken as the part 2 constituting the second predicted lane line; similarly, there is 2 in the sub-region 3
  • the anchor point positions of the first grid, and the first confidence levels corresponding to the two first grids are 0.75 and 0.92, respectively, then the first predicted lane line corresponding to the first grid with a confidence level of 0.92 is located in the
  • the part of the sub-region 3 serves as the part 3 constituting the second predicted lane line.
  • part 1, part 2, and part 3 are sequentially integrated to obtain a second predicted lane line that is finally output.
  • FIG. 23 is still taken as an example, and another method of how to obtain a second predicted lane line based on the integration of multiple first predicted lane lines in the same group is described above. For example, suppose there are 6 first predicted lane lines in the first group, and the first confidence levels corresponding to these 6 first predicted lane lines are 0.81, 0.76, 0.83, 0.90, and 0.92, then select the first confidence level The first predicted lane line corresponding to 0.92 with the largest value is used as the baseline.
  • each sub-region the first predicted lane line part located in the sub-region is replaced according to the first confidence value from low to high, until all After completing the operation, each sub-region leaves the part with the largest first confidence value in each sub-region, and these parts constitute the final output second predicted lane line.
  • the points with good performance in the low-scoring prediction lane line are exchanged with the points with poor performance in the high-scoring prediction lane line, so as to use the low score to predict the points with good performance in the lane line.
  • Continuously revise the poor part of the high-scoring predicted lane line prediction so as to finally obtain a second predicted lane line closer to the real lane line.
  • Processing method 2 Perform a weighting operation on the part of the first predicted lane line corresponding to the at least two first grids in the first subregion, and the obtained calculation result is used as the first part of the second predicted lane line.
  • FIG. 24 For ease of understanding, the following description takes FIG. 24 as an example. It is assumed that the first subregion shown in FIG. 24 includes two first predicted lane lines A and B where the grid anchor point positions are located (not shown in FIG. 24). ), then both the predicted lane line A and the predicted lane line B fall in the first sub-area, as shown in Figure 24.
  • each predicted lane line is connected by various coordinate points
  • the coordinate points of the predicted lane lines A and B falling in the first subregion can be weighted, and then based on the weighted result of each coordinate point (ie a new series of coordinate points) obtained
  • the second predicted lane line W falls on the part of the first sub-region (ie the dotted line).
  • a coordinate point is taken as an example to illustrate how to perform the weighting operation. Since the predicted lane line has been assumed The first confidence of A is 0.8, and the first confidence of predicted lane line B is 0.9. Based on the different values of the first confidence, the weights in the weighting calculation are also different.
  • the specific weight ratio can be based on the preset value.
  • each region or sub-region is not limited.
  • It is a circular area, an elliptical area, a trapezoidal area, or even an irregular area, as long as it can realize the function of dividing the first predicted lane line, and it is not limited here.
  • the output second feature map merges the multiple levels of features of the first feature map output by different layers of the neural network, and the second feature map has both low-level features.
  • the relevant features of the receptive field make it conducive to the accurate regression of the lane line, and the relevant features of the high-level receptive field make it easy to judge whether the lane line exists;
  • group the output predicted lane line set in each group Due to the limitation of the receptive field, each predicted lane line has its own optimal prediction interval.
  • the embodiment of this application selects the optimal prediction interval of each line to integrate the lane lines, and finally combines the advantages of different predicted lane lines. Form the final output predicted lane line.
  • the detection equipment deployed with the above-mentioned trained lane line detection framework may specifically be various wheeled mobile equipment, for example, wheeled construction equipment, autonomous driving vehicles, auxiliary driving vehicles, etc., as long as they are Devices with wheeled mobility are all referred to as wheeled mobile devices described in this application.
  • Autonomous vehicles can be cars, trucks, motorcycles, buses, boats, airplanes, helicopters, lawn mowers, and recreational vehicles.
  • the embodiments of this application are not particularly limited.
  • Table 1 is the CULane data set
  • Table 2 is the CULane data set
  • the CULane dataset is annotated with 133,235 images, which are divided into 88,880 as the training set, 9675 as the verification set, and 34680 as the test set.
  • the data set includes night, day, glare, shadows and other different situations.
  • the collection area includes different application scenarios such as urban, rural, and high-speed.
  • nms-C4 is the evaluation result of the original Line-NMS algorithm on the single-layer feature map
  • ensemble-C4 is The evaluation result of the algorithm provided by the embodiment of this application on a single-layer feature map, ensemble-C3C4C5
  • the evaluation result of the algorithm provided by the embodiment of this application on a multi-layer feature map, ensemble-C3C4C5-masked is the algorithm provided by the embodiment of this application
  • the evaluation result on the multi-level feature map can be seen from the table that each sub-item in the present invention has a contribution of approximately 0.1 to F1.
  • CurveLane-S has almost the same performance as SCNN at 9GFLOPS
  • CurveLane-L has a leading advantage in almost every sub-item regardless of the amount of calculation.
  • the trained lane line detection framework in the embodiments of the present application can be used in the fields of smart cities, smart driving, etc., to detect lane lines, the following will introduce multiple application scenarios that are applied to products.
  • lane line detection is very important for the accurate positioning of autonomous vehicles and the influence on driving decisions. As one of the most important indication information on the road, the lane line can effectively guide the intelligent vehicle to drive in the constrained road area.
  • Real-time detection of the lane line on the road is an important link in the intelligent vehicle assisted driving system. This technology is conducive to Assist path planning, carry out road deviation warning and other functions, and can provide reference for precise navigation.
  • the lane line detection framework trained in the embodiments of the present application can be deployed on the autonomous vehicle as shown in FIG. 10 for real-time lane line detection on the road-related images to be detected collected by the mounted camera.
  • the integrated navigation information and lane line detection results enable the system to calculate the precise position of the vehicle and the best time to change lanes based on the vehicle's location and lane line detection results, and optimize the guidance information
  • the presentation form reduces the user’s receiving and understanding costs, what you see is what you get, and users only need to act according to the navigation information.
  • the combination of accurate lane line detection information and reasonable navigation information can effectively guide the vehicle to drive in the restricted road area, and improve the experience of using the on-board navigation software at this stage.
  • FIG. 26 is a schematic diagram of a detection device provided by an embodiment of the application.
  • the detection device 2600 may specifically include: a fusion module 2601, a first acquisition module 2602, a second acquisition module 2603, a grouping module 2604, and an integration module 2601.
  • the fusion module 2601 is used to perform feature fusion on the first feature maps output by different layers of the trained neural network to obtain a second feature map, and the input of the trained neural network is the image to be detected;
  • the first obtaining module 2602 is configured to divide the second feature map into multiple grids and obtain a first confidence level of a first grid in the multiple grids, where the first confidence level is used to indicate The first probability that the real lane line in the second feature map crosses the first grid and the first probability exceeds the first preset threshold;
  • the second acquisition module 2603 is used to pass the trained prediction head model Obtain the first predicted lane line corresponding to the first grid;
  • the grouping module 2604 is configured to divide the n first predicted lane lines into m groups when there are n first grids, m ⁇ n And n ⁇ 2;
  • the integration module 2605 used for when there are q first predicted lane lines in the first group of the m groups, according to the q first confidence levels corresponding to the q first predicted lane lines respectively And the
  • the fused second feature map is obtained.
  • the second feature map after the fusion processing has multiple levels of features.
  • the relevant features of low-level receptive fields are used to facilitate the accurate regression of lane lines, and the relevant features of high-level receptive fields are used to determine whether the lane lines exist; secondly, the output predicted lane line sets are grouped in each group. Due to the limitation of the receptive field, each predicted lane line has its own optimal prediction interval.
  • the embodiment of this application selects the optimal prediction interval of each predicted lane line to integrate the lane lines, and finally combines the different predicted lane lines. Advantages, forming the final output predicted lane line, thereby improving the accuracy of lane line detection.
  • the integration module 2605 is specifically used to: first, divide the area occupied by the q first predicted lane lines in the second feature map into multiple sub-areas, where each sub-areas The region includes the center point position of at least one of the q first grids; when the first subregion of the plurality of subregions includes the center point positions of at least two first grids, then A first grid is selected from the at least two first grids as the second grid according to the first confidence levels corresponding to the at least two first grids, and the first grid corresponding to the second grid is selected as the second grid.
  • the part of a predicted lane line located in the first sub-region is used as the first portion of the second predicted lane line, or the first predicted lane lines corresponding to the at least two first grids are located in the first sub-region.
  • Part of the region is weighted, and the result of the calculation is used as the first part of the second predicted lane line; when the second subregion of the plurality of subregions only includes the center point position of the first grid, the second The part of the first predicted lane line corresponding to the first grid included in the sub-region and located in the second sub-region is used as the second part of the second predicted lane line; the first part and the second part are integrated , To obtain the second predicted lane line.
  • the integration module 2605 is specifically further configured to: select the first confidence level from the at least two first grids according to the first confidence levels respectively corresponding to the at least two first grids.
  • the first grid with the largest degree value is used as the second grid.
  • the integration module 2605 is specifically also used to: when there is only one first predicted lane line in the first group in the m group, then the first predicted lane in the first group The line is output as a detection result of a real lane line in the second feature map.
  • the fusion module 2601 is specifically used to: First, construct a search space that is composed of multiple neural network codes and multiple feature fusion model codes, and the multiple The feature fusion model is constructed according to a variety of preset rules; then, sampling is performed in the search space to obtain a first sampling point, where the first sampling point is the coding of the first neural network and the coding composition of the first feature fusion model The first coding combination; and then generate a first lane line detection network according to the first coding combination, and train the first lane line detection network through the training set, and perform the training according to the first lane line detection network after training Performance evaluation, the performance of the first lane line detection network after training is obtained; when the sampling reaches a preset value, and the performance of the first lane line detection network after training meets the preset condition, the first lane line detection network performance after the training is passed
  • the first neural network in a lane line detection network performs feature extraction on the image to be detected to obtain first feature maps output by different layers of the first neural network;
  • the target neural network and the target feature fusion model meeting the requirements can be obtained by searching.
  • This search method can merge from a large number of neural networks and features
  • the relevant structure of the target neural network and the target feature fusion model that best match the detection task is selected in the model, which indirectly improves the detection performance of the model.
  • the multiple preset rules include at least one of the following: first, at least two high-resolution first feature maps are operated through at least one first convolution kernel respectively to obtain at least two The third feature map; and then process the resolution of the at least two third feature maps to obtain at least two fourth feature maps with the same resolution as the low-resolution first feature map; finally, the At least two fourth feature maps and the first feature map of the first resolution are merged in a preset combination manner to obtain at least one second feature map.
  • the feature fusion model is constructed by the embodiment of this application through preset fusion rules (ie, preset rules), the fusion rules are different, and the constructed feature fusion models are also different. This description of the fusion rules facilitates the understanding of the fusion rules described in this application.
  • processing the resolutions of the at least two third feature maps includes at least one of the following methods: performing down-sampling operations, pooling operations, or rolling on the at least two third feature maps.
  • Product operation includes performing down-sampling operations, pooling operations, or rolling on the at least two third feature maps.
  • the performance of the trained first lane line detection network meeting the preset condition includes: the performance of the trained first lane line detection network is the best in the performance of each lane line detection network corresponding to all sampling points.
  • the first acquisition module 2602 is specifically configured to: first divide the second feature map into multiple grids, and obtain the initial confidence of the first grid in the multiple grids , The initial confidence is used to indicate the initial probability that the real lane line in the second feature map crosses the first grid and the initial probability exceeds the initial preset threshold; finally, the The initial confidence is adjusted to obtain the first confidence of the first grid.
  • the first confidence of the first grid is obtained by adjusting the initial confidence of the first grid.
  • the purpose of adjustment is to make the prediction for the far end of the lane line more accurate during the prediction process.
  • the first acquisition module 2602 is specifically used to: First, establish a statistical model according to the distribution of the grid center point position at the far end of the real lane line in each image in the training set, and the statistical model has at least A parameter to be determined; and then estimate the parameter according to the statistical result of the grid center point position at the far end of the real lane line in each image in the training set to obtain an estimate of the parameter; then, in the training process
  • the estimation of the parameter is adjusted to obtain the determined value of the parameter, and the position of the center point of the first grid is calculated by the statistical model that determines the determined value of the parameter to obtain the The weight value of the center point position of the first grid; finally, the initial confidence level of the first grid and the weight value are calculated to obtain the first confidence level of the first grid.
  • a method for obtaining the first confidence degree from the initial confidence degree based on a statistical method is provided, which is operability.
  • the statistical model includes: Gaussian function.
  • the statistical model can be modeled as a Gaussian function.
  • FIG. 27 is a schematic structural diagram of a detection device provided by an embodiment of this application.
  • the detection device 2700 may specifically be expressed as various terminal devices, for example, The related system on the wheeled mobile device (for example, the environment perception system shown in FIG. 10) is not limited here.
  • the detection device 2700 may be deployed with the lane line detection framework described in the embodiment corresponding to FIG. 8 to implement the function of the lane line detection method in the embodiment corresponding to FIG. 11.
  • the detection device 2700 includes: a receiver 2701, a transmitter 2702, a processor 2703, and a memory 2704 (the number of processors 2703 in the detection device 2700 can be one or more, and one processor is taken as an example in FIG. 27) , Where the processor 2703 may include an application processor 27031 and a communication processor 27032. In some embodiments of the present application, the receiver 2701, the transmitter 2702, the processor 2703, and the memory 2704 may be connected by a bus or other methods.
  • the memory 2704 may include a read-only memory and a random access memory, and provides instructions and data to the processor 2703. A part of the memory 2704 may also include a non-volatile random access memory (NVRAM).
  • NVRAM non-volatile random access memory
  • the memory 2704 stores a processor and operating instructions, executable modules or data structures, or a subset of them, or an extended set of them, where the operating instructions may include various operating instructions for implementing various operations.
  • the processor 2703 controls the operation of the detection device 2700.
  • the various components of the detection device 2700 are coupled together through a bus system, where the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus.
  • bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus.
  • various buses are referred to as bus systems in the figure.
  • the method disclosed in the foregoing embodiment corresponding to FIG. 11 of the present application may be applied to the processor 2703 or implemented by the processor 2703.
  • the processor 2703 may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the foregoing method can be completed by hardware integrated logic circuits in the processor 2703 or instructions in the form of software.
  • the above-mentioned processor 2703 may be a general-purpose processor, a digital signal processing (digital signal processing, DSP), a microprocessor or a microcontroller, and may further include an application specific integrated circuit (ASIC), field programmable Field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
  • DSP digital signal processing
  • FPGA field programmable Field-programmable gate array
  • the processor 2703 can implement or execute the methods, steps, and logical block diagrams disclosed in the embodiment corresponding to FIG. 12 of the present application.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers.
  • the storage medium is located in the memory 2704, and the processor 2703 reads the information in the memory 2704, and completes the steps of the foregoing method in combination with its hardware.
  • the receiver 2701 can be used to receive input digital or character information, and generate signal inputs related to related settings and function control of the detection device 2700.
  • the transmitter 2702 can be used to output digital or character information through the first interface; the transmitter 2702 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 2702 can also include display devices such as a display screen .
  • the embodiment of the present application also provides a computer-readable storage medium.
  • the computer-readable storage medium stores a program for signal processing. When it runs on a computer, the computer executes the above-mentioned embodiment description. Perform the steps performed by the device.
  • the detection device may specifically be a chip.
  • the chip includes a processing unit and a communication unit.
  • the processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, a pin, or a circuit.
  • the processing unit can execute the computer-executable instructions stored in the storage unit, so that the chip in the training device executes the lane line detection method described in the foregoing embodiments.
  • the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit may also be a storage unit located outside the chip in the wireless access device, such as a storage unit located outside the chip.
  • Read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), etc.
  • Figure 28 is a schematic diagram of a structure of a chip provided by an embodiment of the application.
  • the Host CPU assigns tasks.
  • the core part of the NPU is the arithmetic circuit 2003.
  • the controller 2004 controls the arithmetic circuit 2003 to extract matrix data from the memory and perform multiplication operations.
  • the arithmetic circuit 2003 includes multiple processing units (process pngine, PE). In some implementations, the arithmetic circuit 2003 is a two-dimensional systolic array. The arithmetic circuit 2003 may also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2003 is a general-purpose matrix processor.
  • the arithmetic circuit 2003 fetches the data corresponding to matrix B from the weight memory 2002 and caches it on each PE in the arithmetic circuit 2003.
  • the arithmetic circuit 2003 fetches the matrix A data and matrix B from the input memory 2001 to perform matrix operations, and the partial result or final result of the obtained matrix is stored in an accumulator 2008.
  • the unified memory 2006 is used to store input data and output data.
  • the weight data directly passes through the storage unit access controller (direct memory access controller, DMAC) 2005, and the DMAC is transferred to the weight memory 2002.
  • the input data is also transferred to the unified memory 2006 through the DMAC.
  • the bus interface unit (ius interface init, BIU) 2010 is used for the interaction between the AXI bus and the DMAC and instruction fetch memory (instruction ietch iuffer, IFB) 2009.
  • the bus interface unit 2010 is also used for the instruction fetch memory 2009 to obtain instructions from the external memory, and is also used for the storage unit access controller 2005 to obtain the original data of the input matrix A or the weight matrix B from the external memory.
  • DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 2006 or the weight data to the weight memory 2002 or the input data to the input memory 2001.
  • the vector calculation unit 2007 includes a plurality of arithmetic processing units, and if necessary, further processes the output of the arithmetic circuit 2003, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison and so on. It is mainly used in the calculation of non-convolutional/fully connected layer networks in neural networks, such as Batch Normalization, pixel-level summation, and upsampling of feature planes.
  • the vector calculation unit 2007 can store the processed output vector to the unified memory 2006.
  • the vector calculation unit 2007 may apply a linear function and/or a non-linear function to the output of the arithmetic circuit 2003, such as linearly interpolating the feature plane extracted by the convolutional layer, and then, for example, a vector of accumulated values, to generate the activation value.
  • the vector calculation unit 2007 generates normalized values, pixel-level summed values, or both.
  • the processed output vector can be used as an activation input to the arithmetic circuit 2003, for example for use in a subsequent layer in a neural network.
  • the instruction fetch buffer 2009 connected to the controller 2004 is used to store instructions used by the controller 2004;
  • the unified memory 2006, the input memory 2001, the weight memory 2002, and the fetch memory 2009 are all On-Chip memories.
  • the external memory is private to the NPU hardware architecture.
  • processor mentioned in any of the foregoing may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the program of the method in the foregoing first aspect.
  • the device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physically separate.
  • the physical unit can be located in one place or distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • the connection relationship between the modules indicates that they have a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.
  • this application can be implemented by means of software plus necessary general hardware.
  • it can also be implemented by dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memory, Dedicated components and so on to achieve.
  • all functions completed by computer programs can be easily implemented with corresponding hardware.
  • the specific hardware structure used to achieve the same function can also be diverse, such as analog circuits, digital circuits or special purpose circuits. Circuit etc.
  • software program implementation is a better implementation in more cases.
  • the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a readable storage medium, such as a computer floppy disk. , U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including several instructions to make a computer device (which can be a personal computer, training device, or network device, etc.) execute the various embodiments of this application method.
  • a computer device which can be a personal computer, training device, or network device, etc.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.
  • the computer instructions may be transmitted from a website, computer, training device, or data.
  • the center transmits to another website site, computer, training equipment, or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.).
  • wired such as coaxial cable, optical fiber, digital subscriber line (DSL)
  • wireless such as infrared, wireless, microwave, etc.
  • the computer-readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device or a data center integrated with one or more available media.
  • the usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, and a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)).

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)

Abstract

本申请实施例公开了一种车道线的检测方法及相关设备，可应用于人工智能的计算机视觉领域中，该方法包括：首先通过对神经网络不同层提取到的特征进行融合，得到融合后的第二特征图，经过融合处理的第二特征图就具有了多个层次的特征，既有低层次感受野的相关特征，以利于车道线的准确回归，又有高层次感受野的相关特征，以利于判断车道线是否存在；其次，对于输出的预测车道线集合进行分组，在每一组中由于感受野的限制，每一条预测车道线都有自己的最优预测区间，本申请实施例通过选取每条预测车道线的最优预测区间做车道线的集成，最终结合不同预测车道线的优势，形成最终输出的预测车道线，从而提高了车道线检测的精度。

Description

一种车道线的检测方法及相关设备
本申请要求于2020年6月12日提交中国专利局、申请号为202010538270.3、申请名称为“一种车道线的检测方法及相关设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能领域，尤其涉及一种车道线的检测方法及相关设备。
背景技术
智能驾驶(如，自动驾驶、辅助驾驶等)技术依靠人工智能、视觉计算、雷达、监控装置和全球定位系统协同合作，让车辆可以在不需要人类主动操作下，实现自动驾驶，自动驾驶的第一步是环境信息的采集与处理，而车道线作为路面最主要的指示信息之一，它可以有效地引导智能车辆在约束的道路区域内行驶。因此，如何实时、准确检测出路面的车道线是智能车辆相关系统设计中的重要环节，可有利于协助路径规划、进行道路偏移预警等功能，并且可为精确导航提供参照。
目前，常用的车道线检测方法是基于卷积特征的车道线检测，具体来说，是将待检测图像输入神经网络进行特征提取，再将提取出来的特征(每个特征图事先划分为多个栅格)通过预测头模型进行解码，生成密集的线簇(即多条预测车道线)，最后，按照每个预测车道线的置信度(也可称为栅格的置信度，置信度反应的是是否有车道线穿过该栅格以及有多大概率穿过该栅格，置信度大于预设值的栅格则用于对车道线进行预测，置信度低于预设值的栅格则被认为对预测没有贡献)取值对线簇进行排序，以置信度取值最大的预测车道线为基线且以其他预测车道线与该基线之间的间距小于阈值为条件将车道线分为一组，以类似的方式将线簇分为若干组，并分别取每组中的基线作为本组最终对一条真实车道线的检测结果输出。
在上述的车道线检测方式中，存在一个前提：假设每一个栅格能够准确预测通过此栅格的一整条车道线，然而现实的车道线跨度过长，车道线远离栅格中心的部分难以有效预测。
发明内容
本申请实施例提供了一种车道线的检测方法及相关设备，用于对训练后的神经网络不同层输出的第一特征图进行特征融合，得到的第二特征图，并基于该第二特征图得到的多个车道线预测结果进行集成，从而提高车道线检测的精度。
基于此，本申请实施例提供以下技术方案：
第一方面，本申请实施例首先提供一种车道线的检测方法，可用于图像处理领域中，该方法包括：首先，检测设备对训练后的神经网络不同层输出的第一特征图进行特征融合，得到一个或多个第二特征图，该第一个或多个第二特征图即为融合后的特征图。这是由于神经网络不同层提取到特征图性能不同，低层特征图分辨率更高，包含更多位置、细节信 息，但是由于经过的卷积更少，其语义性更低，噪声更多；高层特征图具有更强的语义信息，但是分辨率低，对细节的感知能力较差。因此，针对神经网络不同层提取到的特征图进行特征融合，得到的第二特征图就具有多层次特征。针对得到的每个第二特征图，可将其划分为多个栅格，若该多个栅格中存在置信度取值大于预设阈值(如，预设阈值为0.6)的栅格，则大于该预设阈值的置信度可称为第一置信度，与该第一置信度对应的栅格可称为第一栅格。在得到各个第一栅格的第一置信度之后，就可进一步通过训练后的预测头模型得到与每个第一栅格对应的一条预测车道线(即第一预测车道线)。由于一条真实的车道线往往会引起多个栅格的响应，因此在预测阶段对应于一条真实车道线的预测结果会有很多，在根据预测头模型获取到n个第一栅格对应的n个第一预测车道线之后，就需要将预测得到的n个第一预测车道线分为m组，其中每组中都包括有至少一条预测车道线。若m组中的某一组存在q个第一预测车道线，则根据q个第一预测车道线分别对应的q个第一置信度及分别对应的q个第一栅格对这q个第一预测车道线进行集成，得到一个第二预测车道线，该第二预测车道线作为对第二特征图内的一个真实车道线的检测结果输出。
在本申请上述实施方式中，首先，通过特征融合的过程，输出的第二特征图融合了神经网络不同层输出的第一特征图的多个层次的特征，该第二特征图既有低层次的感受野的相关特征，使得利于车道线的准确回归，又有高层次的感受野的相关特征，使得利于判断车道线是否存在；其次，对于输出的预测车道线集合进行分组，在每一组中由于感受野的限制，每一条预测车道线都有自己的最优预测区间，本申请实施例通过选取每条线的最优预测区间做车道线的集成，最终结合不同预测车道线的优势，形成最终输出的预测车道线。
在一种可能的实现方式中，若m组中的第一组存在q个第一预测车道线(q≥2)，则可将这q个第一预测车道线在第二特征图内占据的区域划分为多个子区域，每个子区域中包括q个第一栅格中的至少一个第一栅格的中心点位置(anchor点位置)，若这多个子区域中的第一子区域包括至少两个第一栅格的anchor点位置，则根据这至少两个第一栅格分别对应的第一置信度从该至少两个第一栅格选取一个第一栅格作为第二栅格，例如，从该至少两个第一栅格选取第一置信度取值最大的第一栅格作为第二栅格，并将第二栅格对应的第一预测车道线位于第一子区域的部分作为第二预测车道线的第一部分，或，对该至少两个第一栅格分别对应的第一预测车道线位于所述第一子区域的部分进行加权运算，得到的运算结果作为第二预测车道线的第一部分；若这多个子区域中的第二子区域仅包括一个第一栅格的anchor点位置，则将该第二子区域包括的第一栅格对应的第一预测车道线位于该第二子区域的部分作为第二预测车道线的第二部分；最后，将第一部分和第二部分集成，得到最终输出的第二预测车道线。
在本申请上述实施方式中，具体阐述了如何对一组内的多条第一预测车道线进行集成，即分别取多条第一预测车道线的优势点进行集成，从而形成最终输出的检测结果，即一条第二预测车道线，最终输出的第二预测车道线由于融合了多条第一预测车道线的优点，提高了检测精度。
在一种可能的实现方式中，根据至少两个第一栅格分别对应的第一置信度从该至少两个第一栅格选取一个第一栅格作为第二栅格具体可以是：从该至少两个第一栅格选取第一 置信度取值最大的第一栅格作为第二栅格。
在本申请上述实施方式中，阐述了一种确定第二栅格的方式，即选取第一置信度取值最大的那个，灵活方便，更适用于实际的使用场景。
在一种可能的实现方式中，当m组中的第一组只存在一个第一预测车道线，则直接将该第一组中的第一预测车道线作为对所述第二特征图内的一个真实车道线的检测结果进行输出。
在本申请上述实施方式中，说明当一组中只有一条第一预测车道线时，这时候就不需要集成，直接就将该第一预测车道线作为预测结果输出。针对组里第一预测车道线数量不同，分别对应不同的处理方式，具备灵活性，并且也符合实际应用。
在一种可能的实现方式中，由于神经网络和特征融合模型都是经过训练后确定的，神经网络的网络结构可以有多种选择，本申请实施例自行构建的特征融合模型也有多种，那么如何从多种神经网络和多种特征融合模型中选择出一种神经网络(可称为目标神经网络)以及一种特征融合模型(可称为目标特征融合模型)就显得至关重要，在本申请的一些实施方式中，可通过但不限于如下方式得到：从多种已有的神经网络和本申请构建的多种特征融合模型组成的搜索空间中，搜索得到一组神经网络+特征融合模型的组合。具体地，首先，构建“神经网络+特征融合模型”的搜索空间，也就是每一对“神经网络+特征融合模型”对应的编码组合，每一次搜索的过程就包括从搜索空间中采样，采样之后可以获得一个代表“神经网络+特征融合模型”的字符串(即一个编码组合)，根据此字符串生成一个“神经网络+特征融合模型”的具体实例，即一个车道线检测网络的具体结构；之后，通过训练集中的训练数据对此实例进行训练，得到训练后的车道线检测网络，并在训练完成之后对该训练后的车道线检测网络做性能评估，得到网络性能，类似地，针对每个采样点都进行上述操作，从而得到每个采样点对应的车道线检测网络的性能，当采样达到预设次数，如，400次，则对应获取到的车道线检测网络的网络结构也有400种，并且这400个车道线检测网络的性能也经过性能评估得到了，之后，通过本申请事先设置的预设条件，根据该预设条件从这400个车道线检测网络中选择出一个车道线检测网络作为最终的目标车道线检测网络，该目标车道线检测网络就包括有对应的目标神经网络和目标特征融合模型。最后，就可通过该目标车道线检测网络中的目标神经网络对待检测图像进行特征提取，得到该目标神经网络不同层输出的第一特征图，再将该不同层输出的第一特征图作为目标特征融合模型的输入，由目标特征融合模型输出第二特征图。
在本申请上述实施方式中，通过构建神经网络和特征融合模型的搜索空间，通过搜索得到符合要求的目标神经网络和目标特征融合模型，这种通过搜索的方式能够从众多的神经网络和特征融合模型中选取到最匹配检测任务的目标神经网络和目标特征融合模型的相关结构，间接的提高了模型的检测性能。
在一种可能的实现方式中，目标特征融合模型可以是从本申请事先构建的多个特征融合模型里通过搜索的方式选取出来的。具体地，本申请实施例首先会按照多种预设的融合规则构建出多个特征融合模型，构建出来的每种特征融合模型分别对应一种融合规则，下面对其中一种融合规则进行示意：将至少两个高分辨率的第一特征图分别通过至少一个第 一卷积核操作，得到至少两个第三特征图，之后，对得到的至少两个第三特征图的分辨率进行处理，得到与低分辨率的第一特征图具有相同分辨率的至少两个第四特征图，最后，将至少两个第四特征图与第分辨率的第一特征图按预设组合方式进行融合，得到至少一个第二特征图。
在本申请上述实施方式中，由于特征融合模型是本申请实施例通过预设的融合规则(即预设规则)自行构建的，融合规则不同，构建出来的特征融合模型也不同，通过对其中一种融合规则的说明，便于理解本申请所述的融合规则。
在一种可能的实现方式中，对得到的至少两个第三特征图的分辨率进行处理具体可以是：对得到的至少两个第三特征图进行下采样操作、池化操作或卷积操作等，此处不做限定。
在本申请上述实施方式中，具体阐述了如何降低分辨率的几种处理方式，具备可选择性。
在一种可能的实现方式中，通过本申请事先设置的预设条件，根据该预设条件从多个采样得到的车道线检测网络中选择出一个车道线检测网络作为最终的目标车道线检测网络的方式具体可以包括但不限于如下方式：选择一个车道线检测网络性能在所有采样点对应的各个车道线检测网络性能中最优的作为目标车道线检测网络。
在本申请上述实施方式中，说明满足预设条件的一种方式是性能最优，符合实际应用场景的需求，间接提高了设备整体的性能。
在一种可能的实现方式中，可以先得到第一栅格的初始置信度，再根据预设方式对每个第一栅格的初始置信度进行调整，以得到每个第一栅格的第一置信度。
在本申请上述实施方式中，说明第一栅格的第一置信度是由第一栅格的初始置信度通过调整后得到的，这是由于初始置信度是粗略估计值，在用于对车道线的预测过程中存在精度不够高的问题，调整的目的是使得在预测过程中，针对车道线远端的预测能更加准确。
在一种可能的实现方式中，首先根据训练集中各个图像内真实车道线远端所在栅格anchor点位置的分布建立统计模型，该统计模型具有至少一个待确定的参数，之后根据训练集中各个图像内真实车道线远端所在栅格anchor点位置的统计结果对统计模型的参数进行估计，得到参数的估值，并且在上述对车道线检测网络进行训练的过程(如，搜索过程中的训练、全量训练等)中对该参数的估值进行调整，得到最终统计模型参数的确定值，再通过确定了参数的确定值的统计模型对第一栅格的anchor点位置进行计算，得到第一栅格的anchor点位置的权重值，最后，对第一栅格的初始置信度与权重值进行运算，得到第一栅格的第一置信度，如，对各个第一栅格的初始置信度与各个第一栅格的anchor点位置的权重值进行乘积运算，从而得到各个第一栅格的第一置信度。
在本申请上述实施方式中，提供了一种基于统计方法由初始置信度得到第一置信度的方式，具备可操作性。
在一种可能的设计中，统计模型包括：高斯函数。
在本申请上述实施方式中，由于各个车道线远端所在栅格的anchor点位置的频率热图的横向中央剖面近似高斯函数，因此，可将统计模型建模为高斯函数。
本申请实施例第二方面还提供了一种车道线检测框架，该框架具体可以包括：神经网络、特征融合模型、置信度调整模型、预测头模型和预测车道线集成模型。其中，该神经网络，用于在训练后，对输入的待检测图像进行特征提取；该特征融合模型，用于对训练后的该神经网络不同层输出的第一特征图进行特征融合，得到第二特征图；该置信度调整模型，用于将该第二特征图划分为多个栅格，并得到该多个栅格中第一栅格的第一置信度，该第一置信度用于表示该第二特征图内真实车道线穿过该第一栅格的第一概率且该第一概率超过第一预设阈值；该预测头模型，用于在训练后，得到与该第一栅格对应的第一预测车道线；该预测车道线集成模型，用于当该第一栅格有n个，将n个第一预测车道线分为m组，m≤n且n≥2；该预测车道线集成模型，还用于当该m组中的第一组存在q个第一预测车道线，则根据该q个第一预测车道线分别对应的q个第一置信度及分别对应的q个第一栅格对该q个第一预测车道线进行集成，得到一个第二预测车道线，该第二预测车道线作为对该第二特征图内的一个真实车道线的检测结果输出，q≥2。
在一种可能的设计中，该预测车道线集成模型，具体用于：将该q个第一预测车道线在该第二特征图内占据的区域划分为多个子区域，每个子区域中包括该q个第一栅格中的至少一个第一栅格的中心点位置；当该多个子区域中的第一子区域包括至少两个第一栅格的中心点位置，则根据该至少两个第一栅格分别对应的第一置信度从该至少两个第一栅格选取一个第一栅格作为第二栅格，并将该第二栅格对应的第一预测车道线位于该第一子区域的部分作为该第二预测车道线的第一部分，或，对该至少两个第一栅格分别对应的第一预测车道线位于所述第一子区域的部分进行加权运算，得到的运算结果作为第二预测车道线的第一部分；当该多个子区域中的第二子区域仅包括一个第一栅格的中心点位置，将该第二子区域包括的第一栅格对应的第一预测车道线位于该第二子区域的部分作为该第二预测车道线的第二部分；将该第一部分和该第二部分进行集成，得到该第二预测车道线。
在一种可能的设计中，该预测车道线集成模型，具体还用于：根据该至少两个第一栅格分别对应的第一置信度，从该至少两个第一栅格选取第一置信度取值最大的第一栅格作为第二栅格。
在一种可能的设计中，该预测车道线集成模型，还用于：当该m组中的第一组只存在一个第一预测车道线，则将该第一组中的第一预测车道线作为对该第二特征图内的一个真实车道线的检测结果输出。
在一种可能的设计中，该特征融合模型，具体用于：构建搜索空间，该搜索空间由多个神经网络的编码及多个特征融合模型的编码组合而成，该多个特征融合模型根据多种预设规则构建得到；在该搜索空间进行采样，得到第一采样点，该第一采样点为第一神经网络的编码及第一特征融合模型的编码构成的第一编码组合；根据该第一编码组合生成第一车道线检测网络，并通过训练集对该第一车道线检测网络进行训练；根据训练后的第一车道线检测网络进行性能评估，得到该训练后的第一车道线检测网络性能；当采样达到预设值，且该训练后的第一车道线检测网络性能满足预设条件，则通过该训练后的第一车道线检测网络中的第一神经网络对待检测图像进行特征提取，得到该第一神经网络不同层输出的第一特征图；将该第一神经网络不同层输出的第一特征图作为该训练后的第一车道线检 测网络中第一特征融合模型的输入，输出第二特征图。
在一种可能的设计中，该特征融合模型，具体还用于：将至少两个高分辨率的第一特征图分别通过至少一个第一卷积核操作，得到至少两个第三特征图；对该至少两个第三特征图的分辨率进行处理，得到与低分辨率的第一特征图具有相同分辨率的至少两个第四特征图；将该至少两个第四特征图与该第分辨率的第一特征图按预设组合方式进行融合，得到至少一个第二特征图。
在一种可能的设计中，该特征融合模型，具体还用于：对该至少两个第三特征图进行下采样操作、池化操作或卷积操作。
在一种可能的设计中，该训练后的第一车道线检测网络性能满足预设条件包括：该训练后的第一车道线检测网络性能在所有采样点对应的各个车道线检测网络性能中最优。
在一种可能的设计中，该置信度调整模型，具体用于：将该第二特征图划分为多个栅格，并得到该多个栅格中第一栅格的初始置信度，该初始置信度用于表示该第二特征图内真实车道线穿过该第一栅格的初始概率且该初始概率超过初始预设阈值；通过预设方式对该初始置信度进行调整，得到该第一栅格的第一置信度。
在一种可能的设计中，该置信度调整模型，具体还用于：根据训练集中各个图像内真实车道线远端所在栅格中心点位置的分布建立统计模型，该统计模型具有至少一个待确定的参数；根据该训练集中各个图像内真实车道线远端所在栅格中心点位置的统计结果对该参数进行估计，得到该参数的估值；在训练过程中对该参数的估值进行调整，得到该参数的确定值；通过确定了该参数的确定值的该统计模型对该第一栅格的中心点位置进行计算，得到该第一栅格的中心点位置的权重值；对该第一栅格的初始置信度与该权重值进行运算，得到该第一栅格的第一置信度。
在一种可能的设计中，该统计模型，包括：高斯函数。
在一种可能的设计中，训练后的该框架部署在检测设备上。
在一种可能的设计中，该检测设备包括：轮式移动设备，例如，例如，可以是轮式施工设备、自动驾驶车辆、辅助驾驶车辆等，只要是具备轮式可移动的设备就都称为本申请该的轮式移动设备。为便于理解，在本申请以下实施例中，均以轮式移动设备为自动驾驶车辆为例进行说明，自动驾驶车辆可以为轿车、卡车、摩托车、公共汽车、船、飞机、直升飞机、割草机、娱乐车、游乐场车辆、施工设备、电车、高尔夫球车、火车、和手推车等，本申请实施例不做特别的限定。
本申请实施例第三方面提供一种检测设备，该检测设备具有实现上述第一方面或第一方面任意一种可能实现方式的方法的功能。该功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。该硬件或软件包括一个或多个与上述功能相对应的模块。
本申请实施例第四方面提供一种检测设备，可以包括存储器、处理器以及总线系统，其中，存储器用于存储程序，处理器用于调用该存储器中存储的程序以执行本申请实施例第一方面或第一方面任意一种可能实现方式的方法。
本申请第五方面提供一种计算机可读存储介质，该计算机可读存储介质中存储有指令，当该指令在计算机上运行时，使得计算机可以执行上述第一方面或第一方面任意一种可能 实现方式的方法。
本申请实施例第六方面提供了一种计算机程序或计算机程序产品，当该计算机程序或计算机程序产品在计算机上运行时，使得计算机执行上述第一方面或第一方面任意一种可能实现方式的方法。
附图说明
图1为特征融合的两种方式的对比示意图；
图2为安装在自车上的摄像机拍摄到的行驶路面的两个图像的示意图；
图3为利用车道线检测算法预测到的车道线的可视化效果的一个示意图；
图4为车道线检测中的不同长尾情况的一个示意图；
图5为一种典型的车道线检测架构的一个示意图；
图6为预测头模型的工作原理的一个示意图；
图7为通过Line-NMS处理后，得到预测车道线的一个示意图；
图8为本申请实施例构建的经过训练后的车道线检测框架的一个示意图；
图9为本申请实施例提供的图像处理系统架构的一个示意图；
图10为本申请实施例提供的训练后的车道线检测框架在自动驾驶车辆上部署位置的一个示意图；
图11为本申请实施例提供的车道线的检测方法的一种流程示意图；
图12为本申请实施例提供的预设的融合规则的一个示意图；
图13为本申请实施例提供的编码规则的一个示意图；
图14为本申请实施例提供的一种通过搜索得到目标车道线检测网络的示意图；
图15为本申请实施例提供的将神经网络的基础快划分为5个阶段的示意图；
图16为本申请实施例提供的全量训练的一个示意图；
图17为本申请实施例提供的将待检测图像划分为多个栅格以及从多个栅格中确定第一栅格的一个示意图；
图18为本申请实施例提供的由初始置信度计算得到第一置信度的一个示意图；
图19为本申请实施例提供的基于统计方法由初始置信度得到第一置信度一个示意图；
图20为本申请实施例提供的各个车道线远端所在栅格中心点的频率热图及高斯函数的示意图；
图21为本申请实施例提供的通过训练后的预测头模型得到的第一预测车道线具有不同层次特征的一个示意图；
图22为本申请实施例提供的得到第二预测车道线的一个流程图；
图23为本申请实施例提供的根据同一组内的多个第一预测车道线集成得到一个第二预测车道线的方法的一个示意图；
图24为本申请实施例提供的根据同一组内的多个第一预测车道线集成得到一个第二预测车道线的方法的另一示意图；
图25为本申请实施例提供的一种应用场景的示意图；
图26为本申请实施例提供的检测设备的一个示意图；
图27为本申请实施例提供的检测设备的另一示意图；
图28为本申请实施例提供的芯片的一种结构示意图。
具体实施方式
本申请实施例提供了一种车道线的检测方法及相关设备，用于对训练后的神经网络不同层输出的第一特征图进行特征融合，得到的第二特征图，并基于该第二特征图得到的多个车道线预测结果进行集成，从而提高车道线检测的精度。
本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的术语在适当情况下可以互换，这仅仅是描述本申请的实施例中对相同属性的对象在描述时所采用的区分方式。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，以便包含一系列单元的过程、方法、系统、产品或设备不必限于那些单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它单元。
本申请实施例涉及了许多关于车道线检测及神经网络的相关知识，为了更好地理解本申请实施例的方案，下面先对本申请实施例可能涉及的相关概念及背景知识进行介绍。
(1)神经网络
神经网络可以是由神经单元组成的，具体可以理解为具有输入层、隐含层、输出层的神经网络，一般来说第一层是输入层，最后一层是输出层，中间的层数都是隐含层。其中，具有很多层隐含层的神经网络则称为深度神经网络(deep neural network，DNN)。神经网络中的每一层的工作可以用数学表达式
Figure PCTCN2021092070-appb-000001
来描述，从物理层面，神经网络中的每一层的工作可以理解为通过五种对输入空间(输入向量的集合)的操作，完成输入空间到输出空间的变换(即矩阵的行空间到列空间)，这五种操作包括：1、升维/降维；2、放大/缩小；3、旋转；4、平移；5、“弯曲”。其中1、2、3的操作由
Figure PCTCN2021092070-appb-000002
完成，4的操作由“+b”完成，5的操作则由“a()”来实现。这里之所以用“空间”二字来表述是因为被分类的对象并不是单个事物，而是一类事物，空间是指这类事物所有个体的集合，其中，W是神经网络各层的权重矩阵，该矩阵中的每一个值表示该层的一个神经元的权重值。该矩阵W决定着上文所述的输入空间到输出空间的空间变换，即神经网络每一层的W控制着如何变换空间。训练神经网络的目的，也就是最终得到训练好的神经网络的所有层的权重矩阵。因此，神经网络的训练过程本质上就是学习控制空间变换的方式，更具体的就是学习权重矩阵。在训练神经网络的过程中，因为希望神经网络的输出尽可能的接近真正想要预测的值，可以通过比较当前网络的预测值和真正想要的目标值，再根据两者之间的差异情况来更新每一层神经网络的权重矩阵(当然，在第一次更新之前通常会有初始化的过程，即为神经网络中的各层预先配置参数)，比如，如果网络的预测值高了，就调整权重矩阵让它预测低一些，不断的调整，直到神经网络能够预测出真正想要的目标值。因此，就需要预先定义“如何比较预测值和目标值之间的差异”，这便是损失函数(loss function)或目标函数(objective function)，它们是用于衡量预测值和目标值的差异的重要方程。其 中，以损失函数举例，损失函数的输出值(loss)越高表示差异越大，那么神经网络的训练就变成了尽可能缩小这个loss的过程。
(2)特征融合(feature fusion)
将神经网络提取的不同特征通过某种方法生成新的特征，从而使新特征对分类、识别或检测等更有效，特征融合一般具有两种方式：concat和add。其中，concat是系列特征融合方式，即直接将两个特征进行连接，两个输入特征x和y的维数若为p和q，输出特征z的维数为p+q；add则是一种并行融合策略，是将两个特征向量进行组合，对于输入特征x和y，得到通道数不变的新的特征z。具体如图1所示，也就是说，add是描述图像的特征下的信息量增多了，但是描述图像的维度本身并没有增加，只是每一维下的信息量在增加；而concat是通道数的合并，也就是说描述图像本身的特征增加了，而每一特征下的信息是没有增加。
(3)轮式移动设备
是集环境感知、动态决策与规划、行为控制与执行等多功能于一体的综合系统，也可称为轮式移动机器人或轮式智能体，例如，可以是轮式施工设备、自动驾驶车辆、辅助驾驶车辆等，只要是具备轮式可移动的设备就都称为本申请所述的轮式移动设备。为便于理解，在本申请以下实施例中，均以轮式移动设备为自动驾驶车辆为例进行说明，自动驾驶车辆可以为轿车、卡车、摩托车、公共汽车、船、飞机、直升飞机、割草机、娱乐车、游乐场车辆、施工设备、电车、高尔夫球车、火车、和手推车等，本申请实施例不做特别的限定。
(4)感知
保证感知的准确性是自动驾驶能够安全进行的首要条件，感知从传感器角度讲可以有多种模块，如激光感知模块、视觉感知模块、毫米波感知模块等，而视觉感知模块作为关键模块之一，被广泛应用于诸如先进驾驶辅助系统(advanced driver assistant system，ADAS)、自动驾驶系统(autonomous driving system，ADS)中，其能够给安装有该系统的轮式移动设备(如，自动驾驶车辆)提供周围环境的相关信息(如，路面上的车道线)，该相关信息也可称为感知信息，感知信息为规控合理的决策提供了坚实的依据。
在本申请实施例中，所利用的感知信息为视觉感知模块(如，安装在轮式移动设备上的摄像头)拍摄的自车行驶路面的相关视频(后续需提取图像)或图像，如图2所示，为安装在自车上的摄像机拍摄到的行驶路面的两个图像(图像1和图像2)。获取到的相关图像则作为待检测图像，用于对行驶路面的车道线进行检测，图3为利用车道线检测算法，对视觉感知模块实时输入的待检测图像进行检测，从而预测到的车道线位置，其可视化结果如图3所示，对获取到的4张待检测图像分别检测并输出4条、2条、5条、3条预测车道线。
(5)目标检测与车道线检测
目标检测，也叫目标提取，是一种基于目标几何和统计特征的图像分割，其关注图像中特定的物体目标，要求同时获得这一特定物体目标的类别信息和位置信息，其准确性和实时性是整个系统的一项重要能力，目标检测在很多场景都有应用，如，无人驾驶的安防 系统、智能化交通系统、智能监控系统等。
车道线检测可以认为是一种特殊的目标检测，由于一般的目标检测和车道线检测在目标特征方面具有较大差异，一般的目标检测中的目标物体(如，猫、狗、人脸等)的所占据的区域具有一定长宽比，容易被识别，而车道线检测中的车道线是细长型的，其跨度较长，并且宽度较窄，很难积累足够的鲁棒的特征去描述整条车道线，不容易被识别，这对神经网络提出了苛刻的要求，因此车道线检测比一般的目标检测具有更大挑战性。此外，驾驶场景通常相当复杂，随着自动驾驶车辆的自动化等级越来越高，车道线检测中的一些长尾现象也越来越受到重视，如图4所示的弯曲车道线、远端车道线、分叉车道线等都会存在长尾现象，这些车道线能否被良好检测出决定了自动驾驶车辆的安全性能是否良好。
在智能驾驶领域中，车道线检测，对于自动驾驶车辆准确定位，以及对驾驶决策的影响是相当重要的。而车道线作为路面最主要的指示信息之一，它可以有效地引导智能驾驶车辆在约束的道路区域内行驶，实时地检测出路面的车道线是智能驾驶车辆的ADAS或ADS中的重要环节，有利于协助路径规划、进行道路偏移预警等功能，并且可为精确导航提供参照。现有的车道检测方法大致分为如下两类：
a、基于手工特征的车道线检测。
基于手工特征的车道线检测方法依赖于手工制作的特征(包括颜色、边缘信息等)，这些特征结合霍夫(Hough)变换和卡尔曼(Kalman)滤波实现对车道线的检测，在大部分情况下具有鲁棒的检测性能。然而，由于不能很好地适应现实生活中复杂多变的车道线，这种基于手工特征的车道线检测方式无法有效地为自动驾驶的下游模块(如，规控系统)提供可靠的车道线检测结果，在自动驾驶研究中已很少提及。
b、基于卷积特征的车道线检测。
随着深度学习的不断发展，现阶段车道线检测领域发展出多种车道线检测方法，其中一种典型的车道线检测架构如图5所示，该车道线检测架构的主要流程为：首先，将待检测图像输入一个神经网络(如，卷积神经网络)进行特征提取，得到神经网络最后一层输出的特征图，并将提取出来的该特征图通过预测头模型进行解码，生成密集的线簇(即多个预测车道线)，最后，通过对线簇进行线性非极大值抑制(line non-maximum suppression，Line-NMS)处理，输出对该待检测图像内车道线的最终预测结果。
具体地，上述预测头模型是按照如图6所示的方式工作的，首先，将神经网络输出的特征图划分为尺寸为w×h的多个栅格，其中一个栅格可记为{G ij}，特征图上的每个栅格G ij的中心点称为anchor点，如果G ij的中心点靠近地面上某条真实车道线，则该G ij负责检测该车道线。由于一条真实的车道线可能跨越多个栅格，因此一条车道线可能对应多个栅格，栅格G ij的置信度S ij反映的是车道线通过该栅格G ij的概率，在对预测头模型进行训练的阶段，由于训练图像上的每条真实车道线都是被标注过的，因此，在训练阶段，一条车道线通过栅格G ij的概率只有两种情况：通过，则栅格G ij的置信度S ij取值为1；不通过，则栅格G ij的置信度S ij取值为0。而在检测阶段，神经网络提取到的特征图上是否存在车道线以及在特定位置存在车道线是以概率的形式存在，因此，一条车道线通过栅格G ij的概率也就是车道线在栅格所在位置存在的概率。对于每个栅格G ij，预测头模型将预测一组偏移量Δx ijz 和一个终点位置(即车道线远端在特征图上消失的位置)，其中Δx ijz是地面真实车道线与通过anchor点的垂线之间的水平距离，如图6中示意的一组偏移量Δx ijz为双向箭头所示意的4个，其中，黑点位置为anchor点。在训练阶段，通过该预测头模型把输入的训练图像上的所有真实车道线按照上述说明进行编码，得到每条真实车道线所穿过的栅格对应的一组Δx ijz和一个终点位置，并利用各个栅格对应的一组Δx ijz和终点位置修正模型输出，在预测阶段，待检测图像中的每条车道线都可以通过模型预测的Δx ijz和终点位置恢复出来。待检测图像上一条真实的车道线往往会引起多个栅格的响应，因此在预测阶段，对应于一条真实车道线的预测结果会有很多条预测的车道线，需要借助Line-NMS进行处理，从而生成最终的预测结果。
通过Line-NMS处理的过程具体是：在获取到预测头模型输出的线簇(即多条预测车道线)后，按照置信度取值对各个预测车道线进行排序，然后以置信度取值最大的预测车道线为基线且以其他预测车道线与该基线之间的间距小于阈值为条件将车道线分为一组，以类似的方式将线簇分为若干组，并分别取每组中的基线作为本组最终对一条真实车道线的检测结果输出，通过Line-NMS处理后输出的预测车道线如图7右边部分所示，最终输出3条预测车道线，图7的左边部分则是根据置信度取值对线簇中的所有预测车道线进行分组的示意(分为3组)。
上述这种基于卷积特征的车道线检测的方式比较原始，整个算法设计有一个前提，就是假设每一个栅格能够准确预测通过此栅格的一整条车道线，然而现实的车道线跨度过长，车道线远离栅格中心点的部分难以有效预测；并且，使用直筒型的网络主干，不能够有效扩展网络感受野，限制了网络的表达能力。
为解决上述现有的车道线检测的相关问题，本申请实施例提供了一种车道线的检测方法，用于对训练后的神经网络不同层输出的第一特征图进行特征融合，得到的第二特征图，并基于该第二特征图得到的多个车道线预测结果进行集成，从而提高车道线检测的精度。
下面结合附图，对本申请的实施例进行描述。本领域普通技术人员可知，随着技术的发展和新场景的出现，本申请实施例提供的技术方案对于类似的技术问题，同样适用。
首先，对实现本申请实施例的车道线检测功能的框架进行介绍，该框架可称为车道线检测框架(也可称为车道线检测算法)，通过本申请构建的车道线检测框架达到对待检测图像中的车道线进行检测的目的，具体请参阅图8，图8为本申请构建的一种经过训练后的车道线检测框架，该车道线检测框架由如下部分组成：神经网络101、特征融合模型102、预测头模型103、预测车道线集成模块104，其中，神经网络101用于从输入的待检测图像提取特征，从而得到神经网络101不同层的多个特征图，不同的特征图提取到的关于该待检测图像的特征也不同，有的特征图具有高层次的感受野，而有的特征图具有低层次的感受野，神经网络101可通过设置搜索参数，从符合搜索参数的各个神经网络构成的搜索空间搜索得到；特征融合模型102用于对神经网络101提取到的不同层的特征图进行特征融合，得到融合后的特征图，融合后的特征图即有低层次的感受野，以利于车道线的准确回归，又有高层次的感受野，以判断车道线是否存在，该特征融合模型102为本申请实施例根据多种不同的预设规则构建得到多个，类似地，再设置搜索参数，从符合设置的搜索参 数的多个构建好的特征融合模型中搜索得到。需要说明的是，神经网络101可在单独设置的搜索空间搜索得到，也可以在与多个特征融合模型共同构成的搜索空间搜索得到，类似地，特征融合模型102可在单独设置的搜索空间搜索得到，也可以在与多个神经网络共同构成的搜索空间搜索得到，此处不做限定。预测头模型103是沟通特征图与真实车道线的桥梁，用于根据融合后具有不同层次的特征图，解码生成对应原图的车道线，具体地，用于针对融合后的特征图划分的栅格进行车道线预测，每个置信度高于预设阈值的栅格都可对应预测出一个车道线；预测车道线集成模块104则是对所有预测的车道线进行分组，在每一组中由于感受野的限制，每一条车道线都有自己的最优预测区间，预测车道线集成模块104通过选中每条车道线的最优预测区间做车道线的集成，最终结合不同车道线的预测优势，形成最终的车道线。
需要说明的是，在本申请的一些实施方式中，所构建的车道线检测框架还可以具有置信度调整模型105，这是由于栅格的anchor点具有局部性，并且在实际应用场景中，用户更加关注车道远端的精度，为进一步增加预测精度，置信度调整模型105用于对栅格的初始置信度进行调整，通过增加车道线远端对应栅格的anchor点所占权重，充分利用车道线远端的局部信息，改善车道线远端精度。
需要说明的是，上述图8所述的车道线检测框架是已经经过训练后的车道线检测框架，在构建好车道线检测框架后，需要先对整个车道线检测框架进行训练，训练好的车道线检测框架才能应用到具体的车道线检测任务中。具体地，可基于图9所示的图像处理系统200对车道线检测框架的整个训练和检测的流程进行介绍，在图9中，图像处理系统200包括检测设备210、训练设备220、数据库230、客户设备240、数据存储系统250和数据采集设备260，检测设备210中包括计算模块211，所述计算模块211实质为本申请实施例提供的训练后的车道线检测框架201。
其中，数据采集设备260(如，搭载在轮式移动设备上的摄像机)用于获取用户需要的开源的大规模数据集(即训练集)，并将数据集存入数据库230中，训练设备220基于数据库230中的维护的数据集对车道线检测框架201内的各个模型进行训练(如，对图8所示的神经网络101、特征融合模型102等进行训练)。检测设备210可以调用数据存储系统250中的数据、代码等，也可以将数据、指令等存入数据存储系统250中。数据存储系统250可以置于检测设备210中，也可以为数据存储系统250相对检测设备210是外部存储器。
经由训练设备220训练的车道线检测框架201可以应用于不同的系统或设备(即检测设备210)中，例如，检测设备210可以是各种轮式移动设备(轮式施工设备、自动驾驶车辆、辅助驾驶车辆等)，自动驾驶车辆还可以为轿车、卡车、摩托车、公共汽车、船、飞机、直升飞机、割草机、娱乐车、游乐场车辆、施工设备、电车、高尔夫球车、火车、和手推车等等。
在图9中，检测设备210配置有I/O接口212，与外部设备进行数据交互，“用户”可以通过客户设备240向I/O接口212输入数据。如，客户设备240可以是自动驾驶车辆搭载的摄像设备，通过该摄像设备拍摄的图像作为输入数据输入至检测设备210的计算模块 211，由计算模块211中的车道线检测框架201对输入的该图像进行检测后得出检测结果，再将该检测结果输出至摄像设备或直接在检测设备210的显示界面(若有)进行显示；此外，在本申请的一些实施方式中，客户设备240也可以集成在检测设备210中，如，当检测设备210为自动驾驶车辆时，则可以直接通过该自动驾驶车辆的摄像头拍摄到图像或者接收其他设备(如，手机)发送的图像，再由该自动驾驶车辆内的计算模块211对该图像进行检测后得出检测结果，并直接将该检测结果呈现在手机的显示界面。此处对检测设备210与客户设备240的产品形态不做限定。
值得注意的，图9仅是本申请实施例提供的一种图像处理系统架构的示意图，图中所示设备、器件、模块等之间的位置关系不构成任何限制，例如，在图9中，数据存储系统250相对检测设备210是外部存储器，在其它情况下，也可以将数据存储系统250置于检测设备210中；在图9中，客户设备240相对检测设备210是外部设备，在其他情况下，客户设备240也可以集成在检测设备210中。
还需要说明的是，在本申请一些实施方式中，车道线检测框架201还可以拆分成多个子模块/子单元以共同实现本申请实施例所提供的方案，具体此处不做限定。
还需要说明的是，上述实施例所述的车道线检测框架201的训练可以是均在云侧实现，例如，可以由云侧的训练设备220(该训练设备220可设置在一个或多个服务器或者虚拟机上)获取训练集，并根据训练集内的多组训练数据对车道线检测框架进行训练，得到训练后的车道线检测框架201，之后，该训练后的车道线检测框架201再发送给检测设备210进行应用，例如，发送给自动驾驶车辆的相关系统(如，ADAS、ADS等)进行车道线检测，示例性地，图9对应的系统架构中所述，就是由训练设备220对车道线检测框架201进行整体训练，训练后的车道线检测框架201再发送给检测设备210进行使用；上述实施例所述的车道线检测框架201的训练也可以是均在终端侧实现，即训练设备220可以是位于终端侧，例如，可以由轮式移动设备(如，自动驾驶车辆、辅助驾驶车辆等)获取训练集，并根据训练集内的多组训练数据对车道线检测框架201进行训练，得到训练后的车道线检测框架201，该训练后的车道线检测框架201就可以直接在该终端设备使用，也可以由该终端设备发送给其他的设备进行使用。具体本申请实施例对车道线检测框架201在哪个设备(云侧或终端侧)上进行训练或应用不做限定。
需要说明的是，由于图8、图9所述的本申请构建的训练后的车道线检测框架(或图9所述的检测设备210)可部署在各种轮式移动设备中，用于对该轮式移动设备上搭载的摄像机拍摄到的路面上的相关感知信息(如，视频或图像)进行处理，输出预测车道线，该预测车道线输入轮式移动设备的下游模块进行进一步处理，下面以轮式移动设备为自动驾驶车辆为例对训练后的车道线检测框架的部署位置及自动驾驶车辆的总体架构进行说明，具体请参阅图10，图10示意的是自上而下的分层式体系架构，各系统之间可有定义接口，用于对系统间的数据进行传输，以保证数据的实时性和完整性。下面对各个系统进行简单介绍：
(1)环境感知系统
环境感知是自动驾驶车辆中最为基础的一个部分，无论是做驾驶行为决策还是全局路 径规划，都需要建立在环境感知的基础上，依据对道路交通环境的实时感知结果，进行相对应的判断、决策和规划，使车辆实现智能驾驶。
环境感知系统主要是利用各种传感器获取相关的环境信息，从而完成对环境模型的构建以及对于交通场景的知识表达，所使用的传感器包括一个或多个摄像机、单线雷达(SICK)、四线雷达(IBEO)、三维激光雷达(HDL-64E)等，其中，摄像机采集到的视频或图像主要是负责红绿灯检测、车道线检测、道路指示牌检测、车辆识别等；其他的激光雷达传感器主要负责动\静态的关键障碍物的检测、识别和跟踪，以及道路边界、灌木带、周边建筑物等非关键障碍物的检测和提取。在本申请实施例中，利用的是摄像机拍摄到的路面相关的感知数据，即路面的待检测图像，该待检测图像(可以是摄像机直接拍摄的图像，也可以是从摄像机拍摄的视频中提取到的图像，这里对待检测图像不做限定)输入到本申请实施例构建的训练后的车道线检测框架，经过该训练后的车道线检测框架检测后，输出对待检测图像中真实车道线的预测结果(即预测车道线)，该预测结果可以直接作为自动驾驶车辆下游模块(如，自主决策系统)进行进一步处理，也可以输入到环境感知系统中的其他模型(如，地图构造模型，图10未示意出)，用于与其他感知信息共同构建得到一张能表达道路环境的占据栅格地图(occupancy grid map，OGM)，最后将该OGM发送给自主决策系统做进一步的决策和规划。
(2)自主决策系统
自主决策系统是自动驾驶车辆中的关键组成部分，该系统主要分为行为决策和运动规划这两个核心子系统，其中，行为决策子系统主要是通过运行全局规划层来获取全局最优行驶路线，以明确具体驾驶任务，再根据环境感知系统发来的当前实时道路信息(如，预测车道线)，基于道路交通规则和驾驶经验，决策出合理的驾驶行为，并将该驾驶行为指令发送给运动规划子系统；运动规划子系统则是根据接收的驾驶行为指令以及当前的局部环境感知信息，基于安全性、平稳性等指标规划处一条可行驾驶轨迹，并发送给控制系统。
(3)控制系统
控制系统具体来说也分为两个部分：控制子系统和执行子系统，其中，控制子系统用于将自主决策系统产生的可行驾驶轨迹转化为各个执行模块的具体执行指令，并传递给执行子系统；执行子系统接收来自控制子系统的执行指令后将其发送给各个控制对象，对车辆的转向、制动、油门、档位等进行合理的控制，从而使车辆自动行驶以完成对应的驾驶操作。
需要说明的是，在自动驾驶车辆的行驶过程中，自动驾驶车辆的驾驶操作的准确性主要取决于控制系统产生的针对各个执行模块的具体执行指令是否准确，而准确与否又取决于自主决策系统，自主决策系统又面临不确定性，该不确定性因素主要包括以下几个方面：1)环境感知系统中的各个传感器特性及标定误差带来的不确定性，不同传感器的感知机理、感知范围以及相应的误差模式是不一样的，并且其在自动驾驶车辆上安装带来的标定误差，最终都会反映在感知信息的不确定性上；2)环境感知系统数据处理延迟带来的不确定性，这是因为道路环境复杂，数据信息量庞大，这使得环境感知系统数据处理的计算量也大，而整个环境是时刻在变化的，这就必然会导致数据信息的延迟，从而影响自主决策系统的 正确判断；3)对感知信息处理方式的不同也会带来不确定性，以本申请实施例为例，若采用传统的基于手工特征的车道线检测方法或如图5所示的基于卷积特征的车道线检测方法，则会带来车道线检测精度不高的问题，已有的车道线检测方式不能满足自动驾驶车辆越来越苛刻的应用场景，若能提高车道线检测精度，也就能相应减少自主决策系统的不确定性，进而提高控制系统产生的针对各个执行模块的具体执行指令的准确性。
还需要说明的是，图10所示的自动驾驶车辆的总体架构仅为示意，在实际应用中，可包含更多或更少的系统/子系统或模块，并且每个系统/子系统或模块可包括多个部件，具体此处不做限定。
本申请实施例所提供的车道线的检测方法可基于上述图8、图9所示的训练后的车道线检测框架进行，并且本申请实施例所提供的车道线的检测方法可应用于对各种智能行驶(如，无人驾驶、辅助驾驶等)的轮式移动设备进行运动规划(如，速度规划、驾驶行为决策、全局路径规划等)的场景中，请参阅图11，图11为本申请实施例提供的车道线的检测方法的一种流程示意图，具体包括：
1101、对训练后的神经网络不同层输出的第一特征图进行特征融合，得到第二特征图。
部署有上述所述车道线检测框架的检测设备首先对训练后的神经网络不同层输出的第一特征图进行特征融合，得到一个或多个第二特征图，该第一个或多个第二特征图即为融合后的特征图。这是由于神经网络不同层提取到特征图性能不同，低层特征图分辨率更高，包含更多位置、细节信息，但是由于经过的卷积更少，其语义性更低，噪声更多；高层特征图具有更强的语义信息，但是分辨率低，对细节的感知能力较差。因此，针对神经网络不同层提取到的特征图进行特征融合，得到的第二特征图就具有多层次特征。
需要说明的是，该步骤1101可基于图8对应实施例中的训练后的特征融合模型102计算得到。具体地，基于图8对应的实施例，待检测图像首先经由训练后的神经网络101提取到不同层的多个特征图(即第一特征图)，再经由训练后的特征融合模型102进行特征融合，输出一个或多个融合后的特征图(即第二特征图)。
还需要说明的是，在本申请的一些实施方式中，特征融合模型102可以是从本申请事先构建的多个特征融合模型里通过搜索的方式选取出来的。具体地，本申请实施例首先会按照多种预设的融合规则构建出多个特征融合模型，构建出来的每种特征融合模型分别对应一种融合规则，下面对其中一种融合规则进行示意，假设将神经网络各个层分为x个阶段(如，x为4)，每个阶段包括神经网络的一层或多层，每个阶段的最后一层提取到的特征图作为第一特征图，由于不同阶段之间具有不同的分辨率，不同的通道数，以及不同的感受野。假设G 1，...t表示神经网络不同阶段输出的第一特征图，并且从特征图F 1到特征图F t的空间分辨率是以一定倍数(如，2倍数)逐渐变小的，假设事先设置待构建的特征融合模型包含y个融合层{O i}，其中，y的取值为超参数，融合层是指对多个第一特征图进行特征融合的一次融合操作，对于每一个融合层，在F 1，...t中挑出z个(如，2个)分辨率较高的第一特征图作为输入，再从F 1，...t中挑选出一个分辨率较低的一个第一特征图作为比对，z个第一特征图首先通过至少一个卷积核操作，输出z个第三特征图，再对这z个第三特征图的分辨率进行处理(如，下采样操作、池化操作、卷积操作等，此处不做限定)，得到与挑选 出的分辨率较低的第一特征图具有相同分辨率的z个第四特征图，然后将z个第四特征图按照预设组合方式进行再次融合，得到至少一个第二特征图。不同的融合规则，根据事先设置好的编码方式，会对应生成一个编码，通过该编码，检测设备可识别出是哪种融合规则，继而可知道是哪种特征融合模型。
为便于理解，下面以图12为例，对其中一种融合规则以及对应的编码进行说明：假设神经网络不同阶段输出的第一特征图(特征图也可称为特征层)分别记为图12所示的“0”、“1”、“2”，其中，特征图“0”和特征图“1”是分辨率较高的两个特征图，特征图“2”是分辨率较低的一个特征图。首先，对特征图“0”和特征图“1”分别做一次1x1的卷积核操作，得到具有相同通道数的输出，再分别对该输出进行上采样或下采样，得到与特征图“2”具有相同分辨率的特征图，在将具有相同分辨率的三个特征图在指定分辨率上做堆叠，形成多个融合后的特征图(即融合层)，再对多个融合层进行再次融合，得到最后输出的一个特征图，上述所述的融合规则就可按照预设的编码规则编码为“012-022”，其各个数字代表的意思如图13所示，根据该编码，检测设备就可识别出对应的融合规则。
需要说明的是，图12对应实施例所示融合规则中涉及的两个特征图之间的融合是以concat方式进行示意的，实际上，在本申请的一些实施方式中，具体的融合方式还可以是上述所述的add方式，此处不做限定。
在本申请实施例中，由于神经网络和特征融合模型都是经过训练后确定的，神经网络的网络结构可以有多种选择，本申请实施例自行构建的特征融合模型也有多种，那么如何从多种神经网络和多种特征融合模型中选择出一种神经网络(可称为目标神经网络)以及一种特征融合模型(可称为目标特征融合模型)就显得至关重要，在本申请的一些实施方式中，可通过但不限于如下方式得到：
(1)从多种已有的神经网络和本申请构建的多种特征融合模型组成的搜索空间中，搜索得到一组神经网络+特征融合模型的组合。
由于车道线检测属于一种特殊的目标检测，因此，现有的很多用于目标检测的神经网络，如，残差网络(RestNet、ResNeXt等)、快速生成区域的卷积神经网络(faster region-convolutional neural network，Faster R-CNN)、单级多框防御(single shot multibox defender，SSD)等在经过一些网络结构的调整后都可以应用于车道线检测中，但是在针对具体的车道线检测任务中，不同的网络结构所表现出来的性能也是不一样的；类似地，虽然本申请实施例通过多种预设的融合规则构建得到了多个特征融合模型，但是在具体的某一类型或某个车道线检测任务中，具有不同融合规则的特征融合模型表现来的性能也是不同的。基于此，本申请对诸多车道线检测模型的进行分析、实验以及评估，确定一种适宜车道线检测任务的基本模型结构为“神经网络+特征融合模型”的组合架构，该组合架构可称为车道线检测网络，然后对“神经网络+特征融合模型”组成的车道线检测网络进行编码，所有的“神经网络+特征融合模型”组成的车道线检测网络对应的编码组成搜索空间。通过设置对应的搜索规则(如，随机采样)在该搜索空间进行搜索，得到符合本申请实施例要求的目标“神经网络+特征融合模型”，即目标车道线检测网络。
具体地，首先构建“神经网络+特征融合模型”的搜索空间，也就是每一对“神经网络 +特征融合模型”对应的编码组合，每一次搜索的过程就包括从搜索空间中采样，采样之后可以获得一个代表“神经网络+特征融合模型”的字符串(即一个编码组合)，根据此字符串生成一个“神经网络+特征融合模型”的具体实例，即一个车道线检测网络的具体结构；之后，通过训练集中的训练数据对此实例进行训练，得到训练后的车道线检测网络，并在训练完成之后对该训练后的车道线检测网络做性能评估，得到网络性能，类似地，针对每个采样点都进行上述操作，从而得到每个采样点对应的车道线检测网络的性能，当采样达到预设次数，如，400次，则对应获取到的车道线检测网络的网络结构也有400种，并且这400个车道线检测网络的性能也经过性能评估得到了，之后，通过本申请事先设置的预设条件，根据该预设条件从这400个车道线检测网络中选择出一个车道线检测网络作为最终的目标车道线检测网络，该目标车道线检测网络就包括有对应的目标神经网络(如，上述训练后的神经网络101)和目标特征融合模型(如，上述训练后的特征融合模型102)。最后，就可通过该目标车道线检测网络中的目标神经网络对待检测图像进行特征提取，得到该目标神经网络不同层输出的第一特征图，再将该不同层输出的第一特征图作为目标特征融合模型的输入，由目标特征融合模型输出第二特征图。
需要说明的是，在本申请的一些实施方式中，通过本申请事先设置的预设条件，根据该预设条件从多个采样得到的车道线检测网络中选择出一个车道线检测网络作为最终的目标车道线检测网络的方式具体可以包括但不限于如下方式：1)选择一个车道线检测网络性能在所有采样点对应的各个车道线检测网络性能中最优的作为目标车道线检测网络；2)选择一个车道线检测网络的算力在所有采样点对应的各个车道线检测网络的算力中最大的作为目标车道线检测网络。
为便于理解，下面以一个具体的示例，对上述过程进行详细说明，具体请参阅图14，图14为本申请实施例提供的其中一种通过搜索得到目标车道线检测网络的方式，首先构建“神经网络+特征融合模型”的搜索空间，也就是每一对“神经网络+特征融合模型”对应的编码组合，每一次搜索的过程就包括从搜索空间中采样，采样之后可以获得一个代表“神经网络+特征融合模型”的字符串，即一个编码组合，该编码组合由采样得到的神经网络的第一编码和采样得到的特征融合模型的第二编码组合而成，根据此字符串生成一个“神经网络+特征融合模型”的具体实例(即一个车道线检测网络的具体结构)，其中，图14示意的采样次数为N＝300，N为预设的采样次数。针对每次采样，根据遗传算法(如，随机生成)分别生成神经网络的第一编码和特征融合模型的第二编码构成的编码组合，之后，再对该编码组合进行解码，生成车道线检测网络，再利用训练集中的训练数据对每个采样得到的车道线检测网络进行训练，并将该车道线检测网络的输出(即得到的融合后的特征图)进一步输入至如图8所示的预测头模型102和预测车道线集成模块104，并针对整个车道线检测框架进行训练，最后基于预测车道线来对车道线检测网络的性能进行评估，具体的评估方式可以包括但不限于如下方式：将搜索产生的每对编码组合输入到如图8所示的车道线检测框架，并输入预测头模型103的参数和预测车道线集成模块104的参数，并根据所有参数初始化该车道线检测框架，之后依次输入测试集中的共M张图像，从而获得车道线检测框架的输出，具体地，针对每张测试集中的图像，先获取车道线检测网络中特征融 合模型102输出的融合后的特征图，在根据该融合后的特征图，从预测车道线集成模块104输出集成后的预测车道线，再根据输出的预测车道线与标签进行F1估计，输出F1的估计值，该F1的估计值就作为该车道线检测网络的性能的指标，最后再根据各个车道线检测网络的性能评估的结果调整整个车道线检测框架的参数(如，预测头模型103、预测车道线集成模块104)，在积累了足够的采样点之后，做出采样点所表示的时延和F1的估计值所组成的散点图，取出其帕累托(pareto)前沿上边的点，然后以该点为基础，做随机失活、随机调换等操作(即遗传算法基础操作)，作为新的采样点。总计采样数量达到预设值(如，300个)后，输出pareto最优点作为最终搜索结果，即最终输出的车道线检测网络。
这里需要说明的是，在图14对应的实施方式中，采用的是通过堆叠的ResNet/ResNeXt基础块(block)构建神经网络，由于ResNet/ResNeXt的block具有很多个(如，10个至45个)，这里可根据需要将ResNet/ResNeXt的block划分为多个阶段，不同阶段的最后一个block输出一个第一特征图，图15示意的是将所有block划分为5个阶段，分别为阶段1至阶段5，基本通道分别为48、64、80、96、128，并且每个阶段可以选择包括不同数量的block，基于此可知，神经网络的block的数量不同、划分阶段的方式不同，那么构建得到的神经网络的网络结构也就有很多种。同时为了灵活地分配计算量，还增加通道数的位置由搜索获得，每个神经网络包括的block数量、划分的阶段数量、每个阶段具体包括的block数量、通道数的位置等都可通过预先设置的规则进行编码(即图14所示的第一编码)，针对每个第一编码就对应一个上述所述的神经网络。
还需要说明的是，在上述图14-15所对应的实施例中，针对搜索得到的每一个车道线检测网络都可利用全量的训练集中的训练数据来对搜索得到的车道线检测网络的性能进行评估。但由于神经网络组成的总搜索空间大约有5×10 12种情况，在每次采样之后，需要对初始化的每个车道线检测网络中的参数进行赋值(即训练)，由于完全训练的车道线检测网络才能够代表搜索到神经网络的精度(没有完全训练的模型精度漂移太严重)，而在全量的训练集上进行完全训练又过于消耗时间，因此在本申请的一些实施方式中，在全量训练集上进行采样，之后再在有代表性的样本上进行完全训练。例如，在图14-15所对应的实施例中，图采样获得的训练集可为10万张，测试集为3万张。具体地，首先在多个神经网络以及多个特征融合模型所构成的搜索空间里进行采样，采样后使用训练集中的10万样本对每一个搜索到的车道线检测网络进行训练，训练完成后再在测试集上的3万样本上进行性能评估，并根据评估结果调整整个车道线检测框架的参数。基于此，在搜索得到目标车道线检测网络后，还需要对该目标车道线检测网络进行全量训练(fully train)，其主要目的是为了训练出可以部署在设备上的车道线检测框架而设置的。在全量训练流程中，本申请实施例使用全量的训练集进行训练，并进行性能评估。具体做法可如图16所示，对搜索出来的目标“神经网络+特征融合模型”的编码组合进行解码，得到目标车道线检测网络，并对此目标车道线检测网络进行训练，在训练结束后进行性能评估，并根据性能评估结果调整车整个车道线检测框架的集体参数，若最终的性能评估结果符合性能要求，则将对应的车道线检测框架的文件输出，用于部署。其具体的训练过程与搜索过程中的训练流程类似，此处不予赘述。
综上所述，搜索过程得到的目标“神经网络+特征融合模型”，是为了保证搜索时间能够让人可以忍受，所以采用的是对训练集进行抽样的策略，搜索以及评估的时候是采用的训练集的子集。而全量训练流程则是对搜索出来的目标“神经网络+特征融合模型”，使用全量(如，100万张图像)的训练集进行训练，训练完成后根据评估结果微调车道线检测框架的参数，其主要目的是为了训练出可以部署在设备上的车道线检测框架而设置的。
(2)从多种已有的神经网络组成的第一搜索空间中搜索出一种神经网络，再从本申请构建的多种特征融合模型组成的第二搜索空间中搜索出一种特征融合模型。
在这种方式中，是分别在多种已有的神经网络组成的第一搜索空间以及本申请构建的多种特征融合模型组成的第二搜索空间分别进行搜索、分别训练、分别进行性能评估等，其中，在第一搜索空间搜索神经网络并对搜索出来的每个神经网络的训练、性能评估过程，以及在第二搜索空间搜索特征融合模型并对搜索出来的每个特征融合模型的训练、性能评估过程，均与上述方式(1)在共同组合的搜索空间搜索“神经网络+特征融合模型”的组合架构的方式是类似的，此处不予赘述，具体可参阅上述方式(1)的过程。与方式(1)不同的是，方式(2)是将两个模块分开分别进行搜索、训练、性能评估，之后，再将训练后的目标神经网络与训练后的目标特征融合模型组合成所需的目标车道线检测网络。
1102、将第二特征图划分为多个栅格，并得到多个栅格中第一栅格的第一置信度。
根据上述步骤1001所述，可得到至少一个第二特征图，针对得到的每个第二特征图，可将其划分为多个栅格，若该多个栅格中存在置信度取值大于预设阈值(如，预设阈值为0.6)的栅格，则大于该预设阈值的置信度可称为第一置信度，与该第一置信度对应的栅格可称为第一栅格。
这里需要说明的是，从多个栅格中挑选出第一置信度对应的栅格的目的，是剔除掉那些对车道线的预测没有贡献的多余栅格，以减少计算量。因为车道线是呈细长型分布在待检测图像中，如图17所示，将车道线检测网络输出的第二特征图划分为尺寸为w×h的多个栅格，其中一个栅格可记为{G ij}，第二特征图上的每个栅格G ij的中心点称为anchor点，如果G ij的中心点靠近地面上某条真实车道线，则该G ij负责检测该车道线。由于一条真实的车道线可能跨越多个栅格，因此一条车道线可能对应多个栅格，栅格G ij的置信度S ij反映的是车道线通过该栅格G ij的概率，在训练阶段，由于训练图像上的每条真实车道线都是被标注过的，一条车道线通过栅格G ij的概率只有两种情况：通过，则栅格G ij的置信度S ij取值为1；不通过，则栅格G ij的置信度S ij取值为0。而在检测阶段，提取到的第二特征图上是否存在车道线以及在特定位置存在车道线是以概率的形式存在，因此，一条车道线通过栅格G ij的概率也就是车道线在栅格所在位置存在的概率。在检测阶段的该待检测图像中，真实存在的车道线有3条，只有分布在车道线附近的那些栅格(即图17中标记有黑点的栅格)对车道线的预测才有意义，因此这些记有黑点的栅格就称为第一栅格。
需要说明的是，针对一张待检测图像，如何得到每个栅格对应的置信度取值是已有的方式，此处不予赘述，但需要注意的是，这种方式得到的是每个栅格的初始置信度(一种粗略估计值)，在本申请的一些实施方式中，可以从这些初始置信度中选取取值大于一定阈值的栅格作为第一栅格，对应该第一栅格的置信度作为第一置信度，用于后续对车道线进 行预测。
还需要说明的是，由于初始置信度是粗略估计值，在用于对车道线的预测过程中存在精度不高的问题，因此，在本申请的另一些实施方式中，可以先得到第一栅格的初始置信度，再根据预设方式对每个第一栅格的初始置信度进行调整，以得到每个第一栅格的第一置信度，调整的目的是使得在预测过程中，针对车道线远端的预测能更加准确。具体包括但不限于如下方式：
(1)基于统计方法由初始置信度得到第一置信度。
首先，根据训练集中各个图像内真实车道线远端所在栅格anchor点位置的分布建立统计模型，该统计模型具有至少一个待确定的参数，之后根据训练集中各个图像内真实车道线远端所在栅格anchor点位置的统计结果对统计模型的参数进行估计，得到参数的估值，并且在上述对车道线检测网络进行训练的过程(如，搜索过程中的训练、全量训练等)中对该参数的估值进行调整，得到最终统计模型参数的确定值，再通过确定了参数的确定值的统计模型对第一栅格的anchor点位置进行计算，得到第一栅格的anchor点位置的权重值，最后，对第一栅格的初始置信度与权重值进行运算，得到第一栅格的第一置信度，如图18所示，是对各个第一栅格的初始置信度与各个第一栅格的anchor点位置的权重值进行乘积运算，从而得到各个第一栅格的第一置信度。
为便于理解，下面对上述统计方法进行示意，具体请参阅图19，首先，统计训练集所有车道线远端(即假设将训练集上所有训练图像上的车道线远端位置集中在一张图上，统计所有车道线远端的位置分布，图19中假设所有车道线总数为M)所对应栅格的anchor点在特征图上的分布，然后就该分布进行建模，并根据训练集所有车道线远端统计结果获得分布模型的参数估计，输出参数估计结果。具体地，首先对训练集中车道线远端对应的栅格anchor点的位置进行统计，获得各个anchor点的频率热图如图20左部分所示，其频率热图的横向中央剖面近似图20右部分所示的高斯函数。因此，在本申请的一些实施方式中，上述统计模型可以建模为如公式(1)中的高斯函数：
Figure PCTCN2021092070-appb-000003
其中，α 1至α 4为该统计模型的4个参数，X为车道线远端所在栅格anchor点位置的坐标。
在确定统计模型为该高斯函数后，再根据训练集上各个车道线远端所在栅格anchor点位置的坐标进行统计，估计出上述高斯函数的参数取值，作为图8车道线检测框架中置信度调整模型105的初始化参数。在上述步骤1101对“神经网络+特征融合模型”组成的车道线检测网络的搜索及全量训练过程中，同时需要调整置信度调整模型105所包括的统计模型的参数，并进一步根据调整了参数取值的公式(1)计算每个第一栅格的anchor点位置所占权重值，在这一步中，由于高斯函数的参数已经确定下来，此时X表示的则是每个第一栅格的anchor点位置的坐标X n＝{x 1，x 2}，通过计算f(X n)与每个第一栅格的初始置信度的乘积，作为新的置信度(即第一置信度)。
(2)基于学习方法由初始置信度得到第一置信度。
方式(1)是基于大量的训练集上的车道线远端分布的统计结果进行建模，而在方式(2)中，还可以通过学习的方式，由第一栅格的初始置信度得到第一栅格的第一置信度，具体地，可构建一个神经网络，如，卷积神经网络(convolutional neural networks,CNN)，将训练集中各训练图像内的第一栅格的初始置信度作为该CNN的输入，输出该第一栅格的第一置信度，通过构建合适的损失函数，那么经过训练后的CNN，可输出满足本申请要求的第一栅格的第一置信度。
1103、通过训练后的预测头模型得到与第一栅格对应的第一预测车道线。
在得到各个第一栅格的第一置信度之后，就可进一步通过训练后的预测头模型(如，图8车道线检测框架中的预测头模型103)得到与每个第一栅格对应的一条预测车道线(即第一预测车道线)。具体地，在本申请实施例中，根据第一栅格得到第一预测车道线是通过预测头模型得到的，而预测头模型的具体工作原理如上述图6对应的实施例所述，此处不予赘述。
但需要说明的是，由于输入预测头模型的第一栅格是基于融合后的第二特征图进行划分的，第二特征图具有多层次的特征，该第二特征图中的低层次特征分辨率更高，包含更多位置、细节信息，但是由于经过的卷积更少，其语义性更低，噪声更多；第二特征图中的高层次特征具有更强的语义信息，但是分辨率低，对细节的感知能力较差。因此，如图21所示，通过训练后的预测头模型得到的第一预测车道线也具有不同层次的特征，其中，低层次的感受野，有利于车道线的准确回归，而高层次的感受野，有利于判断车道线是否存在。
1104、当第一栅格有n个，则将n个第一预测车道线分为m组。
由于一条真实的车道线往往会引起多个栅格的响应，因此在预测阶段对应于一条真实车道线的预测结果会有很多，在根据预测头模型获取到n个第一栅格对应的n个第一预测车道线之后，就需要将预测得到的n个第一预测车道线分为m组，其中每组中都包括有至少一条预测车道线。
1105、当m组中的第一组存在q个第一预测车道线，则根据q个第一预测车道线分别对应的q个第一置信度及分别对应的q个第一栅格对q个第一预测车道线进行集成，得到一个第二预测车道线，该第二预测车道线作为对第二特征图内的一个真实车道线的检测结果输出。
若m组中的某一组存在q个第一预测车道线，则根据q个第一预测车道线分别对应的q个第一置信度及分别对应的q个第一栅格对这q个第一预测车道线进行集成，得到一个第二预测车道线，该第二预测车道线作为对第二特征图内的一个真实车道线的检测结果输出。
为便于理解，下面以图22为例进行示意，说明如何将n条第一预测车道线分为m组，并且得到最终的检测结果。请参阅图22，首先，设置车道线间距阈值T，并将队列D置为空，之后，根据获取到的n个第一置信度(这n个第一置信度可记为S 1＝{s 1，s 2，...，s n，})，对n个第一预测车道线(这n个第一预测车道线可记为L＝{l 1，l 2，...，l n，})进行排序，以S 1中第一置信度最大值s i对应的第一预测车道线作为基线，从剩下的n-1个第一预测车道线中 找出线间距小于阈值T的所有其他第一预测车道线，找出的这其他第一预测车道线与该基线就构成一组第一预测车道线L 1，(假设为第1组)，第1组就用于对一条真实的车道线进行预测，假设针对这一组第一预测车道线得到的预测结果为第二预测车道线l z，则将该第二预测车道线l z加入队列D，同时，从L＝{l 1，l 2，...，l n，}中剔除掉第1组中的所有车道线L 1，接着进一步剔除掉第1组的L是否为空，若L不为空，则继续从L中剩下的第一预测车道线中选择出另一个S 1中第一置信度最大值s j对应的第一预测车道线作为基线，按照上述类似的步骤直至找出所有m组，针对每一组都预测一条第二预测车道线；若L为空，则说明当前组是第m组，此时预测得到的第二预测车道线也有m条，这m条第二预测车道线都按照上述所述的方式输入到队列D中了，只需将这m条第二预测车道线作为最终预测结果输出即可。
需要说明的是，在本申请实施例中，m组中的任意一组都必须包括至少一条第一预测车道线，若m组中存在某一组或多组包括的第一预测车道线大于2条的情况时，则需对该组中的第一预测车道线进行集成，以得到一条第二预测车道线。这里根据每组中第一预测车道线数量的不同分别进行说明：
(1)若m组中的第一组只存在一条第一预测车道线。
当m组中的第一组只存在一个第一预测车道线，则直接将该第一组中的第一预测车道线作为对所述第二特征图内的一个真实车道线的检测结果进行输出。
(2)若m组中的第一组存在q个第一预测车道线，q≥2。
若m组中的第一组存在q个第一预测车道线(q≥2)，则可将这q个第一预测车道线在第二特征图内占据的区域划分为多个子区域，每个子区域中包括q个第一栅格中的至少一个第一栅格的anchor点位置，若这多个子区域中的第一子区域包括至少两个第一栅格的anchor点位置，则可通过但不限于如下方式进行处理：
处理方式1，根据这至少两个第一栅格分别对应的第一置信度从该至少两个第一栅格选取一个第一栅格作为第二栅格，例如，从该至少两个第一栅格选取第一置信度取值最大的第一栅格作为第二栅格，并将第二栅格对应的第一预测车道线位于第一子区域的部分作为第二预测车道线的第一部分；若这多个子区域中的第二子区域仅包括一个第一栅格的anchor点位置，则将该第二子区域包括的第一栅格对应的第一预测车道线位于该第二子区域的部分作为第二预测车道线的第二部分；最后，将第一部分和第二部分集成，得到最终输出的第二预测车道线。
为便于理解，下面以图23为例，对上述如何根据同一组内的多个第一预测车道线集成得到一个第二预测车道线的方法进行示意。假设第一组中存在6条第一预测车道线，那么根据这6条第一预测车道线在特征图内占据的区域划分为多个子区域(假设划分为子区域1、子区域2和子区域3)，其中，子区域1、子区域2和子区域3分别包括第一栅格的anchor点位置及数量如图23所示(图23中的黑点为第一栅格的anchor点位置)，从图23中可以看出，子区域1中仅存在1个第一栅格的anchor点位置，此时就将该第一栅格(其第一置信度为0.81)对应的第一预测车道线位于该子区域1的部分作为构成第二预测车道线的部分1；子区域2中存在3个第一栅格的anchor点位置，并且这3个第一栅格对应的第一置 信度分别为0.76、0.83和0.90，则取置信度为0.90的第一栅格对应的第一预测车道线位于该子区域2的部分作为构成第二预测车道线的部分2；类似地，子区域3中存在2个第一栅格的anchor点位置，并且这2个第一栅格对应的第一置信度分别为0.75和0.92，则取置信度为0.92的第一栅格对应的第一预测车道线位于该子区域3的部分作为构成第二预测车道线的部分3。最后，将部分1、部分2、部分3依次进行集成，得到最终输出的一条第二预测车道线。
需说明的是，在本申请的一些实施方式中，依然以图23为例，对上述如何根据同一组内的多个第一预测车道线集成得到一个第二预测车道线的方法的另一种示例进行示意，假设第一组中存在6条第一预测车道线，这6条第一预测车道线分别对应的第一置信度为0.81、0.76、0.83、0.90和0.92，那么选择第一置信度取值最大的0.92所对应的第一预测车道线作为基线，在每个子区域中按照第一置信度取值从低到高对位于该子区域的第一预测车道线部分进行替换，直至对所有的子区域完成所述操作，最终各个子区域留下第一置信度取值最大的部分，这些部分就构成最终输出的第二预测车道线。这种方式是在每组中，将低分预测车道线中具有良好表现力的点与高分预测车道线中表现不良的点进行交换，从而利用低分预测车道线中具有良好表现力的点不断修正高分预测车道线预测不良的部分，从而最终获得一条更加贴近真实车道线的第二预测车道线。
处理方式2，针对该至少两个第一栅格分别对应的第一预测车道线位于第一子区域的部分进行加权运算，得到的运算结果作为第二预测车道线的第一部分。为便于理解，下面以图24为例进行说明，假设图24中所示的第一子区域包括有两条第一预测车道线A、B分别所在栅格anchor点位置(图24中未示出)，那么该预测车道线A和预测车道线B均有一部分落在该第一子区域，具体如图24所示，由于每条预测车道线都是由各个坐标点连接而成，因此，在本申请实施方式中，可通过对落入第一子区域的预测车道线A、B的坐标点进行加权运算，再基于得到的各个坐标点的加权结果(即新的一系列坐标点)，得到如图24所示的第二预测车道线W落在第一子区域的部分(即虚线部分)，具体地，以一个坐标点为例进行示意，说明如何进行加权运算，由于已经假设预测车道线A的第一置信度为0.8，预测车道线B的第一置信度为0.9，基于各个第一置信度取值的不同，在加权运算中所占的权重也不同，具体的权重比值可根据预设规则自行设定，此处不做限定，这里假设预测车道线A所占权重为0.7，预测车道线B所占权重为0.3，并且假设预测车道线A中的一个坐标点取值为0.8，预测车道线B中的一个坐标点取值为0.1，那么经过如下加权运算：0.7*0.8+0.3*0.1＝0.59，该0.59就为构成第二预测车道线W的其中一个坐标点。
需要说明的是，上述图23、图24示意的区域、子区域等均是以矩形区域进行示意，在本申请的一些实施方式中，对各区域或子区域的形状并不作限定，例如，可以是圆形区域、椭圆形区域、梯形区域，甚至是不规则区域，只要是能够实现将第一预测车道线进行划分的功能的区域均可，此处不做限定。
在本申请上述实施方式中，首先，通过特征融合的过程，输出的第二特征图融合了神经网络不同层输出的第一特征图的多个层次的特征，该第二特征图既有低层次的感受野的相关特征，使得利于车道线的准确回归，又有高层次的感受野的相关特征，使得利于判断 车道线是否存在；其次，对于输出的预测车道线集合进行分组，在每一组中由于感受野的限制，每一条预测车道线都有自己的最优预测区间，本申请实施例通过选取每条线的最优预测区间做车道线的集成，最终结合不同预测车道线的优势，形成最终输出的预测车道线。
需要说明的是，部署有上述所述训练后的车道线检测框架的检测设备具体可以是各种轮式移动设备，例如，可以是轮式施工设备、自动驾驶车辆、辅助驾驶车辆等，只要是具备轮式可移动的设备就都称为本申请所述的轮式移动设备，自动驾驶车辆可以为轿车、卡车、摩托车、公共汽车、船、飞机、直升飞机、割草机、娱乐车、游乐场车辆、施工设备、电车、高尔夫球车、火车、和手推车等，本申请实施例不做特别的限定。
为了对本申请实施例所带来的有益效果有更为直观的认识，以下对本申请实施例所带来的技术效果作进一步的对比，请参阅表1和表2，其中，表1为CULane数据集上本申请实施例所提供的车道线检测框架不同分项对精度的影响，CULane数据集标注了133235张图片，分成88880张作为训练集，9675作为验证集，34680做测试集。数据集包含黑夜、白天、眩光、阴影等不同情况，采集地包含城市、农村、高速等不同应用场景，由于其比较有代表性，现已成为车道线检测领域公认的数据集。在CULane数据集上，本申请实施例所提供的车道线检测框架的相关算法对结果有明显提升，其中nms-C4是原始Line-NMS算法在单层特征图上的评估结果，ensemble-C4是本申请实施例提供的算法在单层特征图上的评估结果，ensemble-C3C4C5本申请实施例提供的算法在多层特征图上的评估结果，ensemble-C3C4C5-masked是本申请实施例提供的算法在多层次特征图上的评估结果，由表可知，本发明中每一分项对F1都有将近0.1左右的贡献。
表1 CULane测试集上本算法不同分项对精度的影响
Figure PCTCN2021092070-appb-000004
表为本申请实施例的算法与SOTA算法的比较，CurveLane-S，CurveLane-M，CurveLane-L为本申请实施例的算法分别在小运算量约束、中运算量约束、大运算量约束下的搜索结果，有如下表2可知：CurveLane-S在9GFLOPS时性能几乎与SCNN持平，CurveLane-L在不考虑运算量的情况下几乎在每一分项都具有领先优势。
表2 CULane测试集上本算法与SOTA算法的比较结果
Figure PCTCN2021092070-appb-000005
由于智慧城市、智能驾驶等领域中都可以用到本申请实施例中的训练好的车道线检测框架来对车道线进行检测，下面将对多个落地到产品的多个应用场景进行介绍。
(1)自动驾驶
在自动驾驶中，车道线检测，对于自动驾驶车辆的准确定位，以及对驾驶决策的影响是相当重要的。车道线作为路面最主要的指示信息之一，它可以有效地引导智能车辆在约束的道路区域内行驶，实时地检测出路面的车道线是智能车辆辅助驾驶系统中的重要环节，该技术有利于协助路径规划、进行道路偏移预警等功能，并且可为精确导航提供参照。本申请实施例训练后的车道线检测框架就可部署在如图10所述的自动驾驶车辆上，用于实时对搭载的摄像机采集到的路面相关的待检测图像进行车道线检测。
(2)增强现实(Augmented Reality，AR)导航
随着汽车数量的与日俱增，人的活动范围也因之拓展，单纯依靠经验驾驶到达目的地的情况越来越少，人们的出行越来越依赖导航。传统的导航根据GPS信号将车辆位置投射到地图上，导航软件根据设定的目的地规划行驶路径，通过屏幕显示和语音播报的形式指引用户行驶至目的地。这样的展示方式需要司机在驾驶过程中将地图指引信息和语音播报信息与当前自车所处的真实世界连接起来，才能理解引导信息的具体含义。如果在驾驶时，遇到岔路却没有听清楚导航播报就很容易错过变道时机。
在AR导航中(如图25)，综合导航信息以及车道线检测结果，能够使系统根据车辆所在位置和车道线检测结果，推算出车辆的精准位置和变道的最佳时机，优化了引导信息展现形式，降低了用户的接收和理解成本，所见即所得，用户只用根据导航信息行动即可。准确的车道线检测信息与合理的导航信息相结合，可以有效地引导车辆在约束的道路区域内行驶，提高现阶段车载导航软件的使用体验。
应理解，上文介绍的自动驾驶和AR导航只是本申请实施例构建的车道线检测框架及车道线检测方法所应用的两个具体场景，本申请实施例训练后的车道线检测框架在应用时并不限于上述场景，其能够应用到任何需要进行车道线识别、车道线检测的场景中。
在上述实施例的基础上，为了更好的实施本申请实施例的上述方案，下面还提供用于实施上述方案的相关设备。具体参阅图26，图26为本申请实施例提供的一种检测设备的示意图，该检测设备2600具体可以包括：融合模块2601、第一获取模块2602、第二获取模块2603、分组模块2604和集成模块2605，其中，该融合模块2601，用于对训练后的神经网络不同层输出的第一特征图进行特征融合，得到第二特征图，所述训练后的神经网络的输入为待检测图像；第一获取模块2602，用于将所述第二特征图划分为多个栅格，并得到所述多个栅格中第一栅格的第一置信度，所述第一置信度用于表示所述第二特征图内真实车道线穿过所述第一栅格的第一概率且所述第一概率超过第一预设阈值；第二获取模块2603，用于通过训练后的预测头模型得到与所述第一栅格对应的第一预测车道线；分组模块2604，用于当所述第一栅格有n个，则将n个第一预测车道线分为m组，m≤n且n≥2；集成模块2605，用于当所述m组中的第一组存在q个第一预测车道线，则根据所述q个第一预测车道线分别对应的q个第一置信度及分别对应的q个第一栅格对所述q个第一预测车道线进行集成，得到一个第二预测车道线，所述第二预测车道线作为对所述第二特征图内的一个真实车道线的检测结果输出，q≥2。
在本申请上述实施方式中，首先通过对神经网络不同层提取到的特征进行融合，得到融合后的第二特征图，经过融合处理的第二特征图就具有了多个层次的特征，既有低层次感受野的相关特征，以利于车道线的准确回归，又有高层次感受野的相关特征，以利于判断车道线是否存在；其次，对于输出的预测车道线集合进行分组，在每一组中由于感受野的限制，每一条预测车道线都有自己的最优预测区间，本申请实施例通过选取每条预测车道线的最优预测区间做车道线的集成，最终结合不同预测车道线的优势，形成最终输出的预测车道线，从而提高了车道线检测的精度。
在一种可能的设计中，该集成模块2605，具体用于：首先，将所述q个第一预测车道线在所述第二特征图内占据的区域划分为多个子区域，其中，每个子区域中包括所述q个第一栅格中的至少一个第一栅格的中心点位置；当所述多个子区域中的第一子区域包括至少两个第一栅格的中心点位置，则根据所述至少两个第一栅格分别对应的第一置信度从所述至少两个第一栅格选取一个第一栅格作为第二栅格，并将所述第二栅格对应的第一预测车道线位于所述第一子区域的部分作为所述第二预测车道线的第一部分，或，对该至少两个第一栅格分别对应的第一预测车道线位于所述第一子区域的部分进行加权运算，得到的运算结果作为第二预测车道线的第一部分；当所述多个子区域中的第二子区域仅包括一个第一栅格的中心点位置，将所述第二子区域包括的第一栅格对应的第一预测车道线位于所述第二子区域的部分作为所述第二预测车道线的第二部分；将所述第一部分和所述第二部分进行集成，得到所述第二预测车道线。
在本申请上述实施方式中，具体阐述了如何对一组内的多条第一预测车道线进行集成，即分别取多条第一预测车道线的优势点进行集成，从而形成最终输出的检测结果，即一条第二预测车道线，最终输出的第二预测车道线由于融合了多条第一预测车道线的优点，提高了检测精度。
在一种可能的设计中，该集成模块2605，具体还用于：根据所述至少两个第一栅格分 别对应的第一置信度，从所述至少两个第一栅格选取第一置信度取值最大的第一栅格作为第二栅格。
在本申请上述实施方式中，阐述了一种确定第二栅格的方式，即选取第一置信度取值最大的那个，灵活方便，更适用于实际的使用场景。
在一种可能的设计中，该集成模块2605，具体还用于：当所述m组中的第一组只存在一个第一预测车道线，则将所述第一组中的第一预测车道线作为对所述第二特征图内的一个真实车道线的检测结果输出。
在本申请上述实施方式中，说明当一组中只有一条第一预测车道线时，这时候就不需要集成，直接就将该第一预测车道线作为预测结果输出。针对组里第一预测车道线数量不同，分别对应不同的处理方式，具备灵活性，并且也符合实际应用。
在一种可能的设计中，该融合模块2601，具体用于：首先，构建搜索空间，所述搜索空间由多个神经网络的编码及多个特征融合模型的编码组合而成，所述多个特征融合模型根据多种预设规则构建得到；之后，在所述搜索空间进行采样，得到第一采样点，所述第一采样点为第一神经网络的编码及第一特征融合模型的编码构成的第一编码组合；再根据所述第一编码组合生成第一车道线检测网络，并通过训练集对所述第一车道线检测网络进行训练，并根据训练后的第一车道线检测网络进行性能评估，得到所述训练后的第一车道线检测网络性能；当采样达到预设值，且所述训练后的第一车道线检测网络性能满足预设条件，则通过所述训练后的第一车道线检测网络中的第一神经网络对待检测图像进行特征提取，得到所述第一神经网络不同层输出的第一特征图；最后，将所述第一神经网络不同层输出的第一特征图作为所述训练后的第一车道线检测网络中第一特征融合模型的输入，输出第二特征图。
在本申请上述实施方式中，通过构建神经网络和特征融合模型的搜索空间，通过搜索得到符合要求的目标神经网络和目标特征融合模型，这种通过搜索的方式能够从众多的神经网络和特征融合模型中选取到最匹配检测任务的目标神经网络和目标特征融合模型的相关结构，间接的提高了模型的检测性能。
在一种可能的设计中，所述多种预设规则至少包括如下一种：首先，将至少两个高分辨率的第一特征图分别通过至少一个第一卷积核操作，得到至少两个第三特征图；再对所述至少两个第三特征图的分辨率进行处理，得到与低分辨率的第一特征图具有相同分辨率的至少两个第四特征图；最后，将所述至少两个第四特征图与所述第分辨率的第一特征图按预设组合方式进行融合，得到至少一个第二特征图。
在本申请上述实施方式中，由于特征融合模型是本申请实施例通过预设的融合规则(即预设规则)自行构建的，融合规则不同，构建出来的特征融合模型也不同，通过对其中一种融合规则的说明，便于理解本申请所述的融合规则。
在一种可能的设计中，对所述至少两个第三特征图的分辨率进行处理包括如下至少一种方式：对所述至少两个第三特征图进行下采样操作、池化操作或卷积操作。
在本申请上述实施方式中，具体阐述了如何降低分辨率的几种处理方式，具备可选择性。
在一种可能的设计中，训练后的第一车道线检测网络性能满足预设条件包括：训练后的第一车道线检测网络性能在所有采样点对应的各个车道线检测网络性能中最优。
在本申请上述实施方式中，说明满足预设条件的一种方式是性能最优，符合实际应用场景的需求，间接提高了设备整体的性能。
在一种可能的设计中，第一获取模块2602，具体用于：首先将所述第二特征图划分为多个栅格，并得到所述多个栅格中第一栅格的初始置信度，所述初始置信度用于表示所述第二特征图内真实车道线穿过所述第一栅格的初始概率且所述初始概率超过初始预设阈值；最后，通过预设方式对所述初始置信度进行调整，得到所述第一栅格的第一置信度。
在本申请上述实施方式中，说明第一栅格的第一置信度是由第一栅格的初始置信度通过调整后得到的，这是由于初始置信度是粗略估计值，在用于对车道线的预测过程中存在精度不够高的问题，调整的目的是使得在预测过程中，针对车道线远端的预测能更加准确。
在一种可能的设计中，第一获取模块2602，具体还用于：首先，根据训练集中各个图像内真实车道线远端所在栅格中心点位置的分布建立统计模型，所述统计模型具有至少一个待确定的参数；再根据所述训练集中各个图像内真实车道线远端所在栅格中心点位置的统计结果对所述参数进行估计，得到所述参数的估值；之后，在训练过程中对所述参数的估值进行调整，得到所述参数的确定值，并通过确定了所述参数的确定值的所述统计模型对所述第一栅格的中心点位置进行计算，得到所述第一栅格的中心点位置的权重值；最后，对所述第一栅格的初始置信度与所述权重值进行运算，得到所述第一栅格的第一置信度。
在本申请上述实施方式中，提供了一种基于统计方法由初始置信度得到第一置信度的方式，具备可操作性。
在一种可能的设计中，统计模型包括：高斯函数。
在本申请上述实施方式中，由于各个车道线远端所在栅格的中心点位置的频率热图的横向中央剖面近似高斯函数，因此，可将统计模型建模为高斯函数。
接下来介绍本申请实施例提供的一种检测设备，请参阅图27，图27为本申请实施例提供的检测设备的一种结构示意图，检测设备2700具体可以表现为各种终端设备，例如，轮式移动设备上的相关系统(如，图10所示的环境感知系统)，此处不做限定。其中，检测设备2700上可以部署有图8对应实施例中所描述的车道线检测框架，用于实现图11对应实施例中车道线的检测方法的功能。具体的，检测设备2700包括：接收器2701、发射器2702、处理器2703和存储器2704(其中检测设备2700中的处理器2703的数量可以一个或多个，图27中以一个处理器为例)，其中，处理器2703可以包括应用处理器27031和通信处理器27032。在本申请的一些实施例中，接收器2701、发射器2702、处理器2703和存储器2704可通过总线或其它方式连接。
存储器2704可以包括只读存储器和随机存取存储器，并向处理器2703提供指令和数据。存储器2704的一部分还可以包括非易失性随机存取存储器(non-volatile random access memory，NVRAM)。存储器2704存储有处理器和操作指令、可执行模块或者数据结构，或者它们的子集，或者它们的扩展集，其中，操作指令可包括各种操作指令，用于实现各种操作。
处理器2703控制检测设备2700的操作。具体的应用中，检测设备2700的各个组件通过总线系统耦合在一起，其中总线系统除包括数据总线之外，还可以包括电源总线、控制总线和状态信号总线等。但是为了清楚说明起见，在图中将各种总线都称为总线系统。
本申请上述图11对应实施例揭示的方法可以应用于处理器2703中，或者由处理器2703实现。处理器2703可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器2703中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器2703可以是通用处理器、数字信号处理器(digital signal processing，DSP)、微处理器或微控制器，还可进一步包括专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field-programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。该处理器2703可以实现或者执行本申请图12对应的实施例中公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器2704，处理器2703读取存储器2704中的信息，结合其硬件完成上述方法的步骤。
接收器2701可用于接收输入的数字或字符信息，以及产生与检测设备2700的相关设置以及功能控制有关的信号输入。发射器2702可用于通过第一接口输出数字或字符信息；发射器2702还可用于通过第一接口向磁盘组发送指令，以修改磁盘组中的数据；发射器2702还可以包括显示屏等显示设备。
本申请实施例中还提供一种计算机可读存储介质，该计算机可读存储介质中存储有用于进行信号处理的程序，当其在计算机上运行时，使得计算机执行如前述所示实施例描述中执行设备所执行的步骤。
本申请实施例提供的检测设备具体可以为芯片，芯片包括：处理单元和通信单元，所述处理单元例如可以是处理器，所述通信单元例如可以是输入/输出接口、管脚或电路等。该处理单元可执行存储单元存储的计算机执行指令，以使训练设备内的芯片执行上述各实施例描述的车道线检测方法。可选地，所述存储单元为所述芯片内的存储单元，如寄存器、缓存等，所述存储单元还可以是所述无线接入设备端内的位于所述芯片外部的存储单元，如只读存储器(read-only memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(random access memory，RAM)等。
具体的，请参阅图28，图28为本申请实施例提供的芯片的一种结构示意图，所述芯片可以表现为神经网络处理器NPU 200，NPU 200作为协处理器挂载到主CPU(Host CPU)上，由Host CPU分配任务。NPU的核心部分为运算电路2003，通过控制器2004控制运算电路2003提取存储器中的矩阵数据并进行乘法运算。
在一些实现中，运算电路2003内部包括多个处理单元(process pngine,PE)。在一些实现中，运算电路2003是二维脉动阵列。运算电路2003还可以是一维脉动阵列或者能够执行例如乘法和加法这样的数学运算的其它电子线路。在一些实现中，运算电路2003是通用 的矩阵处理器。
举例来说，假设有输入矩阵A，权重矩阵B，输出矩阵C。运算电路2003从权重存储器2002中取矩阵B相应的数据，并缓存在运算电路2003中每一个PE上。运算电路2003从输入存储器2001中取矩阵A数据与矩阵B进行矩阵运算，得到的矩阵的部分结果或最终结果，保存在累加器(accumulator)2008中。
统一存储器2006用于存放输入数据以及输出数据。权重数据直接通过存储单元访问控制器(direct memory access controller，DMAC)2005，DMAC被搬运到权重存储器2002中。输入数据也通过DMAC被搬运到统一存储器2006中。
总线接口单元(ius interface init，BIU)2010，用于AXI总线与DMAC和取指存储器(instruction ietch iuffer，IFB)2009的交互。
总线接口单元2010，还用于取指存储器2009从外部存储器获取指令，还用于存储单元访问控制器2005从外部存储器获取输入矩阵A或者权重矩阵B的原数据。
DMAC主要用于将外部存储器DDR中的输入数据搬运到统一存储器2006或将权重数据搬运到权重存储器2002中或将输入数据数据搬运到输入存储器2001中。
向量计算单元2007包括多个运算处理单元，在需要的情况下，对运算电路2003的输出做进一步处理，如向量乘，向量加，指数运算，对数运算，大小比较等等。主要用于神经网络中非卷积/全连接层网络计算，如Batch Normalization(批归一化)，像素级求和，对特征平面进行上采样等。
在一些实现中，向量计算单元2007能将经处理的输出的向量存储到统一存储器2006。例如，向量计算单元2007可以将线性函数和/或非线性函数应用到运算电路2003的输出，例如对卷积层提取的特征平面进行线性插值，再例如累加值的向量，用以生成激活值。在一些实现中，向量计算单元2007生成归一化的值、像素级求和的值，或二者均有。在一些实现中，处理过的输出的向量能够用作到运算电路2003的激活输入，例如用于在神经网络中的后续层中的使用。
控制器2004连接的取指存储器(instruction fetch buffer)2009，用于存储控制器2004使用的指令；
统一存储器2006，输入存储器2001，权重存储器2002以及取指存储器2009均为On-Chip存储器。外部存储器私有于该NPU硬件架构。
其中，上述任一处提到的处理器，可以是一个通用中央处理器，微处理器，ASIC，或一个或多个用于控制上述第一方面方法的程序执行的集成电路。
另外需说明的是，以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。另外，本申请提供的装置实施例附图中，模块之间的连接关系表示它们之间具有通信连接，具体可以实现为一条或多条通信总线或信号线。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到本申请可借助软 件加必需的通用硬件的方式来实现，当然也可以通过专用硬件包括专用集成电路、专用CPU、专用存储器、专用元器件等来实现。一般情况下，凡由计算机程序完成的功能都可以很容易地用相应的硬件来实现，而且，用来实现同一功能的具体硬件结构也可以是多种多样的，例如模拟电路、数字电路或专用电路等。但是，对本申请而言更多情况下软件程序实现是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在可读取的存储介质中，如计算机的软盘、U盘、移动硬盘、ROM、RAM、磁碟或者光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，训练设备，或者网络设备等)执行本申请各个实施例所述的方法。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。
所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、训练设备或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、训练设备或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存储的任何可用介质或者是包含一个或多个可用介质集成的训练设备、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘(solid state disk，SSD))等。

Claims (28)

  1. 一种车道线的检测方法，其特征在于，包括：
    对训练后的神经网络不同层输出的第一特征图进行特征融合，得到第二特征图，所述训练后的神经网络的输入为待检测图像；
    将所述第二特征图划分为多个栅格，并得到所述多个栅格中第一栅格的第一置信度，所述第一置信度用于表示所述第二特征图内真实车道线穿过所述第一栅格的第一概率且所述第一概率超过第一预设阈值；
    通过训练后的预测头模型得到与所述第一栅格对应的第一预测车道线；
    当所述第一栅格有n个，则将n个第一预测车道线分为m组，m≤n且n≥2；
    当所述m组中的第一组存在q个第一预测车道线，则根据所述q个第一预测车道线分别对应的q个第一置信度及分别对应的q个第一栅格对所述q个第一预测车道线进行集成，得到一个第二预测车道线，所述第二预测车道线作为对所述第二特征图内的一个真实车道线的检测结果输出，q≥2。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述q个第一预测车道线分别对应的q个第一置信度及分别对应的q个第一栅格对所述q个第一预测车道线进行集成，得到一个第二预测车道线包括：
    将所述q个第一预测车道线在所述第二特征图内占据的区域划分为多个子区域，每个子区域中包括所述q个第一栅格中的至少一个第一栅格的中心点位置；
    当所述多个子区域中的第一子区域包括至少两个第一栅格的中心点位置，则根据所述至少两个第一栅格分别对应的第一置信度从所述至少两个第一栅格选取一个第一栅格作为第二栅格，并将所述第二栅格对应的第一预测车道线位于所述第一子区域的部分作为所述第二预测车道线的第一部分，或，对所述至少两个第一栅格分别对应的第一预测车道线位于所述第一子区域的部分进行加权运算，得到的运算结果作为所述第一部分；
    当所述多个子区域中的第二子区域仅包括一个第一栅格的中心点位置，将所述第二子区域包括的第一栅格对应的第一预测车道线位于所述第二子区域的部分作为所述第二预测车道线的第二部分；
    将所述第一部分和所述第二部分进行集成，得到所述第二预测车道线。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述至少两个第一栅格分别对应的第一置信度从所述至少两个第一栅格选取一个第一栅格作为第二栅格包括：
    根据所述至少两个第一栅格分别对应的第一置信度，从所述至少两个第一栅格选取第一置信度取值最大的第一栅格作为第二栅格。
  4. 根据权利要求1-3中任一项所述的方法，其特征在于，所述方法还包括：
    当所述m组中的第一组只存在一个第一预测车道线，则将所述第一组中的第一预测车道线作为对所述第二特征图内的一个真实车道线的检测结果输出。
  5. 根据权利要求1-4中任一项所述的方法，其特征在于，所述对训练后的神经网络不同层输出的第一特征图进行特征融合，得到第二特征图包括：
    构建搜索空间，所述搜索空间由多个神经网络的编码及多个特征融合模型的编码组合 而成，所述多个特征融合模型根据多种预设规则构建得到；
    在所述搜索空间进行采样，得到第一采样点，所述第一采样点为第一神经网络的编码及第一特征融合模型的编码构成的第一编码组合；
    根据所述第一编码组合生成第一车道线检测网络，并通过训练集对所述第一车道线检测网络进行训练；
    根据训练后的第一车道线检测网络进行性能评估，得到所述训练后的第一车道线检测网络性能；
    当采样达到预设值，且所述训练后的第一车道线检测网络性能满足预设条件，则通过所述训练后的第一车道线检测网络中的第一神经网络对待检测图像进行特征提取，得到所述第一神经网络不同层输出的第一特征图；
    将所述第一神经网络不同层输出的第一特征图作为所述训练后的第一车道线检测网络中第一特征融合模型的输入，输出第二特征图。
  6. 根据权利要求5所述的方法，其特征在于，所述多种预设规则至少包括如下一种：
    将至少两个高分辨率的第一特征图分别通过至少一个第一卷积核操作，得到至少两个第三特征图；
    对所述至少两个第三特征图的分辨率进行处理，得到与低分辨率的第一特征图具有相同分辨率的至少两个第四特征图；
    将所述至少两个第四特征图与所述第分辨率的第一特征图按预设组合方式进行融合，得到至少一个第二特征图。
  7. 根据权利要求6所述的方法，其特征在于，所述对所述至少两个第三特征图的分辨率进行处理包括如下至少一种方式：
    对所述至少两个第三特征图进行下采样操作、池化操作或卷积操作。
  8. 根据权利要求5-7中任一项所述的方法，其特征在于，所述训练后的第一车道线检测网络性能满足预设条件包括：
    所述训练后的第一车道线检测网络性能在所有采样点对应的各个车道线检测网络性能中最优。
  9. 根据权利要求1-8中任一项所述的方法，其特征在于，所述将所述第二特征图划分为多个栅格，并得到所述多个栅格中第一栅格的第一置信度包括：
    将所述第二特征图划分为多个栅格，并得到所述多个栅格中第一栅格的初始置信度，所述初始置信度用于表示所述第二特征图内真实车道线穿过所述第一栅格的初始概率且所述初始概率超过初始预设阈值；
    通过预设方式对所述初始置信度进行调整，得到所述第一栅格的第一置信度。
  10. 根据权利要求9所述的方法，其特征在于，所述通过预设方式对所述初始置信度进行调整，得到所述第一栅格的第一置信度包括：
    根据训练集中各个图像内真实车道线远端所在栅格中心点位置的分布建立统计模型，所述统计模型具有至少一个待确定的参数；
    根据所述训练集中各个图像内真实车道线远端所在栅格中心点位置的统计结果对所述 参数进行估计，得到所述参数的估值；
    在训练过程中对所述参数的估值进行调整，得到所述参数的确定值；
    通过确定了所述参数的确定值的所述统计模型对所述第一栅格的中心点位置进行计算，得到所述第一栅格的中心点位置的权重值；
    对所述第一栅格的初始置信度与所述权重值进行运算，得到所述第一栅格的第一置信度。
  11. 根据权利要求10所述的方法，其特征在于，所述统计模型包括：
    高斯函数。
  12. 一种车道线检测框架，其特征在于，包括：神经网络、特征融合模型、置信度调整模型、预测头模型和预测车道线集成模型；
    所述神经网络，用于在训练后，对输入的待检测图像进行特征提取；
    所述特征融合模型，用于对训练后的所述神经网络不同层输出的第一特征图进行特征融合，得到第二特征图；
    所述置信度调整模型，用于将所述第二特征图划分为多个栅格，并得到所述多个栅格中第一栅格的第一置信度，所述第一置信度用于表示所述第二特征图内真实车道线穿过所述第一栅格的第一概率且所述第一概率超过第一预设阈值；
    所述预测头模型，用于在训练后，得到与所述第一栅格对应的第一预测车道线；
    所述预测车道线集成模型，用于当所述第一栅格有n个，将n个第一预测车道线分为m组，m≤n且n≥2；
    所述预测车道线集成模型，还用于当所述m组中的第一组存在q个第一预测车道线，则根据所述q个第一预测车道线分别对应的q个第一置信度及分别对应的q个第一栅格对所述q个第一预测车道线进行集成，得到一个第二预测车道线，所述第二预测车道线作为对所述第二特征图内的一个真实车道线的检测结果输出，q≥2。
  13. 根据权利要求12所述的框架，其特征在于，所述预测车道线集成模型，具体用于：
    将所述q个第一预测车道线在所述第二特征图内占据的区域划分为多个子区域，每个子区域中包括所述q个第一栅格中的至少一个第一栅格的中心点位置；
    当所述多个子区域中的第一子区域包括至少两个第一栅格的中心点位置，则根据所述至少两个第一栅格分别对应的第一置信度从所述至少两个第一栅格选取一个第一栅格作为第二栅格，并将所述第二栅格对应的第一预测车道线位于所述第一子区域的部分作为所述第二预测车道线的第一部分，或，对所述至少两个第一栅格分别对应的第一预测车道线位于所述第一子区域的部分进行加权运算，得到的运算结果作为所述第一部分；
    当所述多个子区域中的第二子区域仅包括一个第一栅格的中心点位置，将所述第二子区域包括的第一栅格对应的第一预测车道线位于所述第二子区域的部分作为所述第二预测车道线的第二部分；
    将所述第一部分和所述第二部分进行集成，得到所述第二预测车道线。
  14. 根据权利要求13所述的框架，其特征在于，所述预测车道线集成模型，具体还用于：
    根据所述至少两个第一栅格分别对应的第一置信度，从所述至少两个第一栅格选取第一置信度取值最大的第一栅格作为第二栅格。
  15. 根据权利要求12-14中任一项所述的框架，其特征在于，所述预测车道线集成模型，还用于：
    当所述m组中的第一组只存在一个第一预测车道线，则将所述第一组中的第一预测车道线作为对所述第二特征图内的一个真实车道线的检测结果输出。
  16. 根据权利要求12-15中任一项所述的框架，其特征在于，所述特征融合模型，具体用于：
    构建搜索空间，所述搜索空间由多个神经网络的编码及多个特征融合模型的编码组合而成，所述多个特征融合模型根据多种预设规则构建得到；
    在所述搜索空间进行采样，得到第一采样点，所述第一采样点为第一神经网络的编码及第一特征融合模型的编码构成的第一编码组合；
    根据所述第一编码组合生成第一车道线检测网络，并通过训练集对所述第一车道线检测网络进行训练；
    根据训练后的第一车道线检测网络进行性能评估，得到所述训练后的第一车道线检测网络性能；
    当采样达到预设值，且所述训练后的第一车道线检测网络性能满足预设条件，则通过所述训练后的第一车道线检测网络中的第一神经网络对待检测图像进行特征提取，得到所述第一神经网络不同层输出的第一特征图；
    将所述第一神经网络不同层输出的第一特征图作为所述训练后的第一车道线检测网络中第一特征融合模型的输入，输出第二特征图。
  17. 根据权利要求16所述的框架，其特征在于，所述特征融合模型，具体还用于：
    将至少两个高分辨率的第一特征图分别通过至少一个第一卷积核操作，得到至少两个第三特征图；
    对所述至少两个第三特征图的分辨率进行处理，得到与低分辨率的第一特征图具有相同分辨率的至少两个第四特征图；
    将所述至少两个第四特征图与所述第分辨率的第一特征图按预设组合方式进行融合，得到至少一个第二特征图。
  18. 根据权利要求17所述的框架，其特征在于，所述特征融合模型，具体还用于：
    对所述至少两个第三特征图进行下采样操作、池化操作或卷积操作。
  19. 根据权利要求16-18中任一项所述的框架，其特征在于，所述训练后的第一车道线检测网络性能满足预设条件包括：
    所述训练后的第一车道线检测网络性能在所有采样点对应的各个车道线检测网络性能中最优。
  20. 根据权利要求12-19中任一项所述的框架，其特征在于，所述置信度调整模型，具体用于：
    将所述第二特征图划分为多个栅格，并得到所述多个栅格中第一栅格的初始置信度， 所述初始置信度用于表示所述第二特征图内真实车道线穿过所述第一栅格的初始概率且所述初始概率超过初始预设阈值；
    通过预设方式对所述初始置信度进行调整，得到所述第一栅格的第一置信度。
  21. 根据权利要求20所述的框架，其特征在于，所述置信度调整模型，具体还用于：
    根据训练集中各个图像内真实车道线远端所在栅格中心点位置的分布建立统计模型，所述统计模型具有至少一个待确定的参数；
    根据所述训练集中各个图像内真实车道线远端所在栅格中心点位置的统计结果对所述参数进行估计，得到所述参数的估值；
    在训练过程中对所述参数的估值进行调整，得到所述参数的确定值；
    通过确定了所述参数的确定值的所述统计模型对所述第一栅格的中心点位置进行计算，得到所述第一栅格的中心点位置的权重值；
    对所述第一栅格的初始置信度与所述权重值进行运算，得到所述第一栅格的第一置信度。
  22. 根据权利要求20所述的框架，其特征在于，所述统计模型，包括：
    高斯函数。
  23. 根据权利要求12-22中任一项所述的框架，其特征在于，训练后的所述框架部署在检测设备上。
  24. 根据权利要求23所述的框架，其特征在于，所述检测设备包括：
    轮式移动设备。
  25. 一种检测设备，其特征在于，所述检测设备具有实现所述权利要求1-11中任一项方法的功能，所述功能可以通过硬件实现，或，通过硬件执行相应的软件实现，所述硬件所述软件包括一个或多个与所述功能相对应的模块。
  26. 一种检测设备，包括处理器和存储器，所述处理器与所述存储器耦合，其特征在于，
    所述存储器，用于存储程序；
    所述处理器，用于执行所述存储器中的程序，使得所述检测设备执行如权利要求1-11中任一项所述的方法。
  27. 一种计算机可读存储介质，包括程序，当其在计算机上运行时，使得计算机执行如权利要求1-11中任一项所述的方法。
  28. 一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机执行如权利要求1-11中任一项所述的方法。
PCT/CN2021/092070 2020-06-12 2021-05-07 一种车道线的检测方法及相关设备 WO2021249071A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP21821650.5A EP4152204A4 (en) 2020-06-12 2021-05-07 LANE LINE DETECTION METHOD AND ASSOCIATED APPARATUS
US18/064,016 US20230144209A1 (en) 2020-06-12 2022-12-09 Lane line detection method and related device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010538270.3A CN111860155B (zh) 2020-06-12 2020-06-12 一种车道线的检测方法及相关设备
CN202010538270.3 2020-06-12

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/064,016 Continuation US20230144209A1 (en) 2020-06-12 2022-12-09 Lane line detection method and related device

Publications (1)

Publication Number Publication Date
WO2021249071A1 true WO2021249071A1 (zh) 2021-12-16

Family

ID=72986818

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/092070 WO2021249071A1 (zh) 2020-06-12 2021-05-07 一种车道线的检测方法及相关设备

Country Status (4)

Country Link
US (1) US20230144209A1 (zh)
EP (1) EP4152204A4 (zh)
CN (1) CN111860155B (zh)
WO (1) WO2021249071A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115147812A (zh) * 2022-07-05 2022-10-04 小米汽车科技有限公司 车道线检测方法、装置、车辆和存储介质

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11273836B2 (en) * 2017-12-18 2022-03-15 Plusai, Inc. Method and system for human-like driving lane planning in autonomous driving vehicles
CN111860155B (zh) * 2020-06-12 2022-04-29 华为技术有限公司 一种车道线的检测方法及相关设备
CN112373474B (zh) * 2020-11-23 2022-05-17 重庆长安汽车股份有限公司 车道线融合及横向控制方法、系统、车辆及存储介质
CN112541900B (zh) * 2020-12-15 2024-01-02 平安科技（深圳）有限公司 基于卷积神经网络的检测方法、装置、计算机设备及存储介质
JP2022103693A (ja) * 2020-12-28 2022-07-08 京セラドキュメントソリューションズ株式会社 画像認識方法、画像認識装置、および画像認識プログラム
US20220219731A1 (en) * 2021-01-14 2022-07-14 Cavh Llc Intelligent information conversion for automatic driving
US12008821B2 (en) * 2021-05-07 2024-06-11 Google Llc Machine-learned models for unsupervised image transformation and retrieval
US11845429B2 (en) * 2021-09-30 2023-12-19 GM Global Technology Operations LLC Localizing and updating a map using interpolated lane edge data
US11987251B2 (en) 2021-11-15 2024-05-21 GM Global Technology Operations LLC Adaptive rationalizer for vehicle perception systems toward robust automated driving control
CN114185352B (zh) * 2021-12-08 2024-01-19 东风悦享科技有限公司 一种高精度高实时的自动驾驶局部路径规划方法
CN114494158A (zh) * 2022-01-07 2022-05-13 华为技术有限公司 一种图像处理方法、一种车道线检测方法及相关设备
CN114475630B (zh) * 2022-01-19 2024-02-13 上汽通用五菱汽车股份有限公司 控车协同决策方法、装置、车辆及计算机可读存储介质
CN115071756A (zh) * 2022-06-17 2022-09-20 合众新能源汽车有限公司 一种车道线的确定方法及装置
CN116453121B (zh) * 2023-06-13 2023-12-22 合肥市正茂科技有限公司 一种车道线识别模型的训练方法及装置
CN117671647B (zh) * 2024-01-31 2024-04-26 无锡车联天下信息技术有限公司 一种多任务道路场景感知方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107045629A (zh) * 2017-04-19 2017-08-15 南京理工大学 一种多车道线检测方法
US20190035101A1 (en) * 2017-07-27 2019-01-31 Here Global B.V. Method, apparatus, and system for real-time object detection using a cursor recurrent neural network
CN109766878A (zh) * 2019-04-11 2019-05-17 深兰人工智能芯片研究院（江苏）有限公司 一种车道线检测的方法和设备
CN111860155A (zh) * 2020-06-12 2020-10-30 华为技术有限公司 一种车道线的检测方法及相关设备

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA3067177A1 (en) * 2015-02-10 2016-08-18 Mobileye Vision Technologies Ltd. Sparse map for autonomous vehicle navigation
CN111209780A (zh) * 2018-11-21 2020-05-29 北京市商汤科技开发有限公司 车道线属性检测方法、装置、电子设备及可读存储介质
US10325371B1 (en) * 2019-01-22 2019-06-18 StradVision, Inc. Method and device for segmenting image to be used for surveillance using weighted convolution filters for respective grid cells by converting modes according to classes of areas to satisfy level 4 of autonomous vehicle, and testing method and testing device using the same
CN110263713B (zh) * 2019-06-20 2021-08-10 百度在线网络技术（北京）有限公司 车道线检测方法、装置、电子设备及存储介质
CN111144330B (zh) * 2019-12-29 2022-07-08 浪潮(北京)电子信息产业有限公司 一种基于深度学习的车道线检测方法、装置以及设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107045629A (zh) * 2017-04-19 2017-08-15 南京理工大学 一种多车道线检测方法
US20190035101A1 (en) * 2017-07-27 2019-01-31 Here Global B.V. Method, apparatus, and system for real-time object detection using a cursor recurrent neural network
CN109766878A (zh) * 2019-04-11 2019-05-17 深兰人工智能芯片研究院（江苏）有限公司 一种车道线检测的方法和设备
CN111860155A (zh) * 2020-06-12 2020-10-30 华为技术有限公司 一种车道线的检测方法及相关设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
HANG XU; SHAOJU WANG; XINYUE CAI; WEI ZHANG; XIAODAN LIANG; ZHENGUO LI: "CurveLane-NAS: Unifying Lane-Sensitive Architecture Search and Adaptive Point Blending", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 23 July 2020 (2020-07-23), 201 Olin Library Cornell University Ithaca, NY 14853 , XP081726411 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115147812A (zh) * 2022-07-05 2022-10-04 小米汽车科技有限公司 车道线检测方法、装置、车辆和存储介质

Also Published As

Publication number Publication date
EP4152204A1 (en) 2023-03-22
CN111860155B (zh) 2022-04-29
US20230144209A1 (en) 2023-05-11
CN111860155A (zh) 2020-10-30
EP4152204A4 (en) 2023-11-01

Similar Documents

Publication Publication Date Title
WO2021249071A1 (zh) 一种车道线的检测方法及相关设备
Chen et al. Pointlanenet: Efficient end-to-end cnns for accurate real-time lane detection
JP7430277B2 (ja) 障害物検出方法及び装置、コンピュータデバイス、並びにコンピュータプログラム
US10579058B2 (en) Apparatus and method for generating training data to train neural network determining information associated with road included in image
US20220011122A1 (en) Trajectory prediction method and device
US11768292B2 (en) Three-dimensional object detection
JP7239703B2 (ja) 領域外コンテキストを用いたオブジェクト分類
US11940803B2 (en) Method, apparatus and computer storage medium for training trajectory planning model
CN112740268B (zh) 目标检测方法和装置
JP2021515178A (ja) 自動運転車両においてｒｎｎとｌｓｔｍを用いて時間平滑化を行うｌｉｄａｒ測位
CN113095152B (zh) 一种基于回归的车道线检测方法及系统
CN114882457A (zh) 一种模型的训练方法、车道线的检测方法及设备
CN111931683A (zh) 图像识别方法、装置及计算机可读存储介质
CN113011338A (zh) 一种车道线检测方法及系统
CN114120270A (zh) 一种基于注意力和采样学习的点云目标检测方法
CN115546781A (zh) 一种点云数据的聚类方法以及装置
WO2024008086A1 (zh) 轨迹预测方法及其装置、介质、程序产品和电子设备
US20230196749A1 (en) Training Neural Networks for Object Detection
CN115115084A (zh) 使用占用流场预测代理在环境中的未来移动
Priya et al. Vehicle Detection in Autonomous Vehicles Using Computer Vision Check for updates
Singh et al. Improved YOLOv5l for vehicle detection: an application to estimating traffic density and identifying over speeding vehicles on highway scenes
CN113963027B (zh) 不确定性检测模型的训练、不确定性的检测方法及装置
WO2024093321A1 (zh) 车辆的位置获取方法、模型的训练方法以及相关设备
US12026954B2 (en) Static occupancy tracking
US20220237402A1 (en) Static occupancy tracking

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21821650

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021821650

Country of ref document: EP

Effective date: 20221215

NENP Non-entry into the national phase

Ref country code: DE