CN112199999A - Road detection method, road detection device, storage medium and electronic equipment - Google Patents
Road detection method, road detection device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN112199999A CN112199999A CN202010943653.9A CN202010943653A CN112199999A CN 112199999 A CN112199999 A CN 112199999A CN 202010943653 A CN202010943653 A CN 202010943653A CN 112199999 A CN112199999 A CN 112199999A
- Authority
- CN
- China
- Prior art keywords
- information
- road
- road information
- vanishing point
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 101
- 238000013528 artificial neural network Methods 0.000 claims abstract description 50
- 238000000034 method Methods 0.000 claims description 40
- 238000000605 extraction Methods 0.000 claims description 22
- 238000004590 computer program Methods 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 11
- 238000001914 filtration Methods 0.000 claims description 9
- 238000005516 engineering process Methods 0.000 abstract description 3
- 238000013473 artificial intelligence Methods 0.000 abstract 1
- 230000006870 function Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000012549 training Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000011176 pooling Methods 0.000 description 3
- 230000001629 suppression Effects 0.000 description 3
- 206010039203 Road traffic accident Diseases 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
Abstract
The application provides a road detection method, a road detection device, a storage medium and electronic equipment, belongs to the technical field of computers, and relates to artificial intelligence and computer vision technologies. According to the road detection method, multiple groups of road information can be detected from the image through the detection of the neural network, and more accurate road information can be obtained. And each set of obtained road information comprises vanishing point coordinates, vanishing point confidence coefficient and road curve information, target road information is selected from the sets of road information according to the vanishing point confidence coefficient in each set of road information, the road information output due to error detection can be eliminated, and the road in the image is determined according to the curve information in the target road information with correct detection, so that the accuracy of road detection can be further improved.
Description
Technical Field
The embodiment of the application relates to the field of data processing, in particular to a road detection method, a road detection device, a storage medium and electronic equipment.
Background
With the continuous development of intelligent transportation and automatic driving technologies, the road detection function becomes more obvious. The method and the device have the advantages that the road edges and the lane lines of one road are detected, curve information of the road edges and the lane lines is obtained, early warning information can be provided for vehicles deviating from the road or the lane in the driving process of the vehicles, support is provided for advanced auxiliary driving, meanwhile, traffic violation judgment in intelligent traffic can be helped, urban traffic accidents are reduced, and the traffic is kept smooth.
At present, the detection of the road edge and the lane line mainly adopts an image identification method, firstly, the edge of an image is detected by utilizing the characteristics of color texture and the like, and then the curve information of the road edge and the lane line is obtained by using methods of Hough transform or curve equation fitting and the like on edge points.
The method is easily influenced by environmental factors, and the accuracy of the road detection result is not high.
Disclosure of Invention
In order to solve the existing technical problem, embodiments of the present application provide a road detection method, an apparatus, a storage medium, and an electronic device, which can improve the accuracy of road detection.
In order to achieve the above purpose, the technical solution of the embodiment of the present application is implemented as follows:
in a first aspect, an embodiment of the present application provides a road detection method, where the method includes:
inputting an image containing a road into a trained detection neural network to obtain a plurality of groups of road information in the image; each group of road information comprises vanishing point information and a plurality of pieces of curve information, wherein the vanishing point information comprises vanishing point coordinates and vanishing point confidence coefficients;
and selecting target road information from the multiple groups of road information according to the vanishing point confidence coefficient in each group of road information, and determining the road in the image according to curve information in the target road information.
According to the road detection method provided by the embodiment of the application, multiple groups of road information can be detected from the image through detecting the neural network, and more accurate road information can be obtained. And each set of obtained road information comprises vanishing point coordinates, vanishing point confidence coefficient and curve information, target road information is selected from the sets of road information according to the vanishing point confidence coefficient in each set of road information, the road information output due to error detection can be eliminated, and the road in the image is determined according to the curve information in the target road information with correct detection, so that the accuracy of road detection can be further improved.
In an alternative embodiment, the selecting the target road information from the plurality of sets of road information according to the vanishing point confidence in each set of road information includes:
forming a candidate information set by all road information;
determining vanishing points with the highest vanishing point confidence coefficient in the candidate information set;
removing road information corresponding to vanishing points of which the distance between the vanishing points with the highest vanishing point confidence coefficient is smaller than a set distance threshold from the candidate information set;
removing road information corresponding to the vanishing point with the highest vanishing point confidence coefficient in the candidate information set from the candidate information set, and adding the road information to the target information set;
and returning to execute the step of determining the vanishing point with the highest vanishing point confidence coefficient in the candidate information set until the candidate information set is empty, and taking the road information in the target information set as the target road information.
In the embodiment, the vanishing point with the highest vanishing point confidence coefficient in each group of road information is determined, the road information corresponding to the vanishing point with the highest confidence coefficient is reserved as correct road information, and the interference road information closer to the vanishing point is filtered out, so that the correct target road information is determined, and the accuracy of road information detection is improved.
In an alternative embodiment, the curve information includes curve coordinate information and a curve confidence;
the determining the road in the image according to the curve information in the target road information includes:
and filtering curves with the curve confidence degrees smaller than a set confidence degree threshold value in the target road information, and determining roads in the image according to curve coordinate information of the remaining curves in the target road information.
In this embodiment, a confidence threshold is set for the obtained curve information, and the curve information with low confidence is removed by using the confidence threshold, so that the road side and the lane line in the road can be detected more accurately.
In an alternative embodiment, the curve information includes a curve confidence;
before the selecting the target road information from the multiple sets of road information according to the vanishing point confidence coefficient in each set of road information, the method further includes:
and filtering curves with the curve confidence degrees smaller than the set confidence degree threshold value in each group of road information.
In the embodiment, after the plurality of groups of road information are obtained, the curve information smaller than the set confidence threshold in the road information is directly removed, so that the workload of road information calculation can be reduced, and the accuracy of road detection can be improved.
In an alternative embodiment, the neural network comprises a feature extraction sub-network and a codec sub-network;
inputting an image containing a road into a trained detection neural network to obtain a plurality of groups of road information in the image; the method comprises the following steps:
inputting the image into a feature extraction sub-network in the detection neural network to obtain image features of the image;
and inputting the image characteristics of the image into a coding and decoding sub-network to obtain a plurality of groups of road information in the image.
In the embodiment, the road image is input to the detection neural network to obtain a plurality of groups of road information, so that vanishing point information and a plurality of curve information in the image can be more accurately extracted, and meanwhile, the curve information in different road information can be distinguished.
In a second aspect, an embodiment of the present application further provides a road detection device, including:
the image processing module is used for inputting images containing roads into the trained detection neural network so as to obtain a plurality of groups of road information in the images; each group of road information comprises vanishing point information and a plurality of pieces of curve information, wherein the vanishing point information comprises vanishing point coordinates and vanishing point confidence coefficients;
and the road detection module is used for selecting target road information from the multiple groups of road information according to the vanishing point confidence coefficient in each group of road information and determining the road in the image according to curve information in the target road information.
In an optional embodiment, the road detection module is specifically configured to:
forming a candidate information set by all road information;
determining vanishing points with the highest vanishing point confidence coefficient in the candidate information set;
removing road information corresponding to vanishing points of which the distance between the vanishing points with the highest vanishing point confidence coefficient is smaller than a set distance threshold from the candidate information set;
removing road information corresponding to the vanishing point with the highest vanishing point confidence coefficient in the candidate information set from the candidate information set, and adding the road information to the target information set;
and returning to execute the step of determining the vanishing point with the highest vanishing point confidence coefficient in the candidate information set until the candidate information set is empty, and taking the road information in the target information set as the target road information.
In an alternative embodiment, the curve information includes curve coordinate information and a curve confidence; the road detection module is specifically configured to:
and filtering curves with the curve confidence degrees smaller than a set confidence degree threshold value in the target road information, and determining roads in the image according to curve coordinate information of the remaining curves in the target road information.
In an alternative embodiment, the curve information includes a curve confidence; the road detection module is further configured to:
and filtering curves with the curve confidence degrees smaller than the set confidence degree threshold value in each group of road information.
In an alternative embodiment, the neural network comprises a feature extraction sub-network and a codec sub-network; the image processing module is specifically configured to:
inputting the image into a feature extraction sub-network in the detection neural network to obtain image features of the image;
and inputting the image characteristics of the image into a coding and decoding sub-network to obtain a plurality of groups of road information in the image.
In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the road detection method of the first aspect is implemented.
In a fourth aspect, the present application further provides an electronic device, including a memory and a processor, where the memory stores a computer program operable on the processor, and when the computer program is executed by the processor, the processor is enabled to implement the road detection method of the first aspect.
For technical effects brought by any one implementation manner in the second aspect to the fourth aspect, reference may be made to technical effects brought by a corresponding implementation manner in the first aspect, and details are not described here.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a flowchart of a road detection method according to an embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of a neural network for detection according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of an encoding and decoding subnetwork provided in the embodiment of the present application;
FIG. 4 is a schematic diagram of a graph equation provided in an embodiment of the present application;
fig. 5 is a schematic diagram of a road detection result according to an embodiment of the present disclosure;
fig. 6 is a flowchart of a specific implementation manner provided by an embodiment of the present application;
fig. 7 is a flowchart of another specific implementation manner provided by the embodiment of the present application;
FIG. 8 is a flowchart of a training method for detecting a neural network according to an embodiment of the present disclosure;
fig. 9 is a schematic structural diagram of a road detection device according to an embodiment of the present disclosure;
fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be described in further detail with reference to the accompanying drawings, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that references in the specification of the present application to the terms "comprises" and "comprising," and variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Some of the words that appear in the text are explained below:
1. the term "vanishing point" in the embodiment of the present application refers to an intersection point of a left road extension line and a right road extension line in a road, and in an acquired road image, the vanishing point may not be located in an image area range, and at this time, a midpoint of a connection line between the left road extension line and the right road extension line may be taken as the vanishing point.
2. The term "lane line" in the embodiments of the present application refers to a road line between a left road and a right road of a road.
With the continuous development of intelligent transportation and automatic driving technologies, the road detection function is more and more obvious. In the related art, when detecting a road edge and a lane line of a road, an image including the road is generally acquired through an image acquisition device, an image identification method is mainly adopted, edge detection is performed on the image by using characteristics such as color texture, and then curve information of the road edge and the lane line is acquired by using methods such as hough transform or curve equation fitting on edge points.
The method comprises the steps of inputting images containing roads into a trained detection neural network, and detecting multiple groups of road information from the images through the detection neural network to obtain more accurate road information. The obtained road information of each group comprises vanishing point coordinates, vanishing point confidence coefficient and curve information, the target road information is selected from the road information of each group according to the vanishing point confidence coefficient in the road information of each group, the road information output due to error detection can be eliminated, and the road in the image is determined according to the curve information in the target road information with correct detection, so that the accuracy of road detection can be further improved, and warning information is sent to the vehicles deviating from the road or lane in the driving process of the vehicles.
The method detects the road edges and the lane lines in the road, can provide support for advanced auxiliary driving, and can help to judge traffic illegal behaviors in intelligent traffic, thereby reducing urban traffic accidents and keeping the traffic smooth.
The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.
The embodiment of the application provides a road detection method, as shown in fig. 1, comprising the following steps:
step S101, inputting images containing roads into the trained detection neural network to obtain multiple groups of road information in the images.
And inputting images containing roads into the trained detection neural network to obtain a plurality of groups of road information output by the detection neural network, wherein each group of road information comprises vanishing point information and a plurality of curve information, and the vanishing point information comprises vanishing point coordinates and vanishing point confidence coefficients. The vanishing point coordinates may roughly indicate the main direction of the road. The curve information of each curve may include curve coordinate information and a curve confidence, and the curve may be a curve corresponding to a lane edge or a lane line in the road, for example, a general road includes a left lane edge, a right lane edge, and a lane line between the left lane edge and the right lane edge, and the lane edge and the lane line respectively correspond to one curve in the image, and a road region in the image may be determined by a position of the curve.
And S102, selecting target road information from the multiple sets of road information according to the vanishing point confidence coefficient in each set of road information, and determining the road in the image according to curve information in the target road information.
Since the correct road information is included in the road information output from the neural network, interference information or error information may also be included. Therefore, for each set of obtained road information, correct road information can be selected from the multiple sets of road information as target road information according to the vanishing point confidence in each set of road information, and the road in the image is determined according to the curve information in the target road information, so that a correct road detection result is obtained.
According to the road detection method provided by the embodiment of the application, vanishing point information and a plurality of curve information in a road image are obtained through the detection neural network, confidence degrees are set for the vanishing point information and the curve information, and filtering operation is carried out on the vanishing point information and the curve information with lower confidence degrees according to the set confidence degree threshold value, so that vanishing points, left and right road edges and lane lines in a road can be detected more accurately, the road information is determined, and the accuracy of road detection is improved.
Specifically, in an alternative embodiment, the neural network used in step S101 may include a feature extraction sub-network and a coding and decoding sub-network. As shown in fig. 2, the collected image including the road is input to a feature extraction sub-network, and the feature extraction sub-network is used to perform feature extraction on the input image and output the image features of the road image. Illustratively, the feature extraction sub-network may be implemented using CNN (Convolutional Neural Networks), for example, a Neural network such as ResNet or MobileNet may be used.
In some embodiments, the feature extraction subnetwork may include a convolutional layer and a pooling layer. The road image is input into a convolution layer of a sub-network for feature extraction, the convolution layer is used for carrying out feature extraction on the input road image to obtain a feature vector, and the feature vector is input into a pooling layer, and the convolution layer can use 2 x 2 convolution kernels. And the pooling layer is used for performing dimension reduction processing on the input feature vector to obtain the image features of the road image.
In other embodiments, the feature extraction sub-network may be comprised of a plurality of convolutional layers, e.g., the feature extraction sub-network may be implemented using a full convolutional neural network.
And after the image features output by the feature extraction sub-network are obtained, inputting the image features into a coding and decoding sub-network, wherein the coding and decoding sub-network is used for separating the road information from the image features and outputting a plurality of groups of road information corresponding to the road images. Each set of road information comprises vanishing point information and a plurality of pieces of curve information, wherein the vanishing point information comprises vanishing point coordinates and vanishing point confidence coefficients, and the curve information comprises curve coordinate information and curve confidence coefficients.
In some embodiments, the codec sub-network may employ a transform-like codec network, and the structure of the codec sub-network may be as shown in fig. 3, including a Channel Attention layer (Channel Attention) and a Global Context model layer (Global Context Modeling). The channel attention layer is used for giving attention weights with different sizes to different image characteristic information so as to reflect that the importance of the different image characteristic information to the detection result is different. The global context model layer may also be referred to as a full connection layer, and is configured to perform road detection based on the attention weighted image features output by the channel attention layer, and output the multiple sets of road information. The fully-connected layer may employ a sigmoid function as an activation function.
In a specific embodiment, a road image is input into a feature extraction sub-network in a detection neural network, image features of the road image are extracted, and then the extracted image features are input into a coding and decoding sub-network in the detection neural network, so that a plurality of groups of road information in the road image are obtained, wherein each group of road information comprises a vanishing point information and a plurality of curve information. The vanishing point information comprises vanishing point coordinates and vanishing point confidence coefficients, and the value ranges of the horizontal and vertical coordinates of the vanishing point are [0, 1 ] in the relative coordinate system]And outputting vanishing point coordinates by using a sigmoid function, wherein the vanishing point coordinates can indicate the main direction of the road. The value range of the vanishing point confidence coefficient is [0, 1 ]]The output may be performed by a sigmoid function. Only one vanishing point is arranged on one road, and after a plurality of groups of road information are obtained, the road information with low confidence coefficient can be filtered by using a confidence coefficient threshold value. The curve information comprises curve coordinate information and curve confidence, a confidence is set for each curve, curves with low confidence can be filtered out by using a confidence threshold, and the curve coordinate information comprises a curve coefficient of the curve and a curve section interval. For example, the curve coefficient of a curve may be p0, p1, …, pn, where n represents the order number of the curve equation, and after obtaining the curve coefficient, the curve equation may be established with the transverse direction of the road image as the y-axis and the longitudinal direction as the x-axis, and since the road edge and the lane line in the road image represent a curve segment, it is also necessary to set the maximum value x on the x-axis for a curvemaxAnd the minimum value xminObtaining the curve segment corresponding to the curve. The curve equation of the curve is shown in fig. 4, wherein the order of the curve equation can be selected according to the actual scene. If the shape of the curved road in the image is simpler, the order of the curved road can be selected to be two, and if the road in the image is in an S shape, the order of the curved road can be selected to be three or higher. According to the determined curve equation and the section of the curve segment, the positions of the road edge line and the lane line in the road image can be determined.
In an embodiment, in step S102, after obtaining the plurality of sets of road information, the target road information may be selected from the plurality of sets of road information by a Non-Maximum Suppression (Non-Maximum Suppression) method according to the vanishing point confidence in each set of road information.
Firstly, all the output road information is formed into a candidate information set, and a target information set is established, wherein the target information set is an empty set at the beginning. Determining a vanishing point with the highest vanishing point confidence coefficient in the candidate information set, then removing road information corresponding to vanishing points with the highest vanishing point confidence coefficient smaller than a set distance threshold from the candidate information set, then removing the road information corresponding to the vanishing points with the highest vanishing point confidence coefficient in the candidate information set from the candidate information set, adding the road information into the target information set, returning to the step of determining the vanishing point with the highest vanishing point confidence coefficient in the candidate information set until the candidate information set is empty, and finally taking the road information in the target information set as the target road information.
For example, in one embodiment, an image containing a road is input into a trained detection neural network, and the detection neural network outputs 8 sets of road information, respectively: road information L1, road information L2, road information L3, road information L4, road information L5, road information L6, road information L7 and road information L8, the vanishing point corresponding to the road information L1 is M1, the vanishing point corresponding to the road information L2 is M2, …, and the vanishing point corresponding to the road information L8 is M8. And forming a candidate information set B by using the output 8 groups of road information, and establishing a target information set S, wherein the set S is an empty set at the beginning. In the first round, assuming that the vanishing point with the highest vanishing point confidence coefficient in the set B is determined to be the vanishing point M1, a distance threshold N1 is set, and may be a 20-pixel value, and distances between the vanishing point M3 and the vanishing point M4 and between the vanishing point M1 and the vanishing point M1 are respectively smaller than the 20-pixel value, then the road information L1, the road information L3 and the road information L4 in the set B may be removed, and at this time, the road information in the set B includes the road information L2, the road information L5, the road information L6, the road information L7 and the road information L8, and then the road information L1 is added to the set S. In the second round, assuming that the vanishing point with the highest vanishing point confidence coefficient in the set B is determined to be the vanishing point M2, a distance threshold N2 is set, which may be a 10-pixel value, and the distances between the vanishing point M7 and the vanishing point M8 and the vanishing point M2 are respectively less than 10-pixel values, then the road information L2, the road information L7 and the road information L8 in the set B are removed, and the road information L2 is added to the set S. At this time, the set B includes road information L5 and road information L6, and the set S includes road information L1 and road information L2. Then, the third round is performed, and the above steps are repeatedly executed until the set B is empty, and finally the road information in the set S is set as the target road information.
In another embodiment, an image containing a road is input into a trained detection neural network, and the detection neural network outputs 3 sets of road information, which are respectively: the method comprises the steps of obtaining road information L1, road information L2 and road information L3, enabling a vanishing point corresponding to the road information L1 to be M1, enabling a vanishing point corresponding to the road information L2 to be M2, enabling a vanishing point corresponding to the road information L3 to be M3, enabling output 3 groups of road information to form a candidate information set B, and establishing a target information set S, wherein the set S is an empty set at the beginning. Assuming that the vanishing point with the highest vanishing point confidence coefficient in the set B is determined to be the vanishing point M1, a distance threshold N1 is set, which may be 15 pixel values, and distances between the vanishing point M2 and the vanishing point M3 and the vanishing point M1 are respectively smaller than 15 pixel values, then the road information L1, the road information L2 and the road information L3 in the set B may be removed, and the road information L1 is added to the set S, where the set B is empty, so that the obtained target road information is the road information L1.
In a specific embodiment, the finally obtained target road information is shown in fig. 5, and the road information includes 3 curves and a white point, where the white point represents a vanishing point, two black solid curves represent left and right road edges, respectively, and a black dashed curve represents a lane line. For a road, the vanishing point, i.e. the intersection of the extension lines of the left and right road sides, the lane line in the center of the road and the left and right road sides can determine the road. The vanishing point may not be located in the image area, and at this time, the midpoint of the connection line between the tail ends of the extension lines of the left and right road sides may be taken as the vanishing point.
In an embodiment, a confidence level may be set for each curve output, and when the confidence level of the curve is smaller than a set confidence level threshold, the curve is filtered out, so that the lane line in the road image where the lane line does not exist can be removed by this method.
In another embodiment, a plurality of lane lines exist in the center of the road image, that is, the road image not only includes two left and right lane lines, but also includes more than one lane line, so that the number of curves output by the neural network can be increased. For the road image with the bifurcation crossing, the vanishing points of two roads in the image are far away, and each road side line and the lane line in the road can be effectively detected according to the method, so that the lane lines in different roads can be distinguished.
In order to more conveniently understand the road detection method provided by the embodiment of the present application, the following describes the implementation process thereof by two specific embodiments.
In one embodiment, the road detection method is implemented as shown in fig. 6, and includes the following steps:
step S601, inputting an image including a road into the trained detection neural network to obtain a plurality of sets of road information in the image.
Each set of road information comprises vanishing point information and a plurality of pieces of curve information, the vanishing point information comprises vanishing point coordinates and vanishing point confidence coefficients, and the curve information comprises curve coordinate information and curve confidence coefficients.
Step S602, selecting target road information from the multiple sets of road information according to the vanishing point confidence coefficient in each set of road information.
The specific implementation process of this step can be referred to the introduction above, and is not described herein again.
Step S603, a curve in the target road information, in which the confidence of the curve is smaller than the set confidence threshold, is filtered out.
After the target road information is selected from the plurality of sets of road information, the road in the image may be determined according to the curve information in the target road information. Because some interference or wrong curve information exists in the selected target road information, curves with the curve confidence degrees smaller than the set confidence degree threshold value in the target road information can be filtered according to the curve confidence degrees included in the curve information, and therefore wrong curves in the target road information are removed.
In step S604, the road in the image is determined according to the curve coordinate information of the remaining curves in the target road information.
The curve coordinate information may include a curve coefficient of the curve and an interval of the curve segment. Because the road in the image is presented in the form of a curve segment, firstly, the curve coefficient of the curve is determined, the curve equation of the road is established, and then the section which can be taken by the curve segment is determined. And finally, determining the road in the image according to the curve coefficient of the curve and the section of the curve segment.
For example, in one embodiment, an image containing a road is input into a trained detection neural network, which outputs 3 sets of road information, respectively: road information L1, road information L2, and road information L3. The vanishing point corresponding to the road information L1 is M1, the vanishing point corresponding to the road information L2 is M2, the vanishing point corresponding to the road information L3 is M3, and the target road information selected from the 3 sets of road information is road information L1 according to the vanishing point confidence coefficients in each set of road information, namely, the vanishing point confidence coefficients of the vanishing point M1, the vanishing point M2 and the vanishing point M3. Assuming that the road information L1 includes curve information of 5 curves, where the curve confidence of 3 curves is smaller than the set confidence threshold, after the 3 wrong curves are removed, only curve information of 2 curves remains in the road information L1, and the two curves may correspond to a left side line and a right side line in the road. According to the curve coordinate information of only 2 curves left in the road information L1, that is, according to the curve coefficients and the curve segment sections of the 2 curves, the positions of the left and right road edges in the image can be determined, and further, the road region in the image can be determined.
In another embodiment, the road detection method is implemented as shown in fig. 7, and includes the following steps:
step S701, inputting an image including a road into the trained detection neural network to obtain a plurality of sets of road information in the image.
Step S702, the curves with the curve confidence degrees smaller than the set confidence degree threshold value in each group of road information are filtered.
And step S703, selecting target road information from the multiple sets of road information according to the vanishing point confidence coefficient in each set of road information.
And selecting target road information from the multiple groups of road information by a non-maximum suppression method according to the vanishing point confidence coefficient in each group of road information.
In step S704, a road in the image is determined according to the curve information in the target road information.
For example, in one embodiment, an image containing a road is input into a trained detection neural network, which outputs 3 sets of road information, respectively: road information L1, road information L2, and road information L3. Assuming that the road information L1 includes curve information of 5 curves, where the confidence of 2 curves is less than the set confidence threshold, the curve information of only 3 curves in the road information L1 may be left after the 2 erroneous curves are removed. The road information L2 includes curve information of 4 curves, wherein the curve confidence of 1 curve is less than the set confidence threshold, and after the wrong curve is removed, only curve information of 3 curves remains in the road information L2. The road information L3 includes curve information of 6 curves, wherein the curve confidence of 3 curves is less than the set confidence threshold, and after the 3 curves with errors are removed, only the curve information of 3 curves remains in the road information L2. The vanishing point corresponding to the road information L1 is M1, the vanishing point corresponding to the road information L2 is M2, the vanishing point corresponding to the road information L3 is M3, and the target road information selected from the 3 sets of road information is road information L1 according to the vanishing point confidence coefficients in each set of road information, namely, the vanishing point confidence coefficients of the vanishing point M1, the vanishing point M2 and the vanishing point M3. At this time, the road information L1 only has curve information of 3 curves left, and may correspond to the left side line, the right side line, and the lane line in the road, and the positions of the left side line, the right side line, and the lane line in the image may be determined according to the curve coordinate information of the 3 curves, that is, according to the curve coefficients and the sections of the curve segments of the 3 curves, so as to determine the road area in the image.
Optionally, the training process for detecting a neural network adopted in the above embodiment may be as shown in fig. 8, and includes the following steps:
in step S801, a road image data set is obtained.
The image acquisition device can be used for acquiring a plurality of road images to obtain a road image data set, the road images in the road image data set are marked, and a vanishing point, a left road side line, a right road side line and a middle lane line are marked in the images.
Step S802, inputting the road image data into the detection neural network to be trained to obtain a plurality of groups of road information corresponding to each road image.
And extracting a plurality of road images from the marked road images, inputting the road images into a detection neural network to be trained, and outputting a plurality of groups of road information corresponding to each road image.
In step S803, a loss value is determined from the obtained road information and the input road image data.
And comparing the obtained multiple groups of road information with the input marked road image to determine a loss value. The loss value may be calculated by a likelihood function.
Step S804, determining whether the loss value converges to a preset target value; if not, executing step S805; if so, step S806 is performed. And step S805, adjusting parameters of the detection neural network to be trained according to the determined loss value.
And if the loss value is not converged, adjusting the parameters of the detected neural network to be trained, and after the parameters are adjusted, returning to execute the step S802 to continue the next round of training process.
And step S806, finishing the training to obtain the trained detection neural network.
And if the loss value is converged, taking the currently obtained detection neural network as the trained detection neural network.
The road detection method shown in the figure 1 is based on the same inventive concept, and the embodiment of the application also provides a road detection device. Because the device is a device corresponding to the road detection method and the principle of solving the problems of the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and repeated details are not repeated.
Fig. 9 is a schematic structural diagram of a road detection device according to an embodiment of the present application, and as shown in fig. 9, the road detection device includes an image processing module 901 and a road detection module 902.
The image processing module 901 is configured to input an image including a road into a trained detection neural network to obtain multiple sets of road information in the image; each group of road information comprises vanishing point information and a plurality of pieces of curve information, wherein the vanishing point information comprises vanishing point coordinates and vanishing point confidence coefficients;
the road detection module 902 is configured to select target road information from the multiple sets of road information according to the vanishing point confidence in each set of road information, and determine a road in the image according to curve information in the target road information.
In an alternative embodiment, the road detection module 902 is specifically configured to:
forming a candidate information set by all road information;
determining vanishing points with the highest vanishing point confidence coefficient in the candidate information set;
removing road information corresponding to vanishing points of which the distance between the vanishing points with the highest vanishing point confidence coefficient is smaller than a set distance threshold from the candidate information set;
removing road information corresponding to the vanishing point with the highest vanishing point confidence coefficient in the candidate information set from the candidate information set, and adding the road information to the target information set;
and returning to execute the step of determining the vanishing point with the highest vanishing point confidence coefficient in the candidate information set until the candidate information set is empty, and taking the road information in the target information set as the target road information.
In an alternative embodiment, the curve information includes curve coordinate information and a curve confidence; the road detection module 902 is specifically configured to:
and filtering curves with the curve confidence degrees smaller than the set confidence degree threshold value in the target road information, and determining the road in the image according to the curve coordinate information of the residual curves in the target road information.
In an alternative embodiment, the curve information includes a curve confidence; a road detection module 902, further configured to:
and filtering curves with the curve confidence degrees smaller than the set confidence degree threshold value in each group of road information.
In an alternative embodiment, the neural network includes a feature extraction sub-network and a codec sub-network; the image processing module 901 is specifically configured to:
inputting the image into a feature extraction sub-network in a detection neural network to obtain image features of the image;
and inputting the image characteristics of the image into the coding and decoding sub-network to obtain a plurality of groups of road information in the image.
The electronic equipment is based on the same inventive concept as the method embodiment, and the embodiment of the application also provides the electronic equipment. The electronic device may be used for road detection. In one embodiment, the electronic device may be a server, a terminal device, or other electronic device. In this embodiment, the electronic device may be configured as shown in fig. 10, and include a memory 101, a communication module 103, and one or more processors 102.
A memory 101 for storing a computer program for execution by the processor 102. The memory 101 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, a program required for running an instant messaging function, and the like; the storage data area can store various instant messaging information, operation instruction sets and the like.
The memory 101 may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 101 may also be a non-volatile memory (non-volatile memory) such as, but not limited to, a read-only memory (rom), a flash memory (flash memory), a Hard Disk Drive (HDD) or a solid-state drive (SSD), or the memory 101 may be any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Memory 101 may be a combination of the above.
The processor 102 may include one or more Central Processing Units (CPUs), or be a digital processing unit, etc. A processor 102 for implementing the above-described road detection method when calling a computer program stored in the memory 101.
The communication module 103 is used for communicating with terminal equipment and other servers.
The specific connection medium among the memory 101, the communication module 103 and the processor 102 is not limited in the embodiments of the present application. In fig. 10, the memory 101 and the processor 102 are connected by a bus 104, the bus 104 is represented by a thick line in fig. 10, and the connection manner between other components is merely illustrative and not limited. The bus 104 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 10, but this is not intended to represent only one bus or type of bus.
According to an aspect of the application, a computer program product or computer program is provided, comprising computer instructions, the computer instructions being stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the road detection method in the above-described embodiment. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application.
Claims (10)
1. A method of road detection, the method comprising:
inputting an image containing a road into a trained detection neural network to obtain a plurality of groups of road information in the image; each group of road information comprises vanishing point information and a plurality of pieces of curve information, wherein the vanishing point information comprises vanishing point coordinates and vanishing point confidence coefficients;
and selecting target road information from the multiple groups of road information according to the vanishing point confidence coefficient in each group of road information, and determining the road in the image according to curve information in the target road information.
2. The method of claim 1, wherein selecting the target road information from the plurality of sets of road information based on the vanishing point confidence in each set of road information comprises:
forming a candidate information set by all road information;
determining vanishing points with the highest vanishing point confidence coefficient in the candidate information set;
removing road information corresponding to vanishing points of which the distance between the vanishing points with the highest vanishing point confidence coefficient is smaller than a set distance threshold from the candidate information set;
removing road information corresponding to the vanishing point with the highest vanishing point confidence coefficient in the candidate information set from the candidate information set, and adding the road information to the target information set;
and returning to execute the step of determining the vanishing point with the highest vanishing point confidence coefficient in the candidate information set until the candidate information set is empty, and taking the road information in the target information set as the target road information.
3. The method of claim 1, wherein the curve information includes curve coordinate information and a curve confidence;
the determining the road in the image according to the curve information in the target road information includes:
and filtering curves with the curve confidence degrees smaller than a set confidence degree threshold value in the target road information, and determining roads in the image according to curve coordinate information of the remaining curves in the target road information.
4. The method of claim 1, wherein the curve information includes a curve confidence;
before the selecting the target road information from the multiple sets of road information according to the vanishing point confidence coefficient in each set of road information, the method further includes:
and filtering curves with the curve confidence degrees smaller than the set confidence degree threshold value in each group of road information.
5. The method of any one of claims 1-4, wherein the neural network comprises a feature extraction sub-network and a codec sub-network;
inputting an image containing a road into a trained detection neural network to obtain a plurality of groups of road information in the image; the method comprises the following steps:
inputting the image into a feature extraction sub-network in the detection neural network to obtain image features of the image;
and inputting the image characteristics of the image into a coding and decoding sub-network to obtain a plurality of groups of road information in the image.
6. A road detection device, comprising:
the image processing module is used for inputting images containing roads into the trained detection neural network so as to obtain a plurality of groups of road information in the images; each group of road information comprises vanishing point information and a plurality of pieces of curve information, wherein the vanishing point information comprises vanishing point coordinates and vanishing point confidence coefficients;
and the road detection module is used for selecting target road information from the multiple groups of road information according to the vanishing point confidence coefficient in each group of road information and determining the road in the image according to curve information in the target road information.
7. The apparatus of claim 6, wherein the road detection module is specifically configured to:
forming a candidate information set by all road information;
determining vanishing points with the highest vanishing point confidence coefficient in the candidate information set;
removing road information corresponding to vanishing points of which the distance between the vanishing points with the highest vanishing point confidence coefficient is smaller than a set distance threshold from the candidate information set;
removing road information corresponding to the vanishing point with the highest vanishing point confidence coefficient in the candidate information set from the candidate information set, and adding the road information to the target information set;
and returning to execute the step of determining the vanishing point with the highest vanishing point confidence coefficient in the candidate information set until the candidate information set is empty, and taking the road information in the target information set as the target road information.
8. The apparatus of any one of claims 6-7, wherein the neural network comprises a feature extraction sub-network and a codec sub-network; the image processing module is specifically configured to:
inputting the image into a feature extraction sub-network in the detection neural network to obtain image features of the image;
and inputting the image characteristics of the image into a coding and decoding sub-network to obtain a plurality of groups of road information in the image.
9. A computer-readable storage medium having a computer program stored therein, the computer program characterized by: the computer program, when executed by a processor, implements the method of any of claims 1-5.
10. An electronic device comprising a memory and a processor, the memory having stored thereon a computer program operable on the processor, the computer program, when executed by the processor, implementing the method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010943653.9A CN112199999A (en) | 2020-09-09 | 2020-09-09 | Road detection method, road detection device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010943653.9A CN112199999A (en) | 2020-09-09 | 2020-09-09 | Road detection method, road detection device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112199999A true CN112199999A (en) | 2021-01-08 |
Family
ID=74014942
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010943653.9A Pending CN112199999A (en) | 2020-09-09 | 2020-09-09 | Road detection method, road detection device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112199999A (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107918775A (en) * | 2017-12-28 | 2018-04-17 | 聊城大学 | The zebra line detecting method and system that a kind of auxiliary vehicle safety drives |
| CN108216229A (en) * | 2017-09-08 | 2018-06-29 | 北京市商汤科技开发有限公司 | The vehicles, road detection and driving control method and device |
| CN110084236A (en) * | 2019-04-29 | 2019-08-02 | 北京朗镜科技有限责任公司 | A kind of antidote and device of image |
| CN110222658A (en) * | 2019-06-11 | 2019-09-10 | 腾讯科技（深圳）有限公司 | The acquisition methods and device of road vanishing point position |
| CN110276293A (en) * | 2019-06-20 | 2019-09-24 | 百度在线网络技术（北京）有限公司 | Method for detecting lane lines, device, electronic equipment and storage medium |
| CN111062413A (en) * | 2019-11-08 | 2020-04-24 | 深兰科技（上海）有限公司 | Road target detection method and device, electronic equipment and storage medium |
| CN111077893A (en) * | 2019-12-31 | 2020-04-28 | 驭势科技(南京)有限公司 | Navigation method based on multiple vanishing points, electronic equipment and storage medium |
| CN111174796A (en) * | 2019-12-31 | 2020-05-19 | 驭势科技(南京)有限公司 | Navigation method based on single vanishing point, electronic equipment and storage medium |
| WO2020098708A1 (en) * | 2018-11-14 | 2020-05-22 | 北京市商汤科技开发有限公司 | Lane line detection method and apparatus, driving control method and apparatus, and electronic device |
| CN111210411A (en) * | 2019-12-31 | 2020-05-29 | 驭势科技(南京)有限公司 | Detection method of vanishing points in image, detection model training method and electronic equipment |
| CN111540010A (en) * | 2020-05-15 | 2020-08-14 | 百度在线网络技术（北京）有限公司 | Road monitoring method and device, electronic equipment and storage medium |
-
2020
- 2020-09-09 CN CN202010943653.9A patent/CN112199999A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108216229A (en) * | 2017-09-08 | 2018-06-29 | 北京市商汤科技开发有限公司 | The vehicles, road detection and driving control method and device |
| CN107918775A (en) * | 2017-12-28 | 2018-04-17 | 聊城大学 | The zebra line detecting method and system that a kind of auxiliary vehicle safety drives |
| WO2020098708A1 (en) * | 2018-11-14 | 2020-05-22 | 北京市商汤科技开发有限公司 | Lane line detection method and apparatus, driving control method and apparatus, and electronic device |
| CN110084236A (en) * | 2019-04-29 | 2019-08-02 | 北京朗镜科技有限责任公司 | A kind of antidote and device of image |
| CN110222658A (en) * | 2019-06-11 | 2019-09-10 | 腾讯科技（深圳）有限公司 | The acquisition methods and device of road vanishing point position |
| CN110276293A (en) * | 2019-06-20 | 2019-09-24 | 百度在线网络技术（北京）有限公司 | Method for detecting lane lines, device, electronic equipment and storage medium |
| CN111062413A (en) * | 2019-11-08 | 2020-04-24 | 深兰科技（上海）有限公司 | Road target detection method and device, electronic equipment and storage medium |
| CN111077893A (en) * | 2019-12-31 | 2020-04-28 | 驭势科技(南京)有限公司 | Navigation method based on multiple vanishing points, electronic equipment and storage medium |
| CN111174796A (en) * | 2019-12-31 | 2020-05-19 | 驭势科技(南京)有限公司 | Navigation method based on single vanishing point, electronic equipment and storage medium |
| CN111210411A (en) * | 2019-12-31 | 2020-05-29 | 驭势科技(南京)有限公司 | Detection method of vanishing points in image, detection model training method and electronic equipment |
| CN111540010A (en) * | 2020-05-15 | 2020-08-14 | 百度在线网络技术（北京）有限公司 | Road monitoring method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6802331B2 (en) | Lane processing method and equipment | |
| CN110390306B (en) | Method for detecting right-angle parking space, vehicle and computer readable storage medium | |
| CN111401150B (en) | Multi-lane line detection method based on example segmentation and self-adaptive transformation algorithm | |
| CN111380532B (en) | Path planning method, device, terminal and computer storage medium | |
| CN111433780A (en) | Lane line detection method, lane line detection apparatus, and computer-readable storage medium | |
| CN114581887A (en) | Method, device and equipment for detecting lane line and computer readable storage medium | |
| CN112132892B (en) | Target position labeling method, device and equipment | |
| CN112597995B (en) | License plate detection model training method, device, equipment and medium | |
| CN110807461B (en) | Target position detection method | |
| CN112199999A (en) | Road detection method, road detection device, storage medium and electronic equipment | |
| CN111652181A (en) | Target tracking method and device and electronic equipment | |
| CN114820755B (en) | Depth map estimation method and system | |
| CN111144361A (en) | Road lane detection method based on binaryzation CGAN network | |
| CN112651417A (en) | License plate recognition method, device, equipment and storage medium | |
| Piao et al. | Lane detection of unstructured roads based on WS-P2PNet | |
| CN112990009A (en) | End-to-end-based lane line detection method, device, equipment and storage medium | |
| CN112912894B (en) | Road boundary identification method and device | |
| CN115082888A (en) | Lane line detection method and device | |
| CN115214627A (en) | Parking prompting method and device, electronic equipment and storage medium | |
| CN114495056A (en) | Parking lot pillar detection method, detection device, vehicle and storage medium | |
| CN112686155A (en) | Image recognition method, image recognition device, computer-readable storage medium and processor | |
| CN116558541B (en) | Model training method and device, and track prediction method and device | |
| CN116558540B (en) | Model training method and device, and track generating method and device | |
| CN114155508B (en) | Road change detection method, device, equipment and storage medium | |
| CN117392639A (en) | Training method and device of behavior prediction model, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |