WO2020103892A1 - Lane line detection method and apparatus, electronic device, and readable storage medium - Google Patents

Abstract

Embodiments of the present invention provide a lane line detection method and apparatus, an electronic device, and a readable storage medium. The method comprises: obtaining a road surface image acquired by a vehicle-mounted device on a vehicle; inputting the road surface image to a neural network, and outputting M probability graphs corresponding to the road surface image by means of the neural network, the M probability graphs comprising N lane line probability graphs and M-N non-lane line probability graphs, the N lane line probability graphs respectively corresponding to N lane lines on the road surface and being used for representing the probabilities that the pixel points in the road surface image belong to the corresponding lane lines, and the M-N non-lane line probability graphs corresponding to non-lane lines on the road and being used for representing the probabilities that the pixel points in the road surface image belong to the non-lane lines; and determining the lane lines in the road surface image according to the lane line probability graphs.

Description

Lane line detection method, device, electronic equipment and readable storage medium

This disclosure requires the priority of the Chinese patent application filed on November 21, 2018 with the Chinese Patent Office, application number CN201811392943.8, and the invention titled "lane line detection method, device, electronic equipment, and readable storage medium". The entire contents are incorporated by reference in this disclosure.

Technical field

Embodiments of the present disclosure relate to computer technology, and in particular, to a lane line detection method, device, electronic device, and readable storage medium.

Background technique

Assisted driving and automatic driving are two important technologies in the field of intelligent driving. Through assisted driving or automatic driving, the interval between workshops can be reduced, the occurrence of traffic accidents can be reduced, and the physical and mental burden of the driver can be reduced. Therefore, it plays an important role in the field of intelligent driving. effect.

In assisted driving technology and automatic driving technology, lane line detection is required, that is, detection of the lane line on the road surface of the vehicle. In assisted driving, lane line detection can provide early warning for the vehicle's deviation, and can also issue a warning when the vehicle is about to collide with the vehicle in front. In automatic driving, lane line detection can provide the most basic information for operations such as automatic cruise driving, lane keeping, and vehicle overtaking, thus ensuring the normal driving of the vehicle. Therefore, how to carry out accurate and efficient lane line detection is an important subject worth studying.

Summary of the invention

An embodiment of the present disclosure provides a technical solution for lane line detection.

An aspect of an embodiment of the present disclosure provides a lane line detection method, including:

Obtain the road surface image collected by the on-board equipment installed on the vehicle;

The road surface image is input to a neural network, and M probability maps corresponding to the road surface image are output through the neural network. The M probability maps include N lane line probability maps and MN non-lane line probability maps. The N lane line probability maps respectively correspond to N lane lines on the road surface, and are used to represent the probability that pixels in the road surface image belong to the corresponding lane line; the MN non-lane line probability maps correspond to the road surface The non-lane line of is used to represent the probability that the pixels in the road surface image belong to the non-lane line, where N is a positive integer and M is an integer greater than N;

The lane line in the road surface image is determined according to the lane line probability map.

Another aspect of an embodiment of the present disclosure provides a lane line detection device, including:

The first obtaining module is used to obtain the road surface image collected by the vehicle-mounted equipment installed on the vehicle;

The second acquisition module is used to input the road surface image into a neural network, and output M probability maps corresponding to the road surface image via the neural network, the M probability maps include N lane line probability maps and MN Non-lane line probability map, the N lane-line probability maps respectively correspond to N lane lines on the road surface, and are used to represent the probability that pixels in the road surface image belong to the corresponding lane line; the MN non-lane lines The probability map corresponds to the non-lane line on the road surface and is used to represent the probability that the pixel points in the road surface image belong to the non-lane line, where N is a positive integer and M is an integer greater than N;

The first determining module is configured to determine the lane line in the road surface image according to the lane line probability map.

In another aspect of the embodiments of the present disclosure, a driving control method is provided, including:

The driving control device acquires the lane line detection result of the road surface image, and the lane line detection result of the road surface image is obtained by using the lane line detection method as described in any one of the above embodiments;

The driving control device outputs prompt information according to the lane line detection result and / or performs intelligent driving control on the vehicle.

In still another aspect of the embodiments of the present disclosure, a driving control device is provided, including:

An acquisition module, for acquiring a lane line detection result of a road surface image, the lane line detection result of the road surface image is obtained by using the lane line detection method as described in any of the above embodiments;

The driving control module is configured to output prompt information according to the detection result of the lane line and / or perform intelligent driving control on the vehicle.

In still another aspect of the embodiments of the present disclosure, an electronic device is provided, including:

Memory, used to store program instructions;

The processor is configured to call and execute program instructions in the memory to execute the method steps described in any one of the foregoing embodiments.

According to still another aspect of the embodiments of the present disclosure, an intelligent driving system is provided, including: a camera connected in communication, an electronic device according to any of the above embodiments, and a driving control device according to any of the above embodiments, the The camera is used to obtain road images.

In still another aspect of the embodiments of the present disclosure, a readable storage medium is provided, and the readable storage medium stores a computer program, and the computer program is used to execute the method steps described in any one of the foregoing embodiments.

The lane line detection method, device, electronic device, and readable storage medium provided by the embodiments of the present disclosure use a neural network trained by a road surface training image that includes lane line and / or non-lane line annotation information to obtain pixels in the road surface image The points belong to the probability map of the corresponding lane line, and the lane line in the road surface image is determined according to the probability map of the lane line, so that the accurate lane line detection result can be obtained even in the scene with higher complexity. In addition, the M probability maps in this embodiment include non-lane line probability maps, that is, non-lane line categories are added in addition to the lane line categories, so the accuracy of road image segmentation can be improved, thereby improving the lane line detection results Accuracy.

The technical solutions of the present disclosure will be further described in detail below through the accompanying drawings and embodiments.

BRIEF DESCRIPTION

In order to more clearly explain the technical solutions in the present disclosure or the prior art, the following will briefly introduce the drawings required in the embodiments or the description of the prior art. Obviously, the drawings in the following description are For some of the disclosed embodiments, those of ordinary skill in the art can obtain other drawings based on these drawings without paying any creative labor.

1 is a schematic diagram of a scene of a lane line detection method provided by an embodiment of the present disclosure;

2 is a schematic flowchart of an embodiment of a lane line detection method according to an embodiment of the present disclosure;

3 is a schematic flowchart of another embodiment of a lane line detection method according to an embodiment of the present disclosure;

4 is a schematic flowchart of another embodiment of a lane line detection method according to an embodiment of the present disclosure;

FIG. 5 is a schematic structural diagram of a convolutional neural network corresponding to this example;

6 is a schematic flowchart of still another embodiment of a lane line detection method according to an embodiment of the present disclosure;

7 is a schematic flowchart of another embodiment of a lane line detection method according to an embodiment of the present disclosure;

8 is a module structure diagram of an embodiment of a lane line detection device provided by an embodiment of the present disclosure;

9 is a module structure diagram of another embodiment of a lane line detection device provided by an embodiment of the present disclosure;

10 is a block diagram of another embodiment of a lane line detection device provided by an embodiment of the present disclosure;

11 is a block diagram of another embodiment of a lane line detection device provided by an embodiment of the present disclosure;

12 is a block diagram of another embodiment of a lane line detection device provided by an embodiment of the present disclosure;

13 is a block diagram of another embodiment of a lane line detection device provided by an embodiment of the present disclosure;

14 is a block diagram of another embodiment of a lane line detection device provided by an embodiment of the present disclosure;

15 is a module structure diagram of another embodiment of a lane line detection device provided by an embodiment of the present disclosure;

16 is a block diagram of another embodiment of a lane line detection device provided by an embodiment of the present disclosure;

17 is a physical block diagram of an electronic device provided by an embodiment of the present disclosure;

18 is a schematic flowchart of a driving control method provided by an embodiment of the present disclosure;

19 is a schematic structural diagram of a driving control device provided by an embodiment of the present disclosure;

20 is a schematic diagram of an intelligent driving system provided by an embodiment of the present disclosure;

21 is a schematic structural diagram of an application embodiment of an electronic device of the present disclosure.

detailed description

To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments It is a part of the embodiments of the present disclosure, but not all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without creative efforts fall within the protection scope of the present disclosure.

It should be noted that the relative arrangement of components and steps, numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless otherwise stated.

It should also be understood that in the embodiments of the present disclosure, "a plurality" may refer to two or more, and "at least one" may refer to one, two, or more than two.

Those skilled in the art may understand that the terms “first” and “second” in the embodiments of the present disclosure are only used to distinguish different steps, devices, or modules, etc., and neither represent any specific technical meaning nor represent between them. The inevitable logical order.

It should also be understood that any component, data, or structure mentioned in the embodiments of the present disclosure may be generally understood as one or more if it is not clearly defined or given the opposite enlightenment in the foregoing.

It should also be understood that the description of the embodiments of the present disclosure emphasizes the differences between the embodiments, and the same or similarities can be referred to each other, and for the sake of brevity, they will not be described one by one.

At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

The following description of at least one exemplary embodiment is actually merely illustrative, and in no way serves as any limitation to the present disclosure and its application or use.

Techniques, methods and equipment known to those of ordinary skill in the related art may not be discussed in detail, but where appropriate, the techniques, methods and equipment should be considered as part of the specification.

It should be noted that similar reference numerals and letters indicate similar items in the following drawings, so once an item is defined in one drawing, there is no need to discuss it further in subsequent drawings.

In addition, the term "and / or" in the disclosure is just an association relationship describing the association object, indicating that there may be three kinds of relationships, for example, A and / or B, which may mean: there is A alone, A and B exist at the same time There are three cases of B alone. In addition, the character “/” in the present disclosure generally indicates that the related objects before and after are in an “or” relationship.

The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, vehicle-mounted devices, etc., which can operate together with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with terminal devices, computer systems, servers, vehicle-mounted devices, and other electronic devices include, but are not limited to: vehicle-mounted devices, personal computer systems, server computer systems, Thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, mainframe computer systems, in-vehicle equipment, and distribution including any of the above Cloud computing technology environment, etc.

Electronic devices such as terminal devices, computer systems, servers, in-vehicle devices, etc. may be described in the general context of computer system executable instructions (such as program modules) executed by the computer system. Generally, program modules may include routines, programs, target programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

An embodiment of the present disclosure proposes a lane line detection method. A neural network trained through a large amount of labeled data is used to obtain a probability map of each pixel in the road image belonging to the lane line, and the lane line in the road image is determined according to the probability map of the lane line , The end-to-end method can not only get accurate lane line detection results in some simple scenes, such as scenes with good weather conditions and lighting conditions, but also in scenes with high complexity, such as rainy days, In scenes such as nights and tunnels, accurate lane line detection results can also be obtained.

The neural networks in the embodiments of the present disclosure may be a multi-layer neural network (ie, deep neural network), wherein the neural network may be a multi-layer convolutional neural network, for example, LeNet, AlexNet, GoogLeNet, VGG , ResNet and other arbitrary neural network models. Each neural network may use a neural network of the same type and structure, or a neural network of a different type and / or structure. The embodiments of the present disclosure do not limit this.

FIG. 1 is a schematic diagram of a scene of a lane line detection method provided by an embodiment of the present disclosure. As shown in FIG. 1, this method can be applied to vehicles equipped with in-vehicle devices. Wherein, the vehicle-mounted device may be a device with a shooting function such as a camera or a driving recorder installed on the vehicle. When the vehicle is on the road surface, the road surface image is collected by the vehicle-mounted device on the vehicle, and the lane line on the road surface where the vehicle is located is detected based on the method of the embodiment of the present disclosure, so that the detection result can be applied to assisted driving or automatic driving of the vehicle.

FIG. 2 is a schematic flowchart of an embodiment of a lane line detection method according to an embodiment of the present disclosure. As shown in FIG. 2, the method includes:

S201. Acquire a road surface image collected by an on-board device installed on the vehicle.

Optionally, the vehicle-mounted device installed on the vehicle can collect the road surface image on the road surface of the vehicle in real time, and further, the road surface image collected by the vehicle-mounted device can be continuously input into the neural network to obtain continuously updated lane line detection results.

In an optional example, S201 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the first obtaining module 801 executed by the processor.

S202. Input the road surface image into the neural network, and output M probability maps corresponding to the road surface image through the neural network. The M probability maps include N lane line probability maps and MN non-lane line probability maps. The N The lane line probability maps respectively correspond to N lane lines on the road surface, and are used to represent the probabilities that the pixels in the road surface image belong to the corresponding lane lines. The MN non-lane line probability maps correspond to the non-lane lines on the road surface. It indicates the probability that the pixels in the road surface image belong to non-lane lines.

Among them, N is a positive integer, M is an integer greater than N.

Optionally, the above neural network may include but is not limited to a convolutional neural network.

Optionally, the neural network is pre-trained using road surface training image sets including information marked by lane lines and / or non-lane lines. The road training image set includes a large number of training images. Each training image is obtained through the process of collecting actual road images and marking. Optionally, first collect the actual road surface images in various scenarios such as day, night, rain, tunnel, etc., and then, for each actual road surface image, perform pixel-level annotation, that is, mark the category of each pixel in the actual road surface image It is a lane line or a non-lane line to obtain an image for training. Since the neural network is obtained by supervising the training images collected by the rich scenes, the trained neural network can not only get accurate results under some simple scenes, such as daytime scenes with good weather conditions and lighting conditions. The detection result of the lane line can also obtain accurate detection results of the lane line in scenes with high complexity, such as rainy days, nights, tunnels and other scenes.

The training process of the above neural network will be described in detail in the following embodiments.

Optionally, the above non-lane line may refer to a portion of the road surface of the vehicle other than the lane line, and may also be referred to as a road surface background. Exemplarily, roads other than lane lines, cars on the road, plants on the side of the road, etc., all belong to the category of road background.

As an example, the aforementioned M may be equal to 5, and the aforementioned N may be equal to 4. That is, it can be considered that there are 4 lane lines on the road surface of the vehicle, and the neural network can output 5 probability maps. Among them, there are 4 lane line probability maps in the 5 probability maps, which respectively correspond to 4 lane lines on the road surface. That is, the four lane line probability maps correspond one-to-one to the four lane lines on the road surface. In addition, there is one non-lane line probability map in the five probability maps, which corresponds to the non-lane line on the road surface.

As another example, the aforementioned M may be equal to 3, and the aforementioned N may be equal to 2. That is, there are 2 lane lines on the road surface of the vehicle. Correspondingly, the above neural network can output 3 probability maps, and there are 2 lane line probability maps in the 3 probability maps, which respectively correspond to 2 lane lines on the road surface, that is, the 2 lane line probability maps and the 2 on the road surface The lane lines correspond to each other. In addition, there is one non-lane line probability map in the three probability maps, which corresponds to the non-lane line on the road surface.

Assume that the four lane lines on the road surface are lane line 1, lane line 2, lane line 3, and lane line 4 in the order from the left side to the right side of the vehicle, and the four lane line probability maps in the above five probability maps Probability graph 1, probability graph 2, probability graph 3 and probability graph 4 respectively. The corresponding relationship between the lane line probability map and the lane line may be as shown in Table 1 below.

Table 1

Lane line probability map	Probability diagram 1	Probability graph 2	Probability graph 3	Probability graph 4
Lane line	Lane Line 1	Lane Line 2	Lane Line 3	Lane Line 4

That is, the probability map 1 output by the above neural network corresponds to lane line 1, and the probability map 2 corresponds to lane line 2, and so on.

It should be noted that the above Table 1 is only an example of the correspondence between the lane line probability map and the lane line. In the specific implementation process, the correspondence relationship between the lane line probability map and the lane line can be flexibly set according to needs. Examples do not make specific restrictions on this.

Furthermore, exemplarily, based on the correspondence shown in Table 1 above, the probability map 1 can identify the probability that each pixel in the road surface image belongs to the lane line 1. Assuming that the road surface image is represented by a 200 * 200 size matrix, after inputting the matrix into the above neural network, a 200 * 200 size matrix can be output, where the value of each element in the matrix is that the corresponding pixel belongs to the lane line 1 The probability. For example, in the 200 * 200 size matrix output by the neural network, the value of the first row and first column is 0.4, which means that the probability that the pixel of the first row and first column in the road image belongs to the lane line 1 is 0.4. Furthermore, the matrix output by the neural network can be expressed in the form of a lane line probability map.

In an optional example, S202 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the second obtaining module 802 executed by the processor.

S203. Determine the lane line in the road surface image according to the lane line probability map.

After the above steps, the probability that each pixel in the road image belongs to each lane line can be determined, and based on these probabilities, the lane line in the road image can be determined.

In an optional example, S203 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the first determining module 803 executed by the processor.

Optionally, the N lane line probability maps output by the neural network respectively correspond to the N lane line lines on the road surface. For each lane line probability map, some of the pixel points can be selected according to pre-conditions. Point fitting the lane line corresponding to the lane line probability map to obtain N lane lines.

In this embodiment, a neural network trained using a road training image including lane line and / or non-lane line annotation information is used to obtain a probability map of each pixel in the road image belonging to the corresponding lane line, and is determined according to the probability map of the lane line The lane line in the road surface image, so that the accurate lane line detection result can be obtained even in the scene with higher complexity. In addition, the M probability maps in this embodiment include non-lane line probability maps, that is, non-lane line categories are added in addition to the lane line categories, so the accuracy of road image segmentation can be improved, thereby improving the lane line detection results Accuracy.

Optionally, as mentioned above, the N probability maps in the M probability maps correspond to N lane lines on the road surface, then optionally, the Lth lane line probability in the N lane line probability maps The graph corresponds to the Lth lane line, L is any integer greater than or equal to 1 and less than or equal to M, that is, the Lth lane line probability map is any lane line probability map of the N lane line probability maps.

For the above Lth lane line probability map, the Lth lane line may be fitted based on a plurality of pixels with a probability value greater than or equal to a preset threshold in the probability map.

Optionally, in response to the Lth lane line probability map including a plurality of pixels with a probability value greater than or equal to a preset threshold, a plurality of pixels with a probability value greater than or equal to the preset threshold are fitted to the Lth lane line.

First, after the road surface image is input to the neural network, in the Lth lane line probability map output by the neural network, each pixel has a probability value. If the probability value is greater than or equal to the preset threshold, it means that the pixel belongs to the The probability of L lane lines is larger.

Furthermore, after selecting multiple pixels with a probability value greater than or equal to a preset threshold from the Lth lane line probability map, the selected pixels can be calculated for the maximum connected domain, and then based on the maximum connected domain Lane line fitting, so that the lane line in the road image can be obtained.

Exemplarily, the preset threshold may be 0.5, for example.

In an example, assume that the Lth lane line probability map includes the probability values of three pixels, where the probability value of pixel A is 0.5, the probability value of pixel B is 0.6, and the probability value of pixel C is 0.2 That is, the probability value of pixel point A and pixel point B is greater than the preset threshold, then the Lth lane line can be fitted through pixel point A and pixel point B.

In another case, if the Lth lane line probability map does not satisfy the condition that includes multiple pixels with a probability value greater than or equal to the preset threshold, it means that the Lth lane line probability map does not exist in the current road surface image. Corresponding Lth lane line.

In a specific implementation process, there may be a case where the probability values of the same pixel point in multiple probability maps are all greater than or equal to a preset threshold value. In this case, it can be handled as follows.

Optionally, in response to the multiple probability values corresponding to the first pixel point in the multiple lane line probability maps are all greater than or equal to the preset threshold, the first pixel point is used as the pixel point when fitting the first lane line, wherein , The first lane line is the lane line corresponding to the lane line probability map corresponding to the maximum probability value among the multiple probability values.

Exemplarily, assuming that the preset threshold is 0.5, the neural network outputs a total of 4 lane line probability maps. The probability value of the first pixel in the first lane line probability map is 0.5, and the probability of the second lane line probability is 0.5. The probability value in the figure is 0.6, the probability in the third lane line probability map is 0.7, and the probability in the fourth lane line probability map is 0.2, that is, the first pixel is in the first, second, and third lanes The probabilities in the line probability map are greater than or equal to the preset threshold. At this time, it can be considered that the first pixel belongs to the lane line corresponding to the third lane line probability map, that is, the first pixel point is used to fit the third lane line probability map The corresponding lane line.

Through the above processing, the purpose of effectively removing noise can be achieved, and a situation where one pixel point belongs to multiple lane lines can be avoided.

In another embodiment, as described above, the MN probability maps in the M probability maps correspond to non-lane lines on the road surface, and optionally, the S-th lane line in the MN lane line probability maps is optional. The probability map corresponds to the non-lane line, and S is any integer greater than or equal to 1 and less than or equal to MN, that is, the S-th non-lane line probability map is any one of the MN non-lane line probability maps.

For the above S-th lane line probability map, the non-lane line may be determined based on a plurality of pixels with a probability value greater than or equal to a preset threshold in the probability map.

Optionally, in response to the S-th non-lane line probability map including a plurality of pixels with a probability value greater than or equal to a preset threshold, the non-lane line is determined according to a plurality of pixels with a probability value greater than or equal to the preset threshold.

First, after the road surface image is input to the neural network, in the Sth non-lane line probability map output by the neural network, each pixel has a probability value. If the probability value is greater than or equal to the preset threshold, it means that the pixel belongs to The probability of non-lane lines is greater.

Furthermore, after selecting a plurality of pixels with a probability value greater than or equal to a preset threshold from the Sth lane line probability map, the selected pixels can be calculated, for example, to find the maximum connected domain to obtain the road surface image. Non-lane line area.

Exemplarily, the preset threshold may be 0.5, for example.

In an example, assume that the Sth non-lane line probability map includes the probability values of three pixels, where the probability value of pixel A is 0.5, the probability value of pixel B is 0.6, and the probability value of pixel C is 0.2, that is, the probability value of the pixel point A and the pixel point B is greater than the preset threshold, then the non-lane line in the road surface image can be determined by the pixel point A and the pixel point B.

Further, after the lane line in the road surface image is determined through the foregoing embodiment, optionally, the color of the pixel point in the road surface image may be adjusted to the above according to the lane line to which the pixel point in the road surface image belongs. The corresponding color of the lane line to improve the visual effect.

FIG. 3 is a schematic flowchart of another embodiment of a lane line detection method according to an embodiment of the present disclosure. As shown in FIG. 3, the above method further includes:

S301. Perform fusion processing on the above M probability maps to obtain a target probability map.

The M probability maps respectively correspond to a lane line or a non-lane line. After using the M probability maps to fit each lane line and determine the lane line, the M probability maps can be fused into a target probability map. The target probability map contains information for each lane line and information for non-lane lines.

In an optional example, S301 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the fusion module 804 executed by the processor.

S302. Adjust pixel values of pixels corresponding to the first lane line probability map in the target probability map to preset pixel values corresponding to the first lane line probability map.

Wherein, the first lane line probability map is any lane line probability map in the N lane line probability maps, and the pixel points corresponding to the first lane line probability map are composed in the first lane line probability map. Pixels of the combined lane line.

In an optional example, S302 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the adjustment module 805 executed by the processor.

Optionally, after fitting the lane line corresponding to the first lane line probability map by the above embodiment, the pixels constituting the lane line corresponding to the first lane line probability map are determined. In this step, the fused The resulting probability map sets the pixel value of each pixel of the lane line corresponding to the first lane line probability map to the color corresponding to the lane line.

Exemplarily, the color corresponding to each lane line may be set in advance. For example, if there are 4 lane lines on the road surface, the colors of the 4 lane lines may be set to red, yellow, blue, and purple, respectively. After obtaining the target probability map, set the pixel value of each pixel that constitutes each lane line to the corresponding color. After setting, you can get 4 lanes displayed in four colors of red, yellow, blue, and purple line.

In this embodiment, by adjusting the pixel value of the pixel corresponding to the first lane line probability map to the preset pixel value corresponding to the first lane line probability map, the user in the vehicle can view the road surface more intuitively and clearly Lane lanes to enhance user experience.

Based on the above-mentioned embodiment, this embodiment relates to the process of passing the lane line probability map.

FIG. 4 is a schematic flowchart of another embodiment of a lane line detection method according to an embodiment of the present disclosure. As shown in FIG. 4, the above step S202 includes:

S401. Extract low-level feature information of the M channels of the road surface image through at least one convolutional layer of the neural network.

Optionally, the convolutional layer can reduce the resolution of the road surface image and retain the low-level features of the road surface image.

Exemplarily, the low-level feature information of the road surface image may include edge information, straight line information, and curve information in the image.

Optionally, the M channels of the above road surface image respectively correspond to one lane line category, where, assuming there are 4 lane lines on the road surface, there are 5 lane line categories, namely lane line 1, lane line 2, lane line 3. Lane line 4 and non-lane line.

In an optional example, S401 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the first obtaining unit 8021 executed by the processor.

S402. Extract, through at least one residual extraction layer of the neural network, high-level feature information of the M channels of the road surface image based on the low-level feature information of the M channels.

Optionally, the high-level feature information of the M channels of the road surface image extracted through the residual extraction layer includes semantic features, contours, and overall structure.

In an optional example, S402 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the second obtaining unit 8022 executed by the processor.

S403. Upsampling the high-level feature information of the M channels through at least one upsampling layer of the neural network to obtain M probability maps that are as large as the road surface image.

Optionally, through the upsampling process of the upsampling layer, the image can be restored to the original size of the image input to the neural network.

In this step, after upsampling the high-level feature information of the M channels, M probability maps that are as large as the road surface image input to the neural network can be obtained.

In an optional example, S403 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the third obtaining unit 8023 executed by the processor.

Further, optionally, the neural network may further include a normalization layer after the above-mentioned upsampling layer. The normalization layer normalizes the result after the upsampling process, and outputs the above-mentioned lane line probability map.

Exemplarily, the feature map of the road surface image is obtained after the upsampling process, and the value of each pixel in the feature map is normalized, so that the value of each pixel in the feature map is in the range of 0 to 1 To obtain the probability map of the drivable area.

Exemplarily, a normalization method is: first determine the maximum value of the pixels in the feature map, and then divide the value of each pixel by the maximum value, so that the value of each pixel in the feature map In the range of 0 to 1.

It should be noted that the embodiment of the present disclosure does not limit the execution order of the above steps S401 and S402, that is, S401 can be executed before S402, or S402 can be executed before S401.

On the basis of the above-mentioned embodiment, this embodiment relates to this embodiment relates to the training process of establishing the above neural network.

Optionally, based on the foregoing embodiment, it can be known that the neural network involved in the embodiments of the present disclosure may be a convolutional neural network, and the convolutional neural network may include a convolutional layer, a residual extraction layer, an upsampling layer, and a normalization layer . Among them, the order of the convolutional layer and the residual extraction layer can be flexibly set as needed, and the number of each layer can also be flexibly set as needed.

In an alternative manner, the above-mentioned convolutional neural network may include any number of convolutional layers in 6-10 connected, any number of residual extraction layers in 7-12 connected, and any number of 1-4 in convolutional neural networks Upsampling layer.

When the convolutional neural network with this specific structure is used for lane line detection, it can meet the requirements of lane scene detection in multiple scenes or complex scenes, thereby making the detection results more robust.

In one example, the convolutional neural network may include 8 convolutional layers connected, 9 residual extraction layers connected, and 2 upsampling layers connected.

Figure 5 is a schematic diagram of the structure of the convolutional neural network corresponding to this example. As shown in Figure 5, after the road surface image is input, it first passes through 8 consecutive convolutional layers of the convolutional neural network. After that, it includes 9 consecutive residual extraction layers. After the 9 consecutive residual extraction layers, it includes 2 consecutive upsampling layers. After the 2 consecutive upsampling layers, it is a normalized layer, namely Finally, the normalized layer outputs the lane line probability map.

Exemplarily, each of the foregoing residual extraction layers may include 256 filters, and each layer includes 128 filters of 3 * 3 and 128 1 * 1 sizes.

Optionally, before using the neural network to determine the lane line probability map corresponding to the road surface image, the above road network training image set may be used to train the above neural network.

FIG. 6 is a schematic flowchart of still another embodiment of a lane line detection method provided by an embodiment of the present disclosure. As shown in FIG. 6, the training process of the above neural network may be:

S601. Input the training image included in the road surface training image set to the neural network, and obtain a predicted lane line probability map of the training image.

Wherein, the above predicted lane line probability map is the current lane line probability map output by the neural network.

S602: Fit the predicted lane line of the training image according to a plurality of pixel points with a probability value greater than or equal to a preset threshold value included in the predicted lane line probability map.

For the specific process, reference may be made to the above-mentioned part of determining the lane line in the road surface image according to the lane line probability map, which will not be repeated here.

S603: Acquire the loss between the predicted lane line of the training image and the lane line in the truth map of the lane line of the training image.

Wherein, the above lane line truth map is obtained based on the label information of the lane line of the training image.

Alternatively, the loss between the predicted lane line and the lane line in the lane line truth map can be calculated by using a loss function.

S604: Adjust the network parameters of the neural network according to the loss.

Optionally, the network parameters of the neural network may include convolution kernel size and weight information.

In this step, the above-mentioned loss can be back-transmitted in the neural network by means of gradient back propagation, and the network parameters of the neural network can be adjusted.

In an optional example, the S601-S604 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the training module run by the processor.

After this step, a training process is completed and a new neural network is obtained.

Further, iteratively execute the above steps S601-S604 based on the new neural network until the loss of the predicted lane line and the lane line in the true value map of the lane line is within the preset loss range, at this time the trained nerve is obtained The internet.

Exemplarily, the neural network may be trained with one training image at a time, or the neural network may be trained with multiple training images at a time.

7 is a schematic flowchart of another embodiment of a lane line detection method according to an embodiment of the present disclosure. As shown in FIG. 7, before training the neural network, the method further includes:

S701. Collect road surface images in multiple scenes.

S702. Use the image obtained by labeling the road surface in the multiple scenes as the training image.

The above multiple scenes may include, but are not limited to, at least two scenes of daytime scenes, rainy scenes, foggy scenes, straight road scenes, curved road scenes, tunnel scenes, strong light scenes, and night scenes.

In an optional example, the S801-S802 may be executed by the processor invoking the corresponding instruction stored in the memory, or may also be executed by the collection module 806 executed by the processor.

Optionally, the on-vehicle equipment such as the camera on the vehicle can be used to collect the road surface image in each of the above scenarios. Furthermore, the lane line on the collected road surface image can be marked by manual labeling, etc., to obtain each Training images in the scene.

The training images obtained through the above process cover various scenes in practice. Therefore, the neural networks trained using these training images are very robust to the detection of lane lines in various scenarios, and the detection Short time and high accuracy of test results.

As an optional implementation manner, before the road surface image is input to the neural network in step S202, the above road surface image may be de-distorted to further improve the accuracy of the output result of the neural network.

Based on the above embodiments, further, after determining the lane line in the road surface image, the lane line in the road surface image can also be mapped to the world coordinate system to obtain the lane line in the road surface image in the world The position in the coordinate system.

Optionally, each pixel point belonging to the lane line in the road surface image can be coordinate-mapped to obtain lane line information in the world coordinate system, and assist driving based on the obtained lane line information in the world coordinate system Or autonomous driving.

FIG. 8 is a module structure diagram of an embodiment of a lane line detection device provided by an embodiment of the present disclosure. The lane line detection device of the embodiment of the present disclosure may be used to implement the above embodiments of the lane line detection method of the present disclosure. As shown in Figure 8, the device includes:

The first obtaining module 801 is used to obtain the road surface image collected by the vehicle-mounted device installed on the vehicle.

The second acquisition module 802 is configured to input the road surface image into a neural network, and output M probability maps corresponding to the road surface image via the neural network. The M probability maps include N lane line probability maps and MN Non-lane line probability maps, the N lane-line probability maps respectively correspond to N lane lines on the road surface, and are used to represent the probability that pixels in the road surface image belong to the corresponding lane line; the MN non-lane lines The line probability map corresponds to the non-lane line on the road surface, and is used to represent the probability that the pixel point in the road surface image belongs to the non-lane line, where N is a positive integer and M is an integer greater than N.

The first determining module 803 is configured to determine the lane line in the road surface image according to the lane line probability map.

This device is used to implement the foregoing method embodiments, and its implementation principles and technical effects are similar, and will not be repeated here.

9 is a module structure diagram of another embodiment of a lane line detection device according to an embodiment of the present disclosure. As shown in FIG. 9, the first determination module 803 includes: a first determination unit 8031, which is used to determine the probability of the Lth lane line When the figure includes multiple pixels with a probability value greater than or equal to a preset threshold, the Lth lane line is fitted according to the multiple pixels with a probability value greater than or equal to the preset threshold, wherein the Lth lane line probability map It is any lane line probability map of the N lane line probability maps.

FIG. 10 is a module structure diagram of another embodiment of a lane line detection device provided by an embodiment of the present disclosure. As shown in FIG. 10, the first determination module 803 further includes:

The first determining unit 8032 is configured to use the first pixel point as the first lane line when multiple probability values corresponding to the first pixel point in the multiple lane line probability maps are greater than or equal to a preset threshold The pixel point of, where the first lane line is the lane line corresponding to the lane line probability map corresponding to the largest probability value among the multiple probability values.

FIG. 11 is a module structure diagram of Embodiment 4 of a lane line detection device according to an embodiment of the present disclosure. As shown in FIG. 11, the first determination module 803 further includes: a third determination unit 8033, which is used to determine the probability of the Sth non-lane line When the figure includes a plurality of pixels with a probability value greater than or equal to a preset threshold, a non-lane line is determined according to a plurality of pixels with a probability value greater than or equal to a preset threshold, wherein the S-th non-lane line probability map is Describe any of the MN non-lane line probability maps.

FIG. 12 is a module structure diagram of Embodiment 5 of a lane line detection device provided by an embodiment of the present disclosure. As shown in FIG. 12, it further includes: a fusion module 804, configured to perform fusion processing on the M probability maps to obtain a target Probability diagram. The adjustment module 805 is configured to adjust the pixel value of the pixel corresponding to the first lane line probability map in the target probability map to a preset pixel value corresponding to the first lane line probability map. Wherein, the first lane line probability map is any lane line probability map of the N lane line probability maps, and the pixel corresponding to the first lane line probability map is the probability of the first lane line probability map The graph constitutes the pixel points of the fitted lane line.

FIG. 13 is a module structure diagram of yet another embodiment of a lane line detection device provided by an embodiment of the present disclosure. As shown in FIG. 13, the second acquisition module 802 includes: a first acquisition unit 8021, which is used to pass at least A convolution layer extracts the low-level feature information of the M channels of the road surface image. The second obtaining unit 8022 is configured to extract the high-level feature information of the M channels of the road surface image based on the M-channel low-level feature information through at least one residual extraction layer of the neural network. The third obtaining unit 8023 is configured to up-sample the high-level feature information of the M channels through at least one up-sampling layer of the neural network to obtain M probability maps equal to the road surface image.

In another embodiment, the at least one convolutional layer includes any number of contiguous 6-10 convolutional layers, and the at least one residual extraction layer includes any number of contiguous 7-12 residual extraction layers, The at least one upsampling layer includes any number of connected upsampling layers in 1-4.

In another embodiment of the present disclosure, the lane line detection device further includes: a training module (not shown in the figure), which is used to supervise and train a road training image set including lane line and / or non-lane line annotation information The neural network.

In another embodiment, the training module is configured to: input training images included in the road surface training image set to the neural network to obtain a predicted lane line probability map of the training images; according to the predicted lanes A plurality of pixels with a probability value greater than or equal to a preset threshold value included in the line probability map, fitting the predicted lane line of the training image; acquiring the predicted lane line of the training image and the lane of the training image Loss between lane lines in a line truth map, where the lane line truth map is obtained based on the labeling information of the lane lines of the training image; the network parameters of the neural network are adjusted according to the losses.

14 is a module structure diagram of another embodiment of a lane line detection device provided by an embodiment of the present disclosure. As shown in FIG. 14, it further includes: an acquisition module 806, which is used to acquire road surface images in multiple scenes, and The road surface images in the plurality of scenes are obtained by labeling lane lines as training images. Wherein, the plurality of scenes may include, but not limited to, at least two scenes in rainy scenes, foggy scenes, straight road scenes, curved road scenes, tunnel scenes, strong light scenes, and night scenes.

FIG. 15 is a module structure diagram of another embodiment of a lane line detection device provided by an embodiment of the present disclosure. As shown in FIG. 15, it further includes: a preprocessing module 807, configured to perform distortion-removing processing on the road surface image.

FIG. 16 is a module structure diagram of another embodiment of a lane line detection device provided by an embodiment of the present disclosure. As shown in FIG. 16, it further includes: a mapping module 808 for mapping the lane line in the road surface image to world coordinates In the system, the position of the lane line in the road surface image in the world coordinate system is obtained.

FIG. 17 is a physical block diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 17, the electronic device 1700 includes:

The memory 1701 is used to store program instructions.

The processor 1702 is configured to call and execute program instructions in the memory 1701 to execute the method steps described in any embodiment of the present disclosure.

FIG. 18 is a schematic flowchart of a driving control method provided by an embodiment of the present disclosure. Based on the foregoing embodiment, an embodiment of the present disclosure also provides a driving control method, including:

S1801: The driving control device acquires the lane line detection result of the road surface image.

In an optional example, the S1801 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the obtaining module 1901 executed by the processor.

S1802. The driving control device outputs prompt information according to the lane line detection result and / or performs intelligent driving control on the vehicle.

In an optional example, the S201 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the driving control module 1902 executed by the processor.

The execution subject of this embodiment is a driving control device. The driving control device of this embodiment and the electronic device described in the foregoing embodiment may be located in the same device, or may be separate devices in different devices. Among them, the driving control device of this embodiment is communicatively connected with the above-mentioned electronic device.

Wherein, the detection result of the lane line of the road surface image is obtained by the detection method of the lane line of the above embodiment, and the specific process refers to the description of the above embodiment, which will not be repeated here.

Optionally, the electronic device executes the above lane line detection method, obtains the lane line detection result of the road surface image, and outputs the lane line detection result of the road surface image. The driving control device acquires the lane line detection result of the road surface image, and outputs prompt information and / or performs intelligent driving control on the vehicle according to the lane line detection result of the road surface image.

The prompt information may include a warning warning of lane line departure, or a reminder of keeping lane line.

The intelligent driving in this embodiment includes assisted driving and / or automatic driving.

The above-mentioned intelligent driving control may include: braking, changing the driving speed, changing the driving direction, keeping lane lines, changing the state of the lights, driving mode switching, etc., wherein the driving mode switching may be switching between assisted driving and automatic driving, for example To switch from assisted driving to automatic driving.

In the driving control method provided in this embodiment, the driving control device obtains the lane line detection result of the road surface image, and outputs prompt information and / or performs intelligent driving control on the vehicle according to the lane line detection result of the road surface image, thereby improving the intelligent driving Safety and reliability.

FIG. 19 is a schematic structural diagram of a driving control device provided by an embodiment of the present disclosure. Based on the foregoing embodiment, the driving control device 1900 of the embodiment of the present disclosure includes:

The obtaining module 1901 is used to obtain a lane line detection result of a road surface image. The lane line detection result of the road surface image is obtained by using the lane line detection method as in any of the above embodiments;

The driving control module 1902 is configured to output prompt information according to the lane line detection result and / or perform intelligent driving control on the vehicle.

The driving control device of the embodiment of the present disclosure may be used to execute the technical solutions of the above-described method embodiments, and its implementation principles and technical effects are similar, and will not be repeated here.

FIG. 20 is a schematic diagram of an intelligent driving system provided by an embodiment of the present disclosure. As shown in FIG. 20, the intelligent driving system 2000 of this embodiment includes: a communication-connected camera 2001, an electronic device 1700, and a driving control device 1900, wherein the electronic device 1700 As shown in FIG. 17, the driving control device 1900 is shown in FIG. 19, and the camera 2001 is used to capture a road surface image.

Optionally, as shown in FIG. 20, in actual use, the camera 2001 captures the road surface image and sends the road surface image to the electronic device 1700. After receiving the road surface image, the electronic device 1700 performs the road surface image detection according to the above lane line detection method. Processing to obtain the lane detection result of the road surface image. Next, the electronic device 1700 sends the obtained lane line detection result of the road surface image to the driving control device 1900, and the driving control device 1900 outputs prompt information and / or performs intelligent driving control on the vehicle according to the lane line detection result of the road surface image.

21 is a schematic structural diagram of an application embodiment of an electronic device of the present disclosure. 21, which shows a schematic structural diagram of an electronic device suitable for implementing a terminal device or a server of an embodiment of the present disclosure. As shown in FIG. 21, the electronic device includes one or more processors, a communication section, etc. The one or more processors are, for example, one or more central processing units (CPUs) 2101, and / or one or more An image processor (GPU) 2113, etc. The processor can execute various instructions according to the executable instructions stored in the read only memory (ROM) 2102 or the executable instructions loaded from the storage section 2108 into the random access memory (RAM) 2103 Appropriate actions and handling. The communication part 2112 may include but is not limited to a network card, and the network card may include but not limited to an IB (Infiniband) network card. The processor may communicate with the read-only memory 2102 and / or the random access memory 2103 to execute executable instructions through the bus 2104 It is connected to the communication unit 2112 and communicates with other target devices via the communication unit 2112 to complete the operation corresponding to any lane line detection method or any driving control method provided by the embodiments of the present disclosure.

In addition, in RAM 2103, various programs and data necessary for device operation can also be stored. The CPU 2101, ROM 2102, and RAM 2103 are connected to each other via a bus 2104. In the case of RAM 2103, ROM 2102 is an optional module. The RAM 2103 stores executable instructions or writes executable instructions to the ROM 2102 at runtime. The executable instructions cause the processor 2101 to perform operations corresponding to the lane detection method or the driving control method provided in any of the foregoing embodiments. An input / output (I / O) interface 2105 is also connected to the bus 2104. The communication part 2112 may be integratedly provided, or may be provided with multiple sub-modules (for example, multiple IB network cards), and are on the bus link.

The following components are connected to the I / O interface 2105: an input section 2106 including a keyboard, a mouse, etc .; an output section 2107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 2108 including a hard disk, etc. ; And a communication section 2109 including a network interface card such as a LAN card, a modem, etc. The communication section 2109 performs communication processing via a network such as the Internet. The driver 2110 is also connected to the I / O interface 2105 as needed. A removable medium 2111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 2110 as necessary, so that the computer program read out therefrom is installed into the storage portion 2108 as needed.

It should be noted that the architecture shown in FIG. 21 is only an optional implementation method. In practice, the number and type of components in FIG. 21 can be selected, deleted, added, or replaced according to actual needs; For the setting of functional components, separate or integrated settings can also be adopted. For example, the GPU and the CPU can be set separately or the GPU can be integrated on the CPU. The communication department can be set separately or on the CPU or GPU Wait. These alternative embodiments all fall within the protection scope of the present disclosure.

In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product including a computer program tangibly contained on a machine-readable medium, the computer program containing program code for performing the method shown in the flowchart, the program code may include a corresponding The instruction corresponding to the lane line detection method or the driving control method provided by any embodiment of the present disclosure is executed.

Any method provided by the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server. Alternatively, any method provided by the embodiments of the present disclosure may be executed by the processor, for example, the processor executes any method mentioned by the embodiments of the present disclosure by calling corresponding instructions stored in the memory. The embodiments of the present disclosure will not be repeated here.

Persons of ordinary skill in the art may understand that all or part of the steps of the foregoing method embodiments may be completed by a program instructing relevant hardware. The aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the steps including the foregoing method embodiments are executed; and the foregoing storage medium includes various media that can store program codes, such as ROM, RAM, magnetic disk, or optical disk.

The method and apparatus of the embodiments of the present disclosure may be implemented in many ways. For example, the method and apparatus of the embodiments of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above sequence of steps for the method is for illustration only, and the steps of the method of the embodiments of the present disclosure are not limited to the above-described sequence unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, and these programs include machine-readable instructions for implementing the method according to an embodiment of the present disclosure. Thus, the present disclosure also covers the recording medium storing the program for executing the method according to the embodiment of the present disclosure.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, but not to limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features thereof may be equivalently replaced; and these modifications or replacements do not deviate from the essence of the corresponding technical solutions of the technical solutions of the embodiments of the present disclosure range.

Claims (29)

1. A lane line detection method, characterized in that it includes:

   Obtain the road surface image collected by the on-board equipment installed on the vehicle;

   The road surface image is input to a neural network, and M probability maps corresponding to the road surface image are output through the neural network. The M probability maps include N lane line probability maps and MN non-lane line probability maps. The N lane line probability maps respectively correspond to N lane lines on the road surface, and are used to represent the probability that pixels in the road surface image belong to the corresponding lane line; the MN non-lane line probability maps correspond to the road surface The non-lane line of is used to represent the probability that the pixels in the road surface image belong to the non-lane line, where N is a positive integer and M is an integer greater than N;

   The lane line in the road surface image is determined according to the lane line probability map.
2. The method according to claim 1, wherein the determining the lane line in the road surface image according to the lane line probability map includes:

   In response to the Lth lane line probability map including a plurality of pixels with a probability value greater than or equal to a preset threshold, the Lth lane line is fitted according to a plurality of pixels with a probability value greater than or equal to the preset threshold, wherein, the The Lth lane line probability map is any lane line probability map of the N lane line probability maps.
3. The method according to claim 1 or 2, wherein the determining the lane line in the road surface image according to the lane line probability map includes:

   In response to the first pixel point corresponding to multiple probability values in multiple lane line probability maps being greater than or equal to a preset threshold, the first pixel point is used as the pixel point when fitting the first lane line, wherein, the The first lane line is the lane line corresponding to the lane line probability map corresponding to the largest probability value among the multiple probability values.
4. The method according to any one of claims 1 to 3, wherein the determining the lane line in the road surface image according to the lane line probability map includes:

   In response to the S-th non-lane line probability map including a plurality of pixels with a probability value greater than or equal to a preset threshold, a non-lane line is determined according to a plurality of pixels with a probability value greater than or equal to a preset threshold, where the S The non-lane line probability maps are any non-lane line probability maps in the MN non-lane line probability maps.
5. The method according to any one of claims 1-4, further comprising:

   Performing fusion processing on the M probability maps to obtain a target probability map;

   Adjusting pixel values of pixels corresponding to the first lane line probability map in the target probability map to preset pixel values corresponding to the first lane line probability map;

   Wherein, the first lane line probability map is any lane line probability map of the N lane line probability maps, and the pixel corresponding to the first lane line probability map is the probability of the first lane line probability map The graph constitutes the pixel points of the fitted lane line.
6. The method according to any one of claims 1-5, wherein the outputting the M probability maps corresponding to the road surface image via the neural network includes:

   Extracting low-level feature information of the M channels of the road surface image through at least one convolutional layer of the neural network;

   Extracting high-level feature information of the M channels of the road surface image based on the M-channel low-level feature information through at least one residual extraction layer of the neural network;

   Up-sampling the high-level feature information of the M channels through at least one up-sampling layer of the neural network to obtain M probability maps equal to the road surface image.
7. The method according to claim 6, wherein the at least one convolutional layer includes any number of contiguous 6-10 convolutional layers, and the at least one residual extraction layer includes any number of contiguous 7-12 A number of residual extraction layers, the at least one up-sampling layer includes any number of connected up-sampling layers in 1-4.
8. The method according to any one of claims 1-7, wherein the neural network is obtained by supervising training using a road surface training image set including lane line and / or non-lane line annotation information.
9. The method according to claim 8, wherein the neural network is supervised and trained using a road surface training image set including lane line and / or non-lane line annotation information, including:

   Input training images included in the road training image set to the neural network, and obtain a predicted lane line probability map of the training images;

   Fitting the predicted lane line of the training image according to a plurality of pixel points with a probability value greater than or equal to a preset threshold value included in the predicted lane line probability map;

   Obtaining the loss between the predicted lane line of the training image and the lane line in the lane line truth map of the training image, wherein the lane line truth map is based on the lane line of the training image Obtain information for labeling;

   Adjust the network parameters of the neural network according to the loss.
10. The method according to claim 8 or 9, characterized in that, before obtaining the neural network by supervising training using a road surface training image set including lane line and / or non-lane line annotation information, the method further includes:

    Collect road images under multiple scenes;

    Use the image obtained by labeling the road surface in the multiple scenes as a training image;

    Wherein, the plurality of scenes include at least two scenes of daytime scenes, rainy scenes, foggy scenes, straight road scenes, curved road scenes, tunnel scenes, strong light scenes, and night scenes.
11. The method according to any one of claims 1-10, wherein before the inputting the road surface image to the neural network, the method further comprises:

    De-distortion processing is performed on the road surface image.
12. The method according to any one of claims 1-11, further comprising:

    The lane line in the road surface image is mapped into the world coordinate system to obtain the position of the lane line in the road surface image in the world coordinate system.
13. A lane line detection device, characterized in that it includes:

    The first obtaining module is used to obtain the road surface image collected by the vehicle-mounted equipment installed on the vehicle;

    The second acquisition module is used to input the road surface image into a neural network, and output M probability maps corresponding to the road surface image via the neural network, the M probability maps include N lane line probability maps and MN Non-lane line probability map, the N lane-line probability maps respectively correspond to N lane lines on the road surface, and are used to represent the probability that pixels in the road surface image belong to the corresponding lane line; the MN non-lane lines The probability map corresponds to the non-lane line on the road surface and is used to represent the probability that the pixel points in the road surface image belong to the non-lane line, where N is a positive integer and M is an integer greater than N;

    The first determining module is configured to determine the lane line in the road surface image according to the lane line probability map.
14. The apparatus according to claim 13, wherein the first determining module comprises:

    The first determining unit is configured to fit the Lth item according to a plurality of pixels with a probability value greater than or equal to a preset threshold when the Lth lane line probability map includes a plurality of pixels with a probability value greater than or equal to a preset threshold A lane line, wherein the Lth lane line probability map is any lane line probability map of the N lane line probability maps.
15. The apparatus according to claim 13 or 14, wherein the first determining module further comprises:

    The second determining unit is configured to use the first pixel point as the first fitting point when the first pixel point corresponds to a plurality of probability values in multiple lane line probability maps that are greater than or equal to a preset threshold. Pixel points, wherein the first lane line is the lane line corresponding to the lane line probability map corresponding to the largest probability value among the multiple probability values.
16. The apparatus according to any one of claims 13-15, wherein the first determining module further comprises:

    The third determining unit is configured to determine the non-lane line according to the plurality of pixels with a probability value greater than or equal to the preset threshold when the S-th non-lane line probability map includes multiple pixels with a probability value greater than or equal to the preset threshold , Wherein the Sth non-lane line probability map is any non-lane line probability map of the MN non-lane line probability maps.
17. The device according to any one of claims 13-16, further comprising:

    A fusion module, configured to fuse the M probability maps to obtain a target probability map;

    An adjustment module, configured to adjust the pixel value of the pixel corresponding to the first lane line probability map in the target probability map to a preset pixel value corresponding to the first lane line probability map;

    Wherein, the first lane line probability map is any lane line probability map of the N lane line probability maps, and the pixel corresponding to the first lane line probability map is the probability of the first lane line probability map The graph constitutes the pixel points of the fitted lane line.
18. The apparatus according to any one of claims 13-17, wherein the second acquisition module includes:

    A first acquiring unit, configured to extract low-level feature information of the M channels of the road surface image through at least one convolutional layer of the neural network;

    A second acquiring unit, configured to extract high-level feature information of the M channels of the road surface image based on the M-channel low-level feature information through at least one residual extraction layer of the neural network;

    The third acquiring unit is configured to up-sample the high-level feature information of the M channels through at least one up-sampling layer of the neural network to obtain M probability maps equal to the road surface image.
19. The apparatus according to claim 18, wherein the at least one convolutional layer includes any number of contiguous 6-10 convolutional layers, and the at least one residual extraction layer includes any one of 7-12 contiguously connected A number of residual extraction layers, the at least one up-sampling layer includes any number of connected up-sampling layers in 1-4.
20. The device according to any one of claims 13-19, further comprising:

    The training module is used to supervise and train the neural network by using a road surface training image set including lane line and / or non-lane line annotation information.
21. The apparatus according to claim 20, wherein the training module is configured to:

    Input training images included in the road training image set to the neural network, and obtain a predicted lane line probability map of the training images;

    Fitting the predicted lane line of the training image according to a plurality of pixel points with a probability value greater than or equal to a preset threshold value included in the predicted lane line probability map;

    Obtaining the loss between the predicted lane line of the training image and the lane line in the lane line truth map of the training image, wherein the lane line truth map is based on the lane line of the training image Obtain information for labeling;

    Adjust the network parameters of the neural network according to the loss.
22. The device according to claim 20 or 21, further comprising:

    The collection module is used for collecting road surface images in multiple scenes, and the images obtained by labeling the road surface images in the multiple scenes as training images;

    Wherein, the multiple scenes include at least two scenes of rainy scenes, foggy scenes, straight road scenes, curved road scenes, tunnel scenes, strong light scenes, and night scenes.
23. The device according to any one of claims 13-22, further comprising:

    The pre-processing module is used for de-distorting the road surface image.
24. The device according to any one of claims 13-23, further comprising:

    The mapping module is used to map the lane line in the road surface image to the world coordinate system, and obtain the position of the lane line in the road surface image in the world coordinate system.
25. A driving control method, characterized in that it includes:

    The driving control device acquires the lane line detection result of the road surface image, and the lane line detection result of the road surface image is obtained by using the lane line detection method according to any one of claims 1-12;

    The driving control device outputs prompt information according to the lane line detection result and / or performs intelligent driving control on the vehicle.
26. A driving control device is characterized by comprising:

    An acquisition module, for acquiring a lane line detection result of a road surface image, the lane line detection result of the road surface image is obtained by using the lane line detection method according to any one of claims 1-12;

    The driving control module is configured to output prompt information according to the detection result of the lane line and / or perform intelligent driving control on the vehicle.
27. An electronic device, characterized in that it includes:

    Memory, used to store program instructions;

    The processor is configured to call and execute the program instructions in the memory and execute the method steps of any one of claims 1-12.
28. An intelligent driving system is characterized by comprising: a camera connected in communication, the electronic device according to claim 27 and the driving control device according to claim 26, the camera is used to acquire a road surface image.
29. A readable storage medium, characterized in that a computer program is stored in the readable storage medium, and the computer program is used to perform the method steps of any one of claims 1-12.

Images