CN115063762A

CN115063762A - Method, device and equipment for detecting lane line and storage medium

Info

Publication number: CN115063762A
Application number: CN202210556360.4A
Authority: CN
Inventors: 郭湘; 孙鹏; 韩旭
Original assignee: Guangzhou Weride Technology Co Ltd
Current assignee: Guangzhou Weride Technology Co Ltd
Priority date: 2022-05-20
Filing date: 2022-05-20
Publication date: 2022-09-16

Abstract

The invention relates to the technical field of automatic driving, and discloses a method, a device, equipment and a storage medium for detecting a lane line, which are used for improving the integrity of lane line detection. The method for detecting the lane line comprises the following steps: acquiring an original image shot by at least one angle camera, and extracting image characteristics of the original image to obtain target image characteristic information; determining a plurality of first sampling points, and performing characteristic collection of each first sampling point through target image characteristic information to obtain characteristic information of the sampling point corresponding to each first sampling point; predicting lane line relevance scores of the characteristic information of the sampling points corresponding to each first sampling point to obtain the lane line relevance scores corresponding to each first sampling point, and determining a plurality of second sampling points in the plurality of first sampling points according to the lane line relevance scores corresponding to each first sampling point; and predicting the lane line through the characteristic information of the sampling point corresponding to each second sampling point to obtain the information of the target lane line.

Description

Method, device and equipment for detecting lane line and storage medium

Technical Field

The invention relates to the technical field of automatic driving, in particular to a method, a device, equipment and a storage medium for detecting lane lines.

Background

With the development of the automatic driving technology, the environment perception capability of the automatic driving vehicle is stronger and stronger, wherein the lane line detection is an important visual perception task in the automatic driving technology.

The existing lane line detection technology generally detects lane line information in an image, and then performs depth estimation on the image to generate 3d lane line information, but because the method detects a single image, and the lane line information detected through the image is scattered and unrelated under the condition that a lane line is blocked, the technical problem of incomplete lane line detection in the prior art is seen.

Disclosure of Invention

The invention provides a method, a device, equipment and a storage medium for detecting a lane line, which are used for improving the integrity of the detection of the lane line.

The invention provides a method for detecting a lane line in a first aspect, which comprises the following steps:

acquiring an original image shot by at least one angle camera, and extracting image characteristics of the original image to obtain target image characteristic information;

determining a plurality of first sampling points, and performing feature collection of each first sampling point through the target image feature information to obtain feature information of the sampling point corresponding to each first sampling point;

performing lane line correlation score prediction on the characteristic information of the sampling point corresponding to each first sampling point to obtain a lane line correlation score corresponding to each first sampling point, and determining a plurality of second sampling points in the plurality of first sampling points according to the lane line correlation score corresponding to each first sampling point;

and predicting the lane line through the characteristic information of the sampling point corresponding to each second sampling point to obtain the information of the target lane line.

Optionally, the determining a plurality of first sampling points, and performing feature collection of each first sampling point through the target image feature information to obtain feature information of a sampling point corresponding to each first sampling point includes:

constructing a 3d space with a target reference object as an origin, and selecting sampling points from the 3d space to obtain a plurality of first sampling points, wherein each first sampling point comprises 3d space position information of the sampling point;

projecting each first sampling point to the original image according to the external parameters between the target reference object and the cameras at all angles to obtain image coordinate information of each first sampling point in the original image;

and collecting the target image characteristic information and the 3d spatial position information of each first sampling point based on the image coordinate information of each first sampling point in the original image to obtain the characteristic information of the sampling point corresponding to each first sampling point.

Optionally, the constructing a 3d space with a target reference object as an origin, and selecting a sampling point from the 3d space to obtain a plurality of first sampling points includes:

constructing a 3d space with a target reference object as an origin, and emitting rays to the 3d space according to a preset offset angle by taking the origin as a starting point to obtain a plurality of target rays;

and selecting a plurality of 3d space points from each item marking line according to a preset sampling point selection strategy to obtain a plurality of first sampling points.

Optionally, the collecting, based on image coordinate information of each first sampling point in the original image, the target image feature information and the 3d spatial position information for each first sampling point to obtain the feature information of the sampling point corresponding to each first sampling point includes:

collecting image characteristic information of image coordinate information of each first sampling point in the original image for the target image characteristic information through a preset collecting operator to obtain first image characteristic information corresponding to each first sampling point;

and generating position characteristic information corresponding to each first sampling point according to the 3d space position information, and superposing the first image characteristic information and the position characteristic information corresponding to each first sampling point to obtain sampling point characteristic information corresponding to each first sampling point.

Optionally, the obtaining an original image captured by at least one angle camera and performing image feature extraction on the original image to obtain target image feature information includes:

the method comprises the steps of obtaining an original image shot by at least one angle camera, and carrying out multi-size image feature extraction on the original image shot by each angle camera through a first convolution neural network and a feature pyramid network in a preset lane line detection model to obtain target image feature information.

Optionally, the predicting the lane line relevance score of the characteristic information of the sampling point corresponding to each first sampling point to obtain the lane line relevance score corresponding to each first sampling point, and determining a plurality of second sampling points in the plurality of first sampling points according to the lane line relevance score corresponding to each first sampling point, includes:

carrying out information fusion on the characteristic information of the sampling point corresponding to each first sampling point through a second convolutional neural network in a preset lane line detection model to obtain the fusion characteristic information corresponding to each first sampling point;

calculating the relevance score of the lane line of the fusion characteristic information corresponding to each first sampling point to obtain the relevance score of the lane line corresponding to each first sampling point;

and sequencing the lane line relevance scores corresponding to each first sampling point to obtain a plurality of second sampling points with the largest lane line relevance scores in the plurality of first sampling points.

Optionally, the predicting the lane line through the characteristic information of the sampling point corresponding to each second sampling point to obtain the information of the target lane line includes:

performing cross attention system lane line information query on the sampling point characteristic information corresponding to each second sampling point through a lane line query tensor in a preset lane line detection model to obtain at least one first lane line information;

and performing lane line information fusion on the at least one first lane line information through a self-attention mechanism to obtain target lane line information, wherein the target lane line information comprises lane line type information and lane line position information of at least one lane line.

Optionally, before the obtaining an original image captured by at least one angle camera and performing image feature extraction on the original image to obtain target image feature information, the method for detecting a lane line further includes:

obtaining a sample image with lane line marking information, and carrying out lane line detection on the sample image through a preset end-to-end neural network model to obtain predicted lane line information, wherein the end-to-end neural network model comprises a first convolutional neural network, a characteristic pyramid network, a second convolutional neural network and a lane line query tensor;

calculating loss values of the predicted lane line information and the lane line marking information through a preset loss function to obtain target loss values, wherein the target loss values comprise lane line type loss values and lane line position loss values;

and generating a lane line detection model or adjusting parameters of a first convolutional neural network, a characteristic pyramid network, a second convolutional neural network and a lane line query tensor in the end-to-end neural network model according to the target loss value.

Optionally, the calculating a loss value of the predicted lane line information and the lane line marking information by using a preset loss function to obtain a target loss value includes:

performing lane line information matching on at least one piece of lane line information in the predicted lane line information and at least one piece of lane line information in the lane line marking information to obtain a plurality of lane line information matching results;

and calculating a loss value of each lane line information matching result through a preset loss function to obtain a first loss value corresponding to each lane line information matching result, and determining a first loss value with the minimum loss value in all the first loss values as a target loss value.

A second aspect of the present invention provides a lane line detection apparatus, including:

the system comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring an original image shot by at least one angle camera and extracting image characteristics of the original image to obtain target image characteristic information;

the collection module is used for determining a plurality of first sampling points and performing characteristic collection of each first sampling point according to the target image characteristic information to obtain the characteristic information of the sampling point corresponding to each first sampling point;

the scoring module is used for predicting the lane line relevance score of the characteristic information of the sampling point corresponding to each first sampling point to obtain the lane line relevance score corresponding to each first sampling point, and determining a plurality of second sampling points in the plurality of first sampling points according to the lane line relevance score corresponding to each first sampling point;

and the prediction module is used for predicting the lane line through the characteristic information of the sampling point corresponding to each second sampling point to obtain the information of the target lane line.

Optionally, the collecting module includes:

the device comprises a construction unit, a data acquisition unit and a data processing unit, wherein the construction unit is used for constructing a 3d space with a target reference object as an origin, selecting sampling points from the 3d space and obtaining a plurality of first sampling points, and each first sampling point comprises 3d space position information of the sampling point;

the projection unit is used for projecting each first sampling point to the original image according to external parameters between the target reference object and each angle camera to obtain image coordinate information of each first sampling point in the original image;

and the collecting unit is used for collecting the target image characteristic information and the 3d space position information of each first sampling point based on the image coordinate information of each first sampling point in the original image to obtain the characteristic information of the sampling point corresponding to each first sampling point.

Optionally, the building unit is specifically configured to:

Optionally, the collecting unit is specifically configured to:

and generating position characteristic information corresponding to each first sampling point through the 3d space position information, and superposing the first image characteristic information and the position characteristic information corresponding to each first sampling point to obtain the sampling point characteristic information corresponding to each first sampling point.

Optionally, the obtaining module is specifically configured to:

Optionally, the scoring module is specifically configured to:

Optionally, the prediction module is specifically configured to:

Optionally, the detection device for the lane line further includes:

the system comprises a sample detection module, a prediction module and a data processing module, wherein the sample detection module is used for obtaining a sample image with lane line marking information and carrying out lane line detection on the sample image through a preset end-to-end neural network model to obtain predicted lane line information, and the end-to-end neural network model comprises a first convolutional neural network, a characteristic pyramid network, a second convolutional neural network and a lane line query tensor;

the loss calculation module is used for calculating the loss value of the predicted lane line information and the lane line marking information through a preset loss function to obtain a target loss value, wherein the target loss value comprises a lane line type loss value and a lane line position loss value;

and the model generation module is used for generating a lane line detection model or adjusting parameters of a first convolution neural network, a characteristic pyramid network, a second convolution neural network and a lane line query tensor in the end-to-end neural network model according to the target loss value.

Optionally, the loss calculating module is specifically configured to:

and calculating the loss value of each lane line information matching result through a preset loss function to obtain a first loss value corresponding to each lane line information matching result, and determining the first loss value with the minimum loss value in all the first loss values as a target loss value.

A third aspect of the present invention provides a lane line detection apparatus, including: a memory and at least one processor, the memory having stored therein a computer program; the at least one processor calls the computer program in the memory to cause the lane line detection apparatus to execute the above-described lane line detection method.

A fourth aspect of the present invention provides a computer-readable storage medium having stored therein a computer program which, when run on a computer, causes the computer to execute the above-described lane line detection method.

In the technical scheme provided by the invention, an original image shot by at least one angle camera is obtained, and image feature extraction is carried out on the original image to obtain target image feature information; determining a plurality of first sampling points, and performing characteristic collection of each first sampling point through target image characteristic information to obtain characteristic information of the sampling point corresponding to each first sampling point; predicting lane line relevance scores of the characteristic information of the sampling points corresponding to each first sampling point to obtain the lane line relevance scores corresponding to each first sampling point, and determining a plurality of second sampling points in the plurality of first sampling points according to the lane line relevance scores corresponding to each first sampling point; and predicting the lane line through the characteristic information of the sampling point corresponding to each second sampling point to obtain the information of the target lane line. In the embodiment of the invention, the characteristic information in the original image shot by the camera at different angles is firstly extracted, a plurality of adopting points are defined for collecting the characteristic information corresponding to each sampling point, including the image characteristic information, so that the characteristic information for detecting the lane line does not need to completely depend on the image characteristics, sampling spaces with different requirements can be constructed, the prediction of the lane line correlation score is carried out based on the characteristic information collected by each sampling point, the sampling point with high lane line correlation degree is obtained, the accuracy and the integrity of the lane line prediction are improved through the sampling point with high lane line correlation degree, and the integrity of the lane line detection is improved.

Drawings

Fig. 1 is a schematic diagram of an embodiment of a lane line detection method according to an embodiment of the present invention;

fig. 2 is a schematic diagram of another embodiment of the lane line detection method according to the embodiment of the present invention;

FIG. 3 is a schematic diagram of an embodiment of a lane marking detection apparatus according to an embodiment of the present invention;

fig. 4 is a schematic view of another embodiment of the lane line detection apparatus according to the embodiment of the present invention;

fig. 5 is a schematic diagram of an embodiment of the detection device for the lane line in the embodiment of the present invention.

Detailed Description

The embodiment of the invention provides a method, a device, equipment and a storage medium for detecting a lane line, which are used for improving the integrity of the detection of the lane line.

The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

It is understood that the executing subject of the present invention may be a lane line detection device, and may also be a terminal or a server, and the terminal may be an automatic driving terminal, which is not limited herein. The embodiment of the present invention is described by taking a terminal as an execution subject.

For convenience of understanding, a specific flow of the embodiment of the present invention is described below, and referring to fig. 1, an embodiment of the method for detecting a lane line in the embodiment of the present invention includes:

101. acquiring an original image shot by at least one angle camera, and extracting image characteristics of the original image to obtain target image characteristic information;

it should be noted that, in the conventional lane line detection method, the pixel point detection of the lane line is generally performed directly on a single image, and this method usually requires training an artificial intelligence model through an image with lane line labeling information to generate a model for detecting the pixel point of the lane line, but even if the pixel point detection of the lane line is performed on images at multiple angles, the lane line information obtained through this method is scattered and the lane line information between different images is difficult to associate, and distortion is also easy to occur in the association process, so that it is difficult to obtain the overall perception information of the automatically driven vehicle through this lane line information, and further, the accuracy of the automatic driving control is reduced, and the safety is also reduced accordingly. Therefore, the method and the device are used for obtaining the complete lane line information in the overall perception environment from end to end, and even if the lane line is shielded in the image, the lane line information of the shielded part can still be predicted, so that the integrity of the lane line detection is improved.

In this embodiment, in order to improve the integrity of lane marking detection, the terminal acquires raw images captured by at least one angle camera, and in one embodiment, the terminal acquires raw images captured by at least one angle camera through cameras installed at different positions of the autonomous vehicle, for example, cameras installed at 4 vertexes of the autonomous vehicle are used to acquire raw images captured by 4 angle cameras. In an embodiment, the terminal performs image feature extraction on an original image shot by each angle camera through an image feature extraction network in a preset lane line detection model to obtain target image feature information, where the image feature extraction network may be a neural network with any structure, such as a convolutional neural network, a residual neural network, and the like, and is not limited herein.

102. Determining a plurality of first sampling points, and performing characteristic collection of each first sampling point through target image characteristic information to obtain characteristic information of the sampling point corresponding to each first sampling point;

it can be understood that, different from the conventional lane line detection technology that lane line information is obtained by directly classifying the target image characteristic information, the target image characteristic information is used for collecting the image characteristics of each first sampling point in the overall perception environment, in one embodiment, each first sampling point is a 3d spatial coordinate point, and the determination of a plurality of first sampling points is independent of the coordinates of each pixel point in the original image and is related to the 3d space where the automatic driving vehicle is located, so that the characteristic information of the image in the 3d space can be obtained without performing depth estimation on the target characteristic information of the original image, and the integrity of lane line detection is improved. The embodiment can also reduce the data processing amount in a sampling mode, thereby improving the efficiency of lane line detection.

In one implementation mode, the terminal takes the automatic driving vehicle as an origin, a 3d vehicle coordinate system of the automatic driving vehicle is constructed, a plurality of sampling points are extracted from the 3d vehicle coordinate system, a plurality of first sampling points are obtained, and then image feature information corresponding to each first sampling point is extracted from the target image feature information according to the relative position relation (namely external parameters) between the 3d vehicle coordinate system and the angle camera coordinate systems, so that the sampling point feature information corresponding to each first sampling point is obtained. According to the embodiment, the accuracy of the 3d space image characteristic information can be obtained, so that the accuracy and the integrity of the 3d space perception of the lane line are improved.

In one embodiment, the characteristic information of the sampling point corresponding to each first sampling point includes, in addition to the image characteristic information collected by the camera, characteristic information collected by other sensors, such as characteristic information collected by a laser radar, a millimeter wave radar, an ultrasonic radar, and the like. Specifically, after the terminal determines a plurality of first sampling points, feature collection of each first sampling point is performed through target image feature information and a preset sensor to obtain feature information of the sampling point corresponding to each first sampling point, wherein the preset sensor is used for indicating sensors except cameras of all angles, and the preset sensor comprises at least one of a laser radar, a millimeter wave radar and an ultrasonic radar. Further, the terminal obtains image characteristic information of each first sampling point and characteristic information of the preset sensor according to relative position relations between the camera and the preset sensor of each angle and the 3d vehicle coordinate system respectively, and obtains characteristic information of the sampling point corresponding to each first sampling point. This embodiment can fuse the characteristic of different sensors and be used for lane line to detect for the accuracy and the wholeness that lane line detected improve.

103. Predicting lane line relevance scores of the characteristic information of the sampling points corresponding to each first sampling point to obtain the lane line relevance scores corresponding to each first sampling point, and determining a plurality of second sampling points in the plurality of first sampling points according to the lane line relevance scores corresponding to each first sampling point;

in an embodiment, the terminal performs lane line relevance score prediction on the characteristic information of the sampling point corresponding to each first sampling point through a preset lane line detection model to obtain a lane line relevance score corresponding to each first sampling point, and specifically, the terminal performs lane line relevance score prediction on the characteristic information of the sampling point corresponding to each first sampling point through a relevance score prediction network in the preset lane line detection model to obtain a lane line relevance score corresponding to each first sampling point, where the relevance score prediction network may be a neural network with any structure, such as a single network including a convolutional neural network and a residual neural network, or multiple networks with a connection relationship, and is not limited herein.

In one embodiment, in order to further screen out a sampling point with a high degree of lane line correlation from the plurality of first sampling points, the terminal performs score screening on a lane line correlation score corresponding to each first sampling point to obtain a plurality of second sampling points in the plurality of first sampling points, wherein the second sampling points are used for indicating the first sampling points with the lane line correlation score higher than a preset score threshold. The embodiment can further acquire the sampling points with high correlation degree of the lane lines, thereby improving the accuracy of lane line prediction.

104. And predicting the lane line through the characteristic information of the sampling point corresponding to each second sampling point to obtain the information of the target lane line.

In one implementation, the terminal performs lane line prediction on the characteristic information of the sampling point corresponding to each second sampling point based on a self-attention (self-attention) mechanism and a cross-attention (cross-attention) mechanism to obtain target lane line information, where the target lane line information includes at least one piece of lane line information. Specifically, the terminal extracts context information from the sampling point feature information corresponding to each second sampling point through a cross attention mechanism to obtain lane line feature information with the context information corresponding to each second sampling point, and then performs feature information fusion on the lane line feature information with the context information corresponding to each second sampling point through a self attention mechanism to obtain target lane line information. The embodiment can convert the characteristics of the sampling points to the characteristic vectors of the output lane lines through the cross attention mechanism and the self attention mechanism, so that the lane line detection is not only stopped on the pixel points, and the integrity of the lane line detection is improved.

In the embodiment of the invention, the characteristic information in the original image shot by the camera at different angles is firstly extracted, a plurality of adopting points are defined for collecting the characteristic information corresponding to each sampling point, including the image characteristic information, so that the characteristic information for detecting the lane line does not need to completely depend on the image characteristics, sampling spaces with different requirements can be constructed, the prediction of the lane line correlation score is carried out based on the characteristic information collected by each sampling point, the sampling point with high lane line correlation degree is obtained, the accuracy and the integrity of the lane line prediction are improved through the sampling point with high lane line correlation degree, and the integrity of the lane line detection is improved.

Referring to fig. 2, another embodiment of the method for detecting a lane line according to the embodiment of the present invention includes:

201. acquiring an original image shot by at least one angle camera, and extracting image characteristics of the original image to obtain target image characteristic information;

specifically, step 201 includes: the method comprises the steps of obtaining an original image shot by at least one angle camera, and carrying out multi-size image feature extraction on the original image shot by each angle camera through a first convolution neural network and a feature pyramid network in a preset lane line detection model to obtain target image feature information.

In this embodiment, a Convolutional Neural Network (CNN) is a type of feed-Forward Neural Network (FNN) that includes convolution calculation and has a depth structure, and a Feature Pyramid Network (FPN) is used to perform feature extraction on objects of multiple sizes, and in order to perform feature extraction on original images of different sizes and improve the feature extraction accuracy of image details in the original images, a terminal performs multi-size image generation on the original images captured by each angle camera through a feature pyramid network in a preset lane line detection model and a first convolution neural network, and performs image feature extraction on the generated multi-size images to obtain target image feature information.

Further, before step 201, the method further includes:

s1, obtaining a sample image with lane line marking information, and carrying out lane line detection on the sample image through a preset end-to-end neural network model to obtain predicted lane line information, wherein the end-to-end neural network model comprises a first convolution neural network, a characteristic pyramid network, a second convolution neural network and a lane line query tensor;

s2, calculating loss values of the predicted lane line information and the lane line marking information through a preset loss function to obtain target loss values, wherein the target loss values comprise lane line type loss values and lane line position loss values;

and S3, generating a lane line detection model or adjusting parameters of the first convolutional neural network, the characteristic pyramid network, the second convolutional neural network and the lane line query tensor in the end-to-end neural network model according to the target loss value.

The embodiment is a training process of a lane line detection model, and the terminal further trains the lane line detection model before acquiring an original image shot by at least one angle camera, so as to generate the lane line detection model after end-to-end training. Specifically, step S1 includes: the method comprises the steps that a terminal obtains a sample image shot by a multi-angle camera with lane line marking information, multi-size image feature extraction is carried out on the sample image shot by each angle camera through a first convolution neural network and a feature pyramid network in a preset untrained end-to-end neural network model to obtain sample image feature information, feature collection of each first sample point is carried out through the sample image feature information to obtain sample feature information of a sampling point corresponding to each first sample point, then lane line relevance score prediction is carried out on the sample feature information of the sampling point corresponding to each first sample point through a second convolution neural network to obtain a lane line relevance score corresponding to each first sample point, and a plurality of second sample points in the plurality of first sample points are determined through the lane line relevance scores corresponding to each first sample point, and finally, performing lane line prediction on the sampling point sample characteristic information corresponding to each second sample sampling point through the lane line query tensor to obtain predicted lane line information. It should be noted that, at the initial training of the lane line detection model, each network and tensor in the model are data initialized at random, and parameter adjustment is performed on each network and tensor in the model through a loss function guidance, so that the training of the model is completed, and the trained lane line detection model is obtained.

It should be noted that the lane line labeling information in the sample image with the lane line labeling information is used for indicating a true value of the lane line information of the sample image, the terminal calculates and predicts a loss value between the lane line information and the lane line labeling information through a preset loss function to obtain a target loss value, and determines whether the target loss value is less than or equal to a preset loss value threshold, if the target loss value is less than or equal to the preset loss value threshold, a trained lane line detection model is generated, if the target loss value is greater than the preset loss value threshold, a first convolution neural network, a feature pyramid network, a second convolution neural network, and a lane line query tensor in the end-to-end neural network model are subjected to parameter adjustment, and steps S1-S3 are repeated until the target loss value is less than or equal to the preset loss value threshold. The embodiment can train the lane line detection model from end to end, so that the lane line detection model can autonomously learn the extraction of the lane line characteristics, integrally sense the lane line information in each angle image, and improve the integrity of lane line detection.

Further, the step S2 includes: performing lane line information matching on at least one piece of lane line information in the predicted lane line information and at least one piece of lane line information in the lane line marking information to obtain a plurality of lane line information matching results; and calculating the loss value of each lane line information matching result through a preset loss function to obtain a first loss value corresponding to each lane line information matching result, and determining the first loss value with the minimum loss value in all the first loss values as a target loss value.

It should be noted that, since the predicted lane line information and the lane line labeling information both include at least one piece of unordered lane line information, before calculating the target loss value, it is necessary to match the two sets of unordered lane line information, so as to accurately calculate the loss value between the predicted lane line information and the true value of the lane line information. In one implementation mode, the terminal performs lane line information matching on at least one piece of lane line information in the predicted lane line information and at least one piece of lane line information in the lane line marking information through a preset Hungarian algorithm to obtain a plurality of lane line information matching results. For example, if the predicted lane line information includes predicted lane line a1 and predicted lane line B1, and the lane line annotation information includes lane line annotation a2 and lane line annotation B2, then by matching { a1, B1} with { a2, B2}, there may be two lane line information matching results, which are: lane line information matching result M: "A1-A2, B1-B2" and lane line information matching result N: "A1-B2, B1-A2". And the terminal calculates the loss value of each lane line information matching result through a preset loss function to obtain a first loss value corresponding to each lane line information matching result, and determines the first loss value with the minimum loss value in all the first loss values as a target loss value. As in the above example, assuming that the first loss value corresponding to the lane line information matching result M is 0.2, and the first loss value corresponding to the lane line information matching result N is 0.1, since 0.1<0.2, the target loss value is determined to be 0.1. According to the embodiment, the loss value between the predicted lane line information and the true value of the lane line information can be accurately calculated in a mode of matching the unordered lane lines, so that the training precision of the lane line detection model is improved, and the accuracy of lane line detection is further improved.

202. Constructing a 3d space with a target reference object as an origin, and selecting sampling points from the 3d space to obtain a plurality of first sampling points, wherein each first sampling point comprises 3d space position information of the sampling point;

in this embodiment, the target reference object may be a vehicle, may also be a camera at each angle, and may also be another real object or a virtual object (such as a virtual bird's-eye view camera) that is associated with the vehicle, and is not limited herein. In one embodiment, the terminal constructs a 3d space with the vehicle as an origin, and selects a sampling point from the 3d space to obtain a plurality of first sampling points. Further, the terminal constructs a 3d space with the vehicle as an origin, emits a preset-shaped line or plane in the 3d space to obtain a plurality of lines or planes, and selects a plurality of 3d space points from the plurality of lines or planes to obtain a plurality of first sampling points, wherein the preset-shaped line can be defined lines such as a circular line, a rectangular line, a directional ray, an undirected straight line and a curved line, and the preset plane can be defined planes such as a circular plane, a rectangular plane and a triangular plane. This embodiment can obtain the sampling point from the 3d space based on the sampling strategy that predetermines for the sampling mode is more nimble, and then improves the flexibility of lane line detection mode in the 3d space.

Specifically, step 202 includes: constructing a 3d space with a target reference object as an origin, and emitting rays to the 3d space according to a preset offset angle by taking the origin as a starting point to obtain a plurality of target rays; and selecting a plurality of 3d space points from each item marking line according to a preset sampling point selection strategy to obtain a plurality of first sampling points.

In this embodiment, since the presenting manner of the lane line in the image is similar to that of the ray, in order to more accurately acquire the 3d space point related to the lane line, the terminal emits the ray to the 3d space according to the preset offset angle, so as to obtain a plurality of target rays. For example, assuming that the preset offset angle is 0.2 degrees, a total of 1800 target rays may be shot, i.e., 360/0.2 ═ 1800. After obtaining a plurality of target rays, the terminal selects a plurality of 3d space points from each item target ray according to a preset sampling point selection strategy to obtain a plurality of first sampling points, wherein the preset sampling point selection strategy is used for indicating a selection standard of the sampling points, for example, a sampling point is selected every 1 meter, a sampling point is selected every 2 coordinate points, and the like, and the specific point is not limited herein. According to the method and the device, the selection precision of the sampling points can be improved based on the divergence characteristics of the lane lines, and therefore the detection precision of the lane lines in the 3d space is improved.

203. Projecting each first sampling point to the original image according to the external parameters between the target reference object and the cameras at all angles to obtain image coordinate information of each first sampling point in the original image;

in the embodiment, in order to acquire the coordinate information of each 3d sampling point in the original image, the terminal projects each first sampling point to the original image according to the external reference between the target reference object and each angle camera, so as to obtain the image coordinate information of each first sampling point in the original image, where the image coordinate information is used for indicating the image coordinate information of each first sampling point in the original image. In one embodiment, the terminal may further project each first sampling point to the preset sensor according to an external parameter between the target reference object and the preset sensor, so as to obtain coordinate information of each first sampling point at the preset sensor, where the preset sensor is used to indicate sensors other than the angle cameras. The embodiment can acquire the projection coordinates of each 3d sampling point on different sensors, so that the characteristic information of different sensors is collected through the projection coordinates, and the accuracy and integrity of lane line detection are improved.

204. Collecting target image characteristic information and 3d spatial position information of each first sampling point based on image coordinate information of each first sampling point in an original image to obtain characteristic information of the sampling point corresponding to each first sampling point;

specifically, step 204 includes: image characteristic information collection of image coordinate information of each first sampling point in the original image is carried out on the target image characteristic information through a preset collection operator, and first image characteristic information corresponding to each first sampling point is obtained; and generating position characteristic information corresponding to each first sampling point through the 3d space position information, and superposing the first image characteristic information and the position characteristic information corresponding to each first sampling point to obtain the sampling point characteristic information corresponding to each first sampling point.

In this embodiment, in order to improve the efficiency of collecting the feature information of the sampling points, the terminal collects the image feature information of the image coordinate information of each first sampling point in the original image from the target image feature information through the preset collection operator to obtain the first image feature information corresponding to each first sampling point, where the preset collection operator may be used to collect slices from the target image feature information based on the index. In one implementation mode, the terminal further extracts the features of the 3d spatial position information of the sampling points in each first sampling point to obtain the position feature information corresponding to each first sampling point, and then superposes the position feature information corresponding to each first sampling point with the first image feature information to obtain the sampling point feature information corresponding to each first sampling point, so that the feature information of each sampling point is more comprehensive and perfect, and the accuracy of lane line detection is improved.

205. Predicting lane line relevance scores of the characteristic information of the sampling points corresponding to each first sampling point to obtain the lane line relevance scores corresponding to each first sampling point, and determining a plurality of second sampling points in the plurality of first sampling points according to the lane line relevance scores corresponding to each first sampling point;

specifically, step 205 includes: carrying out information fusion on the characteristic information of the sampling point corresponding to each first sampling point through a second convolutional neural network in a preset lane line detection model to obtain the fusion characteristic information corresponding to each first sampling point; calculating the relevance score of the lane line of the fusion characteristic information corresponding to each first sampling point to obtain the relevance score of the lane line corresponding to each first sampling point; and sequencing the lane line relevance scores corresponding to each first sampling point to obtain a plurality of second sampling points with the largest lane line relevance scores in the plurality of first sampling points.

In this embodiment, the terminal performs information fusion on the sampling point feature information corresponding to each first sampling point through a second convolutional neural network in a trained lane line detection model to obtain fusion feature information corresponding to each first sampling point, and then performs lane line correlation score calculation on the fusion feature information corresponding to each first sampling point to obtain a lane line correlation score corresponding to each first sampling point. Finally, the terminal sequences the lane line relevance scores corresponding to each first sampling point to obtain a plurality of second sampling points with the largest lane line relevance scores in the plurality of first sampling points, and it can be understood that the lane line relevance scores of each second sampling point are the largest and the same, and are the sampling points with the highest lane line relevance degree in the plurality of first sampling points.

206. And predicting the lane line through the characteristic information of the sampling point corresponding to each second sampling point to obtain the information of the target lane line.

Specifically, step 206 includes: performing cross attention system lane line information query on the sampling point characteristic information corresponding to each second sampling point through a lane line query tensor in a preset lane line detection model to obtain at least one first lane line information; and performing lane line information fusion on the at least one first lane line information through a self-attention mechanism to obtain target lane line information, wherein the target lane line information comprises lane line type information and lane line position information of the at least one lane line.

In this embodiment, the lane line query tensor can be understood as a query value in the lane line detection model, the trained lane line detection model includes a plurality of lane line query tensors, and the lane line information query can be performed on the characteristic information of the sampling point corresponding to each second sampling point based on the cross attention mechanism to obtain at least one piece of first lane line information.

In the present embodiment, the target lane line information includes lane line type information of at least one lane line and lane line position information, for example, the lane line type information may be in the category of white dotted line, white solid line, yellow dotted line, etc., and the lane line position information includes start position coordinates, end position coordinates, and position coordinates of a plurality of intermediate lane lines of the lane line, which represent relative position information of the lane line with respect to the vehicle in the 3d space.

In the embodiment of the invention, the characteristic information in the original image shot by the camera at different angles is firstly extracted, a plurality of 3d adopting points are defined for collecting the characteristic information corresponding to each sampling point, including the image characteristic information, so that the characteristic information for detecting the lane line does not need to completely depend on the image characteristics, 3d sampling spaces with different requirements can be constructed, the lane line correlation score is predicted based on the characteristic information collected by each sampling point, the sampling point with high lane line correlation degree is obtained, the accuracy and integrity of the prediction of the lane line in the 3d sampling space are improved through the sampling point with high lane line correlation degree, and the integrity of the lane line detection is improved.

In the above description of the method for detecting a lane line in the embodiment of the present invention, referring to fig. 3, a device for detecting a lane line in the embodiment of the present invention is described below, where one embodiment of the device for detecting a lane line in the embodiment of the present invention includes:

an obtaining module 301, configured to obtain an original image captured by at least one angle camera, and perform image feature extraction on the original image to obtain target image feature information;

the collecting module 302 is configured to determine a plurality of first sampling points, and perform feature collection on each first sampling point according to the target image feature information to obtain feature information of the sampling point corresponding to each first sampling point;

the scoring module 303 is configured to predict lane line relevance scores of the characteristic information of the sampling points corresponding to each first sampling point to obtain the lane line relevance score corresponding to each first sampling point, and determine a plurality of second sampling points in the plurality of first sampling points according to the lane line relevance scores corresponding to each first sampling point;

and the prediction module 304 is configured to perform lane line prediction according to the characteristic information of the sampling point corresponding to each second sampling point to obtain target lane line information.

Referring to fig. 4, another embodiment of the lane line detection apparatus according to the embodiment of the present invention includes:

Optionally, the collecting module 302 includes:

the construction unit 3021 is configured to construct a 3d space with a target reference object as an origin, and select a sampling point from the 3d space to obtain a plurality of first sampling points, where each first sampling point includes 3d space position information of the sampling point;

the projection unit 3022 is configured to project each first sampling point to the original image according to external parameters between the target reference object and each angle camera, so as to obtain image coordinate information of each first sampling point in the original image;

the collecting unit 3023 is configured to collect target image feature information and 3d spatial position information for each first sampling point based on image coordinate information of each first sampling point in the original image, and obtain feature information of the sampling point corresponding to each first sampling point.

Optionally, the building unit 3021 is specifically configured to:

Optionally, the collecting unit 3023 is specifically configured to:

Optionally, the obtaining module 301 is specifically configured to:

Optionally, the scoring module 303 is specifically configured to:

Optionally, the prediction module 304 is specifically configured to:

and performing lane line information fusion on the at least one first lane line information through a self-attention mechanism to obtain target lane line information, wherein the target lane line information comprises lane line type information and lane line position information of the at least one lane line.

Optionally, the detection device for lane lines further includes:

the sample detection module 305 is configured to obtain a sample image with lane line labeling information, and perform lane line detection on the sample image through a preset end-to-end neural network model to obtain predicted lane line information, where the end-to-end neural network model includes a first convolutional neural network, a feature pyramid network, a second convolutional neural network, and a lane line query tensor;

the loss calculation module 306 is configured to perform loss value calculation on the predicted lane line information and the lane line marking information through a preset loss function to obtain a target loss value, where the target loss value includes a lane line type loss value and a lane line position loss value;

the model generating module 307 is configured to generate a lane line detection model or perform parameter adjustment on the first convolutional neural network, the feature pyramid network, the second convolutional neural network, and the lane line query tensor in the end-to-end neural network model according to the target loss value.

Optionally, the loss calculating module 306 is specifically configured to:

In the embodiment of the invention, firstly, the characteristic information in the original image shot by the camera at different angles is extracted, then a plurality of 3d adopting points are defined for collecting the characteristic information corresponding to each sampling point, including the image characteristic information, so that the characteristic information for detecting the lane line does not need to completely depend on the image characteristics, 3d sampling spaces with different requirements can be constructed, then the prediction of the lane line correlation score is carried out based on the characteristic information collected by each sampling point, the sampling point with high lane line correlation degree is obtained, the prediction accuracy and integrity of the lane line in the 3d sampling space are improved through the sampling point with high lane line correlation degree, and the integrity of the lane line detection is improved.

Fig. 3 and 4 describe the detection apparatus of the lane line in the embodiment of the present invention in detail from the perspective of the modular functional entity, and the following describes the detection apparatus of the lane line in the embodiment of the present invention in detail from the perspective of hardware processing.

Fig. 5 is a schematic structural diagram of a lane line detection apparatus according to an embodiment of the present invention, where the lane line detection apparatus 500 may generate a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) storing applications 533 or data 532. Memory 520 and storage media 530 may be, among other things, transient or persistent storage. The program stored in the storage medium 530 may include one or more modules (not shown), each of which may include a series of computer program operations in the lane line detection apparatus 500. Still further, the processor 510 may be configured to communicate with the storage medium 530, and execute a series of computer program operations in the storage medium 530 on the lane marking detection apparatus 500.

The lane line detection apparatus 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input-output interfaces 560, and/or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that the configuration of the lane marking detection device shown in fig. 5 does not constitute a limitation of the lane marking detection device and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.

The present invention also provides a computer device, which includes a memory and a processor, wherein the memory stores a computer-readable computer program, and when the computer-readable computer program is executed by the processor, the processor executes the steps of the lane line detection method in the above embodiments.

The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, and which may also be a volatile computer-readable storage medium, having stored therein a computer program, which, when run on a computer, causes the computer to execute the steps of the lane line detection method.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes several computer programs to enable a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting a lane line, comprising:

determining a plurality of first sampling points, and performing characteristic collection of each first sampling point through the target image characteristic information to obtain the characteristic information of the sampling point corresponding to each first sampling point;

predicting lane line relevance scores of the characteristic information of the sampling points corresponding to each first sampling point to obtain the lane line relevance scores corresponding to each first sampling point, and determining a plurality of second sampling points in the plurality of first sampling points according to the lane line relevance scores corresponding to each first sampling point;

2. The method for detecting the lane line according to claim 1, wherein the determining a plurality of first sampling points and performing feature collection of each first sampling point according to the target image feature information to obtain feature information of the sampling point corresponding to each first sampling point comprises:

3. The method according to claim 2, wherein the constructing a 3d space with a target reference object as an origin and selecting a sample point from the 3d space to obtain a plurality of first sample points comprises:

4. The method for detecting the lane line according to claim 2, wherein the collecting the target image feature information and the 3d spatial position information for each first sampling point based on the image coordinate information of each first sampling point in the original image to obtain the feature information of the sampling point corresponding to each first sampling point comprises:

5. The method for detecting the lane line according to claim 1, wherein the obtaining an original image captured by at least one angle camera and performing image feature extraction on the original image to obtain target image feature information comprises:

6. The method for detecting the lane line according to claim 1, wherein the predicting the lane line correlation score of the characteristic information of the sampling point corresponding to each first sampling point to obtain the lane line correlation score corresponding to each first sampling point, and determining a plurality of second sampling points in the plurality of first sampling points according to the lane line correlation score corresponding to each first sampling point comprises:

7. The method for detecting the lane line according to claim 1, wherein the predicting the lane line through the characteristic information of the sampling point corresponding to each second sampling point to obtain the target lane line information comprises:

8. The method for detecting a lane line according to any one of claims 5 to 7, wherein before the obtaining of the original image captured by the at least one angle camera and the image feature extraction of the original image to obtain the target image feature information, the method for detecting a lane line further comprises:

9. The method according to claim 8, wherein the calculating a loss value of the predicted lane line information and the lane line labeling information by a preset loss function to obtain a target loss value comprises:

10. A lane line detection apparatus, comprising:

the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring an original image shot by at least one angle camera and extracting image characteristics of the original image to obtain target image characteristic information;

and the prediction module is used for predicting the lane line through the characteristic information of the sampling point corresponding to each second sampling point to obtain target lane line information.

11. A lane line detection apparatus, characterized by comprising: a memory and at least one processor, the memory having a computer program stored therein;

the at least one processor calls the computer program in the memory to cause the lane line detection apparatus to perform the lane line detection method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the lane marking detection method according to any one of claims 1 to 9.