CN114120281A

CN114120281A - Lane line tracking method and device, computer equipment and storage medium

Info

Publication number: CN114120281A
Application number: CN202111433344.8A
Authority: CN
Inventors: 史佳; 李晨光; 程光亮; 石建萍
Original assignee: Shanghai Sensetime Lingang Intelligent Technology Co Ltd
Current assignee: Shanghai Sensetime Lingang Intelligent Technology Co Ltd
Priority date: 2021-11-29
Filing date: 2021-11-29
Publication date: 2022-03-01
Also published as: WO2023093124A1

Abstract

The present disclosure provides a lane line tracking method, apparatus, computer device and storage medium, wherein the method comprises: acquiring a previous frame of image of an image to be identified, and carrying out image identification on the previous frame of image to obtain a first lane line; determining a second lane line in the image to be recognized and offset information of the second lane line relative to a first lane line closest to the first lane line based on a previous frame image and the image to be recognized; and determining the first lane line matched with the at least one second lane line based on the offset information corresponding to each second lane line and each first lane line, and obtaining the tracking result of the at least one second lane line.

Description

Lane line tracking method and device, computer equipment and storage medium

Technical Field

The present disclosure relates to the field of automatic driving technologies, and in particular, to a lane line tracking method, apparatus, computer device, and storage medium.

Background

The lane line detection and tracking are used as necessary processes in automatic driving, and the accuracy of the detection and tracking result is related to the safety of the automatic driving, but the existing lane line detection and tracking technology not only has complex detection and tracking processes, but also cannot ensure the accuracy of the finally determined tracking result.

Disclosure of Invention

The embodiment of the disclosure at least provides a lane line tracking method, a lane line tracking device, computer equipment and a storage medium.

In a first aspect, an embodiment of the present disclosure provides a lane line tracking method, including:

acquiring a previous frame image of an image to be identified, and carrying out image identification on the previous frame image to obtain a first lane line;

determining a second lane line in the image to be recognized and offset information of the second lane line relative to a first lane line closest to the first lane line based on the previous frame image and the image to be recognized;

and determining a first lane line matched with at least one second lane line based on the offset information corresponding to each second lane line and each first lane line, and obtaining a tracking result of at least one second lane line.

In the embodiment, in the process of detecting and tracking the second lane line in the image to be recognized, by combining the previous frame of image, when the second lane line is determined and the offset information of the second lane line relative to the first lane line closest to the first lane line is determined, the image characteristics of the previous frame of image can be combined, which is beneficial to improving the accuracy of the determined second lane line and the offset information corresponding to the second lane line; then, the second lane line is tracked based on the offset amount information with higher accuracy, and the accuracy of the determined tracking result can be improved.

In a possible embodiment, the determining, based on the previous frame image and the image to be recognized, a second lane line in the image to be recognized and offset information of the second lane line with respect to a closest first lane line includes:

determining a first image feature corresponding to the image to be recognized based on the previous frame image, first thermodynamic diagram feature information corresponding to the previous frame image and the image to be recognized;

determining second thermodynamic diagram feature information corresponding to the image to be recognized based on the first image features;

and determining a second lane line in the image to be recognized and offset information corresponding to the second lane line based on the second thermodynamic diagram feature information and the first image feature.

Here, based on the previous frame image, the first thermodynamic diagram feature information corresponding to the previous frame image, and the image to be recognized, the obtained first image feature may include an image feature of the image to be recognized, or the first image feature information corresponding to the previous frame image may be combined with the previous frame image and the first thermodynamic diagram feature information corresponding to the previous frame image, so that the richness of the feature information included in the first image feature is improved, and further, based on the first image feature including the abundant feature information, accurate second thermodynamic diagram feature information may be obtained, and since a thermodynamic value of a lane line is different from a thermodynamic value of another region in the image in which the lane line is located, based on the second thermodynamic diagram feature information, the second lane line may be accurately determined, and then, using the first image feature including the previous frame image, the offset information corresponding to the second lane line may be accurately determined.

In a possible implementation manner, the determining, based on the offset information corresponding to each second lane line and each first lane line, a first lane line matching at least one second lane line to obtain a tracking result of the at least one second lane line includes:

determining an initial pixel point corresponding to each second lane line based on the second thermodynamic diagram feature information;

respectively determining a target pixel point corresponding to each second lane line based on the offset information corresponding to each second lane line and the initial pixel point corresponding to each second lane line;

and determining a first lane line matched with at least one second lane line based on the target pixel point corresponding to each second lane line and the pixel point corresponding to each first lane line, so as to obtain a tracking result of at least one second lane line.

Here, since the thermal force value corresponding to the second lane line is different from the thermal force values of other regions in the image to be recognized, based on the second thermodynamic diagram feature information, the thermal force value of each pixel point can be determined, and further, the initial pixel point corresponding to the second lane line can be determined. The offset information can reflect a prediction error when the second lane line is predicted, the position of the initial pixel point is adjusted by using the determined offset information, the position of the predicted second lane line can be adjusted, the accurate position of the second lane line in the image to be recognized is determined, and further, the distance between the second lane line and each first lane line can be accurately determined by using the target pixel point corresponding to the second lane line and the pixel point corresponding to each first lane line, so that lane line matching is performed based on the determined distance, and the tracking result corresponding to the second lane line can be accurately determined.

In a possible implementation manner, the determining, based on the offset information corresponding to each second lane line and the initial pixel point corresponding to each second lane line, a target pixel point corresponding to each second lane line respectively includes:

for each second lane line, determining an offset value corresponding to each initial pixel point in the second lane line based on offset information corresponding to the second lane line;

and determining a target pixel point corresponding to the second lane line based on the offset value corresponding to each initial pixel point.

Here, based on the offset information corresponding to the second lane line, the offset value of each initial pixel point in the second lane line can be obtained, and then, the position of each initial pixel point can be adjusted by using the offset value, so that an accurate target pixel point corresponding to each initial pixel point is obtained.

In a possible embodiment, the determining, based on the target pixel point corresponding to each of the second lane lines and the pixel point corresponding to each of the first lane lines, a first lane line matching at least one of the second lane lines includes:

for each second lane line, based on the second thermodynamic diagram feature information, screening out pixel points to be matched from target pixel points of the second lane line;

respectively determining a first distance between each pixel point to be matched and each first lane line based on the pixel point corresponding to each first lane line;

determining a second distance between the second lane line and each first lane line based on the first distance corresponding to each pixel point to be matched;

determining a first lane line matching the second lane line based on the determined second distance.

Here, based on the mode of screening out part of pixels to be matched in the second lane line and determining the second distance between the second lane line and each first lane line, the distance determination by using all target pixels in the second lane line is avoided, the calculation amount required by the distance determination is effectively reduced, and the speed and the efficiency of determining the first lane line matched with the second lane line are further improved.

In one possible embodiment, after determining the first lane line matching the second lane line, the method further includes:

and under the condition that a plurality of second lane lines are matched with the same first lane line, determining that the second lane line with the shortest distance to the first lane line is matched with the first lane line.

Like this, can guarantee the one-to-one between two matching lane lines that determine, and do not have other matching lane lines, guarantee the rationality and the accuracy of pursuit result.

In one possible embodiment, the determining a first lane line matching the second lane line based on the determined second distance includes:

and under the condition that the shortest second distance is smaller than the first preset value, taking the first lane line corresponding to the shortest second distance as the lane line matched with the second lane line.

The first preset value is used for representing the maximum distance between two lane lines which can be matched, and the two lane lines are not matched under the condition that the distance between the two lane lines is larger than the maximum distance, so that the shortest first distance is further judged by using the shortest second distance according to the first preset value, and the reasonability and the accuracy of the determined matching result are improved.

In one possible embodiment, the determining, based on the determined second distance, a first lane line matching the second lane line further includes:

and taking the second lane line as a new lane line under the condition that the shortest second distance is greater than a first preset value.

Here, when the shortest second distance is greater than the first preset value, it is described that the two lane lines corresponding to the shortest second distance are not matched, and the corresponding second lane line is a new lane line, so that the second lane line is used as a new lane line, and the accuracy of the determined tracking result is improved.

In a possible implementation, after obtaining the tracking result, the method further includes:

for each first lane line which is not matched successfully, determining the number of times of the first lane line which is not matched successfully; the times of the unmatched success are used for representing the times of the corresponding first lane line which is not matched with the second lane line in the continuous multi-frame images to be recognized;

and deleting the first lane line under the condition that the number of times of the unmatched success is greater than a second preset value.

Here, the first lane line whose number of times of success of unmatching is greater than the second preset value is a lane line that has already disappeared with a high probability, and by deleting the first lane line, the number of stored lane lines can be reduced, the occupation of storage space can be reduced, and the detection efficiency can be improved.

In one possible implementation, the lane line tracking method is performed by a target neural network, and the target neural network is trained by adopting a plurality of sample recognition images.

In one possible embodiment, the target neural network is trained using the following steps:

inputting the sample identification image, first prediction thermodynamic diagram feature information corresponding to a sample image of a previous frame of the sample identification image and the sample image of the previous frame into the target neural network, and outputting second prediction thermodynamic diagram feature information corresponding to the sample identification image, a predicted lane line corresponding to the sample identification image, prediction offset information corresponding to the predicted lane line and a prediction tracking result corresponding to each predicted lane line by the target neural network;

determining a loss value based on the second predictive thermodynamic diagram feature information, the labeled thermodynamic diagram feature information corresponding to the sample identification image, the predicted lane line, the labeled lane line corresponding to the sample identification image, the predicted offset information, the labeled offset information corresponding to the sample identification image, and the predicted tracking result corresponding to each predicted lane line and the labeled tracking result corresponding to each labeled lane line;

and adjusting the network parameter value of the target neural network according to the loss value until a preset training cut-off condition is met, so as to obtain the trained target neural network.

In this way, the target loss values corresponding to all the information can be respectively determined by using the second predictive thermodynamic diagram characteristics, the predicted lane lines, the predicted offset information and the predicted tracking result corresponding to each predicted lane line, and then the network parameter values of the target neural network are adjusted by using the loss values, so that the prediction precision of the target neural network can be improved, and therefore, the target neural network can be ensured to output each accurate information.

In one possible embodiment, the predicted shift amount information includes a shift amount of the predicted lane line in at least one image direction.

Therefore, the offset of the lane line in one direction is output, so that the calculation amount required to be processed by the target neural network when the offset information is output can be effectively reduced, and the speed and the efficiency of information processing can be improved.

In one possible embodiment, the sample image of the previous frame of the sample identification image is obtained by:

and performing random translation and/or rotation operation on the sample identification image to obtain a previous frame image of the sample identification image.

Therefore, the positions of the pixel points in the sample identification image can be changed through random translation and/or rotation operation, and the simulated previous frame image is obtained, so that the training of the target neural network can be completed only by acquiring one frame of sample identification image, and the flexibility of the training of the neural network is improved.

In one possible embodiment, acquiring a first lane line obtained by image recognition of the previous frame image includes:

under the condition that the previous frame image is the first frame image, performing image recognition on the previous frame image, and determining a second image characteristic corresponding to the previous frame image;

determining first thermodynamic diagram feature information corresponding to the previous frame of image based on the second image feature;

determining a first lane line in the previous frame image based on the first thermodynamic diagram feature information and the second image feature.

Therefore, the identification processing of the first frame image can be directly carried out according to the image, the first lane line can be obtained without acquiring the corresponding previous frame image and the corresponding thermodynamic diagram characteristic information, and the flexibility of the processing of the target neural network is improved.

In a possible implementation manner, the determining, based on the previous frame image and the image to be recognized, a second lane line in the image to be recognized and offset information corresponding to the second lane line includes:

respectively carrying out downsampling processing on the image to be identified and the previous frame image according to a preset sampling multiple to obtain a new image to be identified and a new previous frame image;

and determining a second lane line in the image to be recognized and offset information of the second lane line relative to the first lane line closest to the new previous frame image and the new image to be recognized.

Therefore, discretization processing of the lane lines in the image to be recognized (and the previous frame image) can be achieved through a down-sampling processing mode, information of a plurality of pixel points corresponding to the sampling multiple can be represented by each pixel point in the obtained new image to be recognized (and the new previous frame image), and further, more accurate second lane lines and offset information corresponding to the second lane lines can be determined beneficially by utilizing the information of each pixel point in the new image to be recognized (and the new previous frame image).

In a second aspect, an embodiment of the present disclosure further provides a lane line tracking apparatus, including:

the acquisition module is used for acquiring a previous frame image of an image to be identified and a first lane line obtained by carrying out image identification on the previous frame image;

the first determining module is used for determining a second lane line in the image to be recognized and offset information of the second lane line relative to a first lane line closest to the first lane line based on the previous frame image and the image to be recognized;

and the second determining module is used for determining a first lane line matched with at least one second lane line based on the offset information corresponding to each second lane line and each first lane line to obtain a tracking result of at least one second lane line.

In a possible implementation manner, the first determining module is configured to determine a first image feature corresponding to the image to be recognized based on the previous frame image, the first thermodynamic diagram feature information corresponding to the previous frame image, and the image to be recognized;

In a possible implementation manner, the second determining module is configured to determine, based on the second thermodynamic diagram feature information, an initial pixel point corresponding to each of the second lane lines;

In a possible implementation manner, the second determining module is configured to determine, for each second lane line, an offset value corresponding to each initial pixel point in the second lane line based on offset information corresponding to the second lane line;

In a possible implementation manner, the second determining module is configured to, for each second lane line, screen a pixel to be matched from target pixels of the second lane line based on the second thermodynamic diagram feature information;

In a possible embodiment, the second determining module is further configured to, after determining the first lane line that matches the second lane line, determine that the second lane line that is the shortest distance from the first lane line matches the first lane line if a plurality of second lane lines match the same first lane line.

In a possible embodiment, the second determining module is configured to, when the shortest second distance is smaller than a first preset value, use the first lane line corresponding to the shortest second distance as the lane line matched with the second lane line.

In a possible embodiment, the second determining module is further configured to take the second lane line as a new lane line if the shortest second distance is greater than the first preset value.

In a possible embodiment, the apparatus further comprises:

the deleting module is used for determining the number of times of the first lane line which is not matched successfully aiming at each first lane line which is not matched successfully after the tracking result is obtained; the times of the unmatched success are used for representing the times of the corresponding first lane line which is not matched with the second lane line in the continuous multi-frame images to be recognized;

In a possible embodiment, the apparatus further comprises:

a training module for training the target neural network using the steps of:

In a possible implementation, the training module is further configured to obtain a sample image of a frame preceding the sample identification image by:

In a possible implementation manner, the obtaining module is configured to perform image recognition on the previous frame image and determine a second image feature corresponding to the previous frame image when the previous frame image is a first frame image;

In a possible implementation manner, the first determining module is configured to perform downsampling processing on the image to be identified and the previous frame of image according to a preset sampling multiple, so as to obtain a new image to be identified and a new previous frame of image;

In a third aspect, this disclosure also provides a computer device, a processor, and a memory, where the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the machine-readable instructions are executed by the processor to perform the steps in the first aspect or any one of the possible implementations of the first aspect.

In a fourth aspect, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, where the computer program is executed to perform the steps in the first aspect or any one of the possible implementation manners of the first aspect.

For the description of the effects of the lane line tracking apparatus, the computer device, and the computer-readable storage medium, reference is made to the description of the lane line tracking method, which is not repeated herein.

In order to make the aforementioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments accompanied with figures are described in detail below.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly described below, and the drawings herein incorporated in and forming a part of the specification illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the technical solutions of the present disclosure. It is appreciated that the following drawings depict only certain embodiments of the disclosure and are therefore not to be considered limiting of its scope, for those skilled in the art will be able to derive additional related drawings therefrom without the benefit of the inventive faculty.

Fig. 1 shows a flowchart of a lane line tracking method provided by an embodiment of the present disclosure;

FIG. 2 illustrates a schematic view of a first lane line and a second lane line identified in accordance with an embodiment of the present disclosure;

fig. 3 is a schematic diagram illustrating a recognition of determining a second lane line by an inference module in a target neural network performing image recognition on an image to be recognized according to an embodiment of the present disclosure;

fig. 4 is a flowchart illustrating a specific implementation of image recognition on an image to be recognized by an inference module in a target neural network according to an embodiment of the present disclosure;

fig. 5 is a flowchart illustrating a tracking module in a target neural network determining a tracking result corresponding to a second lane line according to an embodiment of the present disclosure;

FIG. 6 is a diagram illustrating output predicted offset information provided by an embodiment of the present disclosure;

FIG. 7 is a schematic flow chart illustrating training of a target neural network according to an embodiment of the present disclosure;

FIG. 8 illustrates a schematic diagram of a lane line tracking apparatus provided by an embodiment of the present disclosure;

fig. 9 shows a schematic structural diagram of a computer device provided by an embodiment of the present disclosure.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, not all of the embodiments. The components of embodiments of the present disclosure, as generally described and illustrated herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the disclosure, as claimed, but is merely representative of selected embodiments of the disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the disclosure without making creative efforts, shall fall within the protection scope of the disclosure.

Furthermore, the terms "first," "second," and the like in the description and in the claims, and in the drawings described above, in the embodiments of the present disclosure are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Reference herein to "a plurality or a number" means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

The research shows that the lane line detection and tracking are used as the necessary process in the automatic driving, and the accuracy of the detection and tracking result is related to the safety of the automatic driving, but most of the existing lane line detection and tracking technologies firstly utilize a neural network to detect the lane line and then use a lengthy traditional algorithm (such as Kalman filtering and Hungarian algorithm) to carry out post-processing on the detected lane line, so that the tracking of the lane line is realized.

Complex sensor models also need to be incorporated in the course of lane line tracking using conventional algorithms. Therefore, the complexity of the detection and tracking process is improved, the detection result is limited by the detection result of each sensor model, and the accuracy of the lane line tracking result is greatly influenced.

Based on the above research, the present disclosure provides a lane line tracking method, apparatus, computer device, and storage medium, in the process of detecting and tracking a second lane line in an image to be recognized, by combining with a previous frame image, when determining the second lane line and determining offset information of the second lane line relative to a first lane line closest to the second lane line, the image feature of the previous frame image can be combined, which is beneficial to improving the accuracy of the determined second lane line and the offset information corresponding to the second lane line; then, the second lane line is tracked based on the offset amount information with higher accuracy, and the accuracy of the determined tracking result can be improved.

The above-mentioned drawbacks are the results of the inventor after practical and careful study, and therefore, the discovery process of the above-mentioned problems and the solutions proposed by the present disclosure to the above-mentioned problems should be the contribution of the inventor in the process of the present disclosure.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.

To facilitate understanding of the present embodiment, a detailed description is first given of a lane line tracking method disclosed in the embodiments of the present disclosure, where an execution subject of the lane line tracking method provided in the embodiments of the present disclosure is generally a computer device with certain computing capability, and in some possible implementation manners, the lane line tracking method may be implemented by a processor calling a computer readable instruction stored in a memory.

The following describes the lane line tracking method provided by the embodiment of the present disclosure by taking an execution subject as a target neural network as an example.

As shown in fig. 1, a flowchart of a lane line tracking method provided in an embodiment of the present disclosure may include the following steps:

s101: acquiring a previous frame image of the image to be identified, and carrying out image identification on the previous frame image to obtain a first lane line.

Here, the image to be recognized may be an image including a lane line captured by an imaging device provided on the target vehicle, and the previous frame image may be previously captured by the imaging device or may be generated from the image to be recognized, which is not limited herein.

In specific implementation, after the image to be recognized is acquired, the image to be recognized may be input to the target neural network, and the target neural network is used to perform image recognition on the previous frame of image, so as to obtain each first lane line in the previous frame of image.

S102: and determining a second lane line in the image to be recognized and offset information of the second lane line relative to the first lane line closest to the first lane line based on the previous frame image and the image to be recognized.

Fig. 2 is a schematic diagram of a first lane line and a second lane line identified according to an embodiment of the disclosure. The first lane line comprises a first lane line 1, a first lane line 2, a first lane line 3 and a first lane line 4, and the second lane line comprises a second lane line 1, a second lane line 2, a second lane line 3 and a second lane line 4.

The offset information is used to represent an offset between the second lane line and the nearest first lane line, and specifically, may be position offset information between a pixel point corresponding to each first lane point in the second lane line and a pixel point corresponding to each second lane point in the nearest first lane line. The lane points may be key points on the lane lines, the offset information may be offsets of each second lane line in the image width direction, and specifically, when predicting the offset corresponding to the lane point on each lane line, the target neural network may fix information of the pixel point in the image height direction for the pixel point corresponding to each lane point, and determine the offset information of the pixel point in the image width direction. Therefore, when the target neural network is used for matching lane lines subsequently, the adjustment of the lane line position can be realized by only calculating the offset information in one direction (image width), and the lane line in the previous frame image matched with the lane line is determined, so that the calculation amount is reduced, and the speed and the efficiency of lane line tracking are improved.

In specific implementation, the target neural network may utilize the convolution layer to perform convolution on the previous frame image and the image to be recognized respectively to obtain feature maps corresponding to the two images respectively, and combine the two obtained feature maps to obtain a target feature map corresponding to the image to be recognized. The feature map corresponding to each image obtained by convolution may include a thermal feature map, a depth feature map, and the like.

Then, based on the image features included in the obtained target feature map, the image feature corresponding to each second lane line may be determined, and further, each second lane line in the image to be recognized may be determined.

Meanwhile, the target neural network can also determine a first lane line in the previous frame of image according to the image characteristics included in the target characteristic diagram; according to the determined second lane lines and the first lane lines, the first lane lines corresponding to the second lane lines and having the closest distance can be respectively determined, and then the offset information of the second lane lines relative to the first lane lines having the closest distance can be obtained.

Wherein the offset information can be predicted by an offset branch in the target neural network.

S103: and determining the first lane line matched with the at least one second lane line based on the offset information corresponding to each second lane line and each first lane line, and obtaining the tracking result of the at least one second lane line.

Here, the tracking result is a result representing whether or not there is a lane line matching the second lane line in the first lane line, that is, it is possible to represent whether or not there is a second lane line identical to the first lane line in the image to be recognized

Each first lane line may have lane line identification information for identifying the identity of the lane line, for example, the lane line identification information may include a lane line number. In specific implementation, if the second lane line has the first lane line matching with the second lane line, the second lane line may be used as the lane line identical to the first lane line, and the lane line number of the first lane line may be used as the lane line number of the second lane line, otherwise, a new lane line number may be generated for the second lane line.

In specific implementation, for each second lane line, the offset information of the second lane line may be utilized to adjust the position of the second lane line to obtain each adjusted second lane line, and then the second lane line is matched with the first lane line to determine whether the first lane line matched with the second lane line exists.

If so, the lane line identification information corresponding to the first lane line matching the second lane line may be used as the lane line identification information of the second lane line, and the lane line identification information may be used as the tracking result of the second lane line. For example, the first lane line includes three first lane lines numbered 1, 2, and 3, and in a case where it is determined that the lane line number of the first lane line matching a certain second lane line is 2, the lane line number 2 may be set as the lane line number of the second lane line.

In addition, the second lane line can be used to replace the first lane line matched with the second lane line stored in the local database. The local database stores each identified lane line, the image corresponding to the lane line, the image characteristics and the lane line identification information, and specifically, all information related to the lane line can be stored in the local database. For example, the second lane line may be used to replace the first lane line matching with the second lane line, and specifically, each second lane point corresponding to the second lane line may be directly used to replace the first lane point corresponding to the first lane line.

If not, the second lane line can be determined to be the newly identified lane line, and lane line identification information corresponding to the lane line is generated and used as the tracking result of the second lane line. In addition, the second lane line may also be stored in a local database. Continuing with the above example, in the event that it is determined that there is no first lane line matching a certain second lane line, a new lane line number 4 may be generated for the second lane line.

Therefore, the detection and the tracking of the lane line can be realized only by one target neural network, the relevance between the detection and the tracking is improved, and the combination of the detection and the tracking is favorable for improving the accuracy of the detection. In addition, the embodiment can realize detection and tracking by utilizing the acquired image to be identified without combining a multi-sensor model, thereby not only reducing the complexity of detection and tracking, but also improving the universality of the lane line tracking method. In addition, the steps of detection and tracking are executed in the same target neural network, so that part of post-processing flow aiming at the tracking process can be reduced, the influence of the detection result of the sensor model on the tracking result is avoided, and the tracking precision is improved. In addition, in the process of tracking the second lane line in the image to be recognized, by combining the previous frame image, when the offset information corresponding to the second lane line is determined, the image characteristics of the previous frame image can be combined, so that the accuracy of the determined offset information is favorably improved, and then the second lane line is tracked based on the offset information with higher accuracy, so that the accuracy of the determined tracking result can be improved.

In one embodiment, for S102, the following steps may be performed:

s102-1: and determining a first image feature corresponding to the image to be recognized based on the previous frame image, the first thermodynamic diagram feature information corresponding to the previous frame image and the image to be recognized.

Here, the thermodynamic characteristic information can reflect a thermodynamic value of each pixel point in the image, and the thermodynamic characteristic information may be expressed in a form of thermodynamic diagram, where the first thermodynamic characteristic information corresponds to the first thermodynamic diagram, and thermodynamic values of pixel points corresponding to different lane lines are different, for example, for a road image, a thermodynamic value of each lane point in a lane line is higher than a thermodynamic value of other position points on the road.

The first thermodynamic diagram feature information corresponding to the previous frame of image can be obtained by performing image recognition on the previous frame of image for the target neural network.

In specific implementation, after acquiring a previous frame image, a first thermodynamic diagram corresponding to the previous frame image, and an image to be identified, the target neural network may perform convolution operations on each image respectively to obtain a convolution result corresponding to each image, then may perform fusion on the convolution results to obtain an initial feature map corresponding to the initial image feature, and then performs feature coding on the initial image feature by using an encoder to obtain a first feature map corresponding to the image to be identified, where the first feature map includes a first image feature corresponding to the image to be identified, and the first image feature may be a feature vector.

S102-2: and determining second thermodynamic diagram feature information corresponding to the image to be recognized based on the first image features.

In specific implementation, the target neural network performs further feature processing on the determined first feature map to obtain second thermodynamic map feature information. For example, a decoding operation may be performed on the first feature map, a thermal value corresponding to each pixel point in the first feature map is determined, and then, second thermodynamic map feature information corresponding to the image to be recognized may be determined.

S102-3: and determining a second lane line in the image to be recognized and offset information corresponding to the second lane line based on the second thermodynamic diagram feature information and the first image feature.

In this step, the thermodynamic value of each pixel point in the second thermodynamic diagram can be determined based on the second thermodynamic diagram feature information, and then the initial pixel point corresponding to the second lane point in the second lane line can be determined, and then feature clustering can be performed on the feature corresponding to each second lane point according to the determined position of each initial pixel point in the second thermodynamic diagram, so as to determine the second lane line to which each second lane point belongs. In specific implementation, feature Embedding branches in the target neural network can be used for performing feature clustering on features corresponding to the second lane points to determine the second lane lines, wherein the feature Embedding branches can be Embedding layers in the target neural network.

In one embodiment, the maximum number of the identified lane lines may be preset, and in the case that the number of the identified lane lines in any frame of the image to be identified is greater than the maximum number, the abnormality prompt information may be generated. That is, the number of lane lines on the road where the target vehicle is located is limited during the traveling of the target vehicle, and if the number of identified lane lines is too large, there may be a problem of detection error, and thus, the safety of the automatic driving will be affected, and therefore, the safety of the automatic driving can be improved by setting the maximum lane line and generating the abnormality prompt information.

Moreover, the target neural network may further determine each first lane line in the previous frame of image according to feature processing of the first image feature, then may determine a distance between each first lane line and each second lane line based on the first lane line and the second lane line, and according to the determined distance, may determine the first lane line of the closest distance corresponding to each second lane line. Further, for each second lane line, offset information of the second lane line with respect to the closest first lane line may be determined. In a specific implementation, the step of determining the second lane line, the offset information corresponding to the second lane line, and the second thermodynamic diagram characteristic information may be performed by an inference module in the target neural network. As shown in fig. 3, a schematic identification diagram for identifying an image to be identified and determining a second lane line is provided for an inference module in a target neural network according to an embodiment of the present disclosure.

In specific implementation, the target neural network may include a reasoning module and a tracking module, where the tracking module is configured to determine a first lane line that matches a second lane line in the image to be recognized, the reasoning module is configured to determine a lane line corresponding to each acquired image to be recognized, offset information of the lane line, and thermodynamic diagram feature information, and after determining the information corresponding to the image to be recognized, input the information into the tracking module to complete tracking of the second lane line in the image to be recognized.

As shown in fig. 4, a flowchart of a specific implementation of performing image recognition on an image to be recognized by an inference module in a target neural network according to an embodiment of the present disclosure is provided. The image used for recognition comprises any frame of image shot by an automatic driving device (target vehicle), the feature embedding branch is used for carrying out feature clustering on the features of initial pixel points corresponding to lane points and determining lane lines in the currently processed N frame of image to be recognized, in addition, in the graph 4, the image to be recognized which is input into the target neural network together with thermodynamic diagram feature information corresponding to the N frame of image to be recognized also comprises an N frame of image to be recognized, and the image to be recognized and the thermodynamic diagram feature information are used for carrying out image recognition on the N +1 frame of image to be recognized.

In an embodiment, for the step of performing image recognition on the previous frame image, when the previous frame image is the first frame image, the previous frame image may be directly subjected to image recognition without using the previous frame image of the previous frame image and corresponding thermodynamic diagram feature information thereof, and the second image feature corresponding to the previous frame image is determined. Furthermore, the second image feature may be processed to determine first thermodynamic characteristic information corresponding to a previous frame of image, and then the first lane line in the previous frame of image may be determined based on the first thermodynamic characteristic information and the second image feature. Regarding the step of determining the first lane line in the previous frame image based on the first thermodynamic diagram feature information and the second target image feature, reference may be made to the step of determining the second lane line in the foregoing embodiment, which is not described herein again.

In addition, when the previous frame image is the first frame image, since the previous frame image does not exist in the first frame image, the lane line in the first frame image does not have an offset from the lane line in the previous frame image, and it is not necessary to match the lane line in the first frame image with the lane line in the previous frame image, and therefore, the offset information corresponding to the first frame image output by the target neural network can be deleted.

In one embodiment, for S103, the following steps may be specifically implemented:

s103-1: and determining an initial pixel point corresponding to each second lane line based on the second thermodynamic diagram feature information.

Here, the initial pixel point is a pixel point in the second thermodynamic diagram.

In specific implementation, based on the second thermodynamic diagram feature information, the thermodynamic value of each pixel point in the second thermodynamic diagram can be determined, and then whether each pixel point belongs to a lane point or not can be determined, and the pixel point belonging to the lane point is used as an initial pixel point corresponding to each second lane line.

S103-2: and respectively determining target pixel points corresponding to each second lane line based on the offset information corresponding to each second lane line and the initial pixel points corresponding to each second lane line.

In this step, for each second lane line, the positions of the initial pixel points corresponding to each second lane point in the second lane line may be adjusted by using the offset information corresponding to each second lane point in the second lane line, so as to determine the target positions corresponding to each initial pixel point, and the initial pixel points at the target positions are used as the target pixel points corresponding to the second lane lines.

In an embodiment, the target pixel point corresponding to each second lane line may be determined according to the following steps:

step one, aiming at each second lane line, determining an offset value corresponding to each initial pixel point in the second lane line based on offset information corresponding to the second lane line.

Here, the offset information corresponding to the second lane line can represent an offset value corresponding to each second lane point in the second lane line.

In this step, for each second lane line, an offset value of an initial pixel point corresponding to each second lane point in the second lane line may be determined according to offset information corresponding to the second lane line.

And step two, determining a target pixel point corresponding to the second lane line based on the offset value corresponding to each initial pixel point.

Here, the position of each initial pixel point may be adjusted by using the offset value of each initial pixel point, and a target pixel point corresponding to each initial pixel point is determined, that is, a target pixel point corresponding to each second lane point in the second lane line is obtained.

And then, determining target pixel points corresponding to each second lane point in each second lane line respectively based on the offset information corresponding to each second lane line and the initial pixel points corresponding to the offset information.

S103-3: and determining the first lane line matched with the at least one second lane line based on the target pixel point corresponding to each second lane line and the pixel point corresponding to each first lane line, so as to obtain the tracking result of the at least one second lane line.

In specific implementation, for each second lane line and each first lane line, the initial distance between each target pixel point corresponding to the second lane line and each pixel point in the first lane line may be determined according to the position of each target pixel point corresponding to each second lane point of the second lane line and the position of each pixel point corresponding to each first lane point in the first lane line. Here, the initial distance may be a distance between the target pixel point and the pixel point in the first lane line at the same height, that is, the target pixel point corresponding to each initial distance and the pixel point in the corresponding first lane line are located at the same height.

Furthermore, the target distance between the second lane line and the first lane line corresponding to each target pixel point can be determined according to the determined initial distance corresponding to each target pixel point, and then the target distance between each second lane line and each first lane line can be determined.

Then, for each first lane line, a second lane line with the shortest target distance therebetween may be determined, and further, the second lane line may be used as a lane line matched with the first lane line, so as to obtain a tracking result of the second lane line. Based on this, the tracking result of each second lane line can be determined.

Or, for each second lane line and each first lane line, the similarity between each first lane line and each second lane line may also be determined according to the characteristics of the target pixel points corresponding to each second lane point in the second lane lines and the characteristics of each pixel point corresponding to each first lane point in the first lane lines, and then the tracking result of each second lane line is determined based on the determined similarity, which is not repeated here.

In one embodiment, for S103-3, the following steps may be specifically implemented:

s103-3-1: and screening out pixel points to be matched from target pixel points of the second lane lines based on the second thermodynamic diagram characteristic information aiming at each second lane line.

And the pixel points to be matched are pixel points which are screened from the target pixel points and are used for determining the first lane line matched with the second lane line.

In specific implementation, for each second lane line, the thermal force value corresponding to each corresponding target pixel point in the second lane line may be determined first. Specifically, for each target pixel point in the second lane line, an initial pixel point corresponding to the target pixel point can be determined, and then, based on the second thermodynamic diagram feature information, a thermodynamic value corresponding to the initial pixel point can be determined, and further, the thermodynamic value can be used as a thermodynamic value corresponding to the target pixel point. In this way, a thermal force value corresponding to each target pixel point in the second lane line may be determined.

Then, according to the corresponding heat value of each target pixel point, sequencing each target pixel point in a descending order, and taking the target pixel points with the sequencing order larger than the preset order as pixel points to be matched.

Or for each target pixel point in each second lane line, the confidence corresponding to each target pixel point can be determined, then, a first target pixel point with the confidence greater than a preset confidence threshold value can be screened out from the target pixel points corresponding to the second lane lines, then, the thermodynamic value of each first target pixel point is determined based on the second thermodynamic diagram characteristic information, the first target pixel points are sequenced based on the thermodynamic values, and the first target pixel points with the sequencing order greater than the preset order are used as pixel points to be matched.

S103-3-2: and respectively determining a first distance between each pixel point to be matched and each first lane line based on the pixel points corresponding to each first lane line.

In this step, for each pixel point to be matched in the second lane line and each first lane line, a pixel point having the same height as the pixel point to be matched is selected from each pixel point corresponding to the first lane line based on the position of the pixel point to be matched and the position of the pixel point corresponding to each first lane line in the first lane line, and then, the initial distance between the pixel point to be matched and the pixel point can be determined, and the initial distance is used as the first distance with the first lane line.

Furthermore, pixel points with the same height as each pixel point to be matched can be screened out from all pixel points corresponding to the first lane line, the initial distance corresponding to each pixel point to be matched is determined based on the screened pixel points, and therefore the first distance between each pixel point to be matched and each first lane line can be obtained.

S103-3-3: and determining second distances between the second lane line and each first lane line based on the first distance corresponding to each pixel point to be matched.

Here, the second distance may be an euclidean distance between two lane lines.

For the second lane line and each first lane line, the second distance between the second lane line and the first lane line can be determined according to the first distance between each pixel point to be matched in the second lane line and the pixel point in the first lane line.

In specific implementation, an average value of first distances corresponding to each pixel point to be matched in the second lane line may be used as the second distance, the smallest first distance may also be selected as the second distance, a weight corresponding to each pixel point to be matched may also be determined according to a confidence corresponding to each pixel point to be matched, and the first distances are summed according to the weight corresponding to each pixel point to be matched and the first distance corresponding thereto to obtain the second distance, where the second distance is not limited.

Therefore, the second distance between each second lane line and each first lane line is determined through the screened pixel points to be matched, calculation is not needed to be carried out by utilizing each target pixel point in the second lane lines, the calculation amount for determining the second distance is reduced, the speed for determining the second distance is improved, and further, the speed for determining the tracking result is favorably improved.

S103-3-4: based on the determined second distance, a first lane line matching the second lane line is determined.

In this step, for each second lane line, a first lane line having a shortest second distance from the second lane line may be determined based on a second distance between the second lane line and each first lane line, and the first lane line is taken as a lane line matching the second lane line.

In one embodiment, in a case where a plurality of second lane lines are matched to the same first lane line, that is, for any first lane line, in a case where a plurality of second lane lines are matched to the first lane line, each of the plurality of second lane lines may be compared with a second distance corresponding to the first lane line; and determining the shortest second distance, and determining the second lane line corresponding to the shortest second distance as the lane line finally matched with the first lane line.

In addition, for each of the second lane lines except for the finally matched second lane line, a second short second distance corresponding to each of the second lane lines may be determined, and then, using the second short second distance and the shortest second distance corresponding to the other second lane lines, it may be determined whether there is a matched first lane line in the second lane lines. Thus, the matching result of each second lane line is determined.

In one embodiment, for each second lane line, after determining the first lane line matching therewith, the second distance between the second lane line and the first lane line may be further compared with the first preset value. Here, the first preset value is used to represent a maximum distance between the matched first lane line and the second lane line, and in the case where the second distance is greater than the maximum distance, even if the second distance between the two lane lines is the shortest, the two lane lines are determined as unmatched lane lines.

In specific implementation, when it is determined that the shortest second distance corresponding to the first lane line is smaller than the first preset value, the first lane line may be used as a final matching lane line of the second lane line corresponding to the shortest second distance.

In another embodiment, when it is determined that the shortest second distance corresponding to the first lane line is greater than the first preset value, the second lane line corresponding to the shortest second distance may be used as a new lane line, that is, it may be determined that there is no lane line matching the second lane line in the first lane line.

In one embodiment, after determining the tracking result corresponding to each second lane line, for each first lane line stored in the local database, based on the tracking result, the number of times of success of unmatching corresponding to each first lane line may be determined. Here, the number of times of success of unmatching is used to represent the number of times the first lane line is continuously unmatched with the second lane line.

Then, for the first lane line, the number of times of successful unmatching may be compared with a second preset value, and when it is determined that the number of times of successful unmatching is greater than the second preset value, it indicates that the first lane line has been successfully unmatched for a plurality of consecutive times, and it indicates that the second lane line has disappeared within the driving range of the target vehicle with a high probability, and the first lane line may be deleted.

If the number of times of the unmatched success is smaller than the second preset value, the first lane line can be stored continuously for the next matching.

In addition, after each first lane line is matched by the second lane line, the number of times of success of non-matching corresponding to the first lane line may be set as an initial value, where the initial value may be 0.

As shown in fig. 5, a flowchart for determining a tracking result corresponding to a second lane line by a tracking module in a target neural network is provided in the embodiment of the present disclosure. After acquiring the lane line and the offset information of the lane line input by the inference module, the tracking module may first determine whether the lane line is the lane line corresponding to the first frame image, that is, determine whether the image to be identified corresponding to the lane line is the first frame image; if the image to be recognized is the first frame image, the inference module can only input the corresponding lane line of the first frame image and does not input the offset information of the lane line. The local database is used for storing information of each lane line determined after image recognition is performed on the image to be recognized, and specifically, the stored information may be lane point information, lane line number information and the like on the lane line. If not, the obtained tracking result of the lane line input by the inference module may be determined according to the lane line tracking method described in the above embodiments, which is not described herein again.

In an embodiment, for S102, after the image to be recognized and the previous frame image are acquired, the image to be recognized and the previous frame image may be respectively downsampled by using a preset sampling multiple, so as to obtain a new image to be recognized corresponding to the image to be recognized and a new previous frame image corresponding to the previous frame image and being sampled. For example, the image to be recognized and the previous frame image may be downsampled by a 4-fold downsampling method or an 8-fold downsampling method.

Then, the target neural network can be used for performing convolution on the new image to be recognized, the new previous frame image and the first thermodynamic diagram corresponding to the previous frame image respectively to obtain the first image feature corresponding to the new image to be recognized. Then, the second lane line in the image to be recognized and offset information corresponding to the second lane line are determined by using the first image feature, which is not described herein again.

Here, since the first thermodynamic diagram is obtained by image recognition of the previous frame image, the downsampling of the previous frame image is already completed in the process of image recognition of the previous frame image, and the obtained first thermodynamic diagram is the first thermodynamic diagram corresponding to the sampled previous frame image, so that the convolution processing can be directly performed on the first thermodynamic diagram, and there is no need to perform downsampling processing on the first thermodynamic diagram.

Therefore, by means of downsampling processing, each pixel point in the obtained new image (and the new previous frame image and the new image to be identified) can represent information of a plurality of pixel points corresponding to the sampling multiple, and further, the information of each pixel point in the new image is utilized, and more accurate second lane lines and offset information corresponding to the second lane lines can be determined.

In an embodiment, since the lane line tracking method provided by the embodiment of the present disclosure is executed by the target neural network, and the target neural network needs to be trained to ensure the prediction accuracy, the embodiment of the present disclosure further provides a step of training the target neural network by using a plurality of sample recognition images:

step T1: inputting the sample identification image, first prediction thermodynamic diagram characteristic information corresponding to a previous frame of sample image of the sample identification image and the previous frame of sample image into a target neural network, and processing the input information through the target neural network to obtain second prediction thermodynamic diagram characteristic information corresponding to the sample identification image, a prediction lane line corresponding to the sample identification image, prediction offset information corresponding to the prediction lane line and a prediction tracking result corresponding to each prediction lane line.

The multiple sample identification images may be images in the same video segment, each sample image includes a lane line, or the multiple sample identification images may also be images including a lane line, which are shot separately.

For example, the plurality of sample recognition images may be images in a video clip taken by the autonomous vehicle that contains visible lane lines.

The first prediction thermodynamic diagram feature information corresponding to the previous frame of sample image may be obtained by performing image recognition on the previous frame of sample image for the target neural network, and the predicted lane line may be a predicted lane line in the sample recognition image output by the target neural network.

In a specific implementation, when the sample identification image is subjected to image identification, the sample identification image, first predicted thermodynamic characteristic information corresponding to a sample image in a frame previous to the sample identification image, and the sample image in the previous frame are input into a target neural network, an inference module in the target neural network is used for processing the input information to determine a sample predicted image characteristic corresponding to the sample identification image, and then a second predicted thermodynamic characteristic corresponding to the sample identification image is determined based on the sample predicted image characteristic. Furthermore, the predicted lane line corresponding to the sample recognition image and the predicted offset information corresponding to the predicted lane line may be determined based on the second predicted thermodynamic characteristic and the sample predicted image characteristic, and then the predicted tracking result corresponding to each predicted lane line may be determined by the tracking module.

Step T2: and determining a loss value based on the second prediction thermodynamic diagram feature information, the labeling thermodynamic diagram feature information corresponding to the sample identification image, the predicted lane line, the labeling lane line corresponding to the sample identification image, the predicted offset information, the labeling offset information corresponding to the sample identification image, the predicted tracking result corresponding to each predicted lane line and the labeling tracking result corresponding to each labeling lane line.

In specific implementation, a first predicted loss value corresponding to a thermodynamic diagram branch in the target neural network may be determined based on the second predicted thermodynamic diagram feature and the labeled thermodynamic diagram feature information (i.e., a true value) corresponding to the sample identification image, a second predicted loss value corresponding to a feature embedding branch in the target neural network may be determined based on the predicted lane line and the labeled lane line corresponding to the sample identification image, a third predicted loss value corresponding to an offset prediction branch in the target neural network may be determined based on the predicted offset information and the labeled offset information corresponding to the sample identification image, and a fourth predicted loss value corresponding to the target neural network may be determined based on the predicted tracking result corresponding to each predicted lane line and the labeled tracking result corresponding to each standard lane line. And then, the first predicted loss value, the second predicted loss value, the third predicted loss value and the fourth predicted loss value can be used as the loss values corresponding to the target neural network.

Step T3: and adjusting the network parameter value of the target neural network according to the loss value until a preset training cut-off condition is met, so as to obtain the trained target neural network.

Here, the preset training cutoff condition may be that the number of rounds of iterative training is greater than a preset number of rounds and/or that the prediction accuracy of the target neural network obtained by training reaches a preset accuracy.

In specific implementation, iterative training can be performed on the target neural network according to the loss value so as to adjust the network parameter value of the target neural network; and under the condition that the preset training cutoff condition is met, finishing the training of the target neural network, taking the network parameter value obtained at the moment as the target network parameter value corresponding to the target neural network, and taking the target neural network finished by the training at the moment as the trained target neural network.

In one embodiment, the predicted offset information includes an offset of the predicted lane line in at least one image direction.

Here, the predicted shift amount information may include a shift amount in the image width direction and/or a shift amount in the image height direction.

In specific implementation, for each predicted lane line, the offset branch in the target neural network may fix one direction when determining the predicted offset corresponding to the predicted lane line, and output the predicted offset of the predicted lane line in another direction compared with the lane line in the previous frame image. For example, the image height direction is fixed, and for each predicted lane line, at each image height, the amount of shift thereof in the image width direction with respect to the lane line in the previous frame image may be output.

Fig. 6 is a schematic diagram of output prediction offset information according to an embodiment of the disclosure. Where X denotes an amount of shift in the image width direction, and each image height in the Y image height direction, the amount of shift in the X direction of each predicted lane line may be changed in a gradual manner as the image height increases or decreases. In fig. 6, the different offset amounts may be represented by different colors, the offset amounts in the X direction represented by the different colors may be shown as color indication bars, and the predicted offset amounts corresponding to the predicted lane line 1, the predicted lane line 2, the predicted lane line 3, and the predicted lane line 4 are shown in fig. 6.

Further, the unit of the offset amount in the X direction may be related to the down-sampling multiple, for example, when the preset sampling multiple is 8 times, the unit may be 8 × the offset amount; when the preset sampling multiple is 4 times, the unit may be 4 × offset.

In addition, in the process of constructing the information of the target neural network about the predicted offset by using the predicted offset information of each predicted lane line and the standard offset information corresponding to each predicted lane line, the smooth loss of the predicted smooth offset can be constructed by using the labeled offset information and the predicted offset information, and the target neural network is trained by using the smooth loss, so that the situation that the offsets corresponding to a plurality of lane points adjacent to the same lane line are changed in a jumping manner in the process of outputting the predicted offset information of the lane line in the image to be recognized by the trained target neural network is ensured.

For example, with respect to the predicted shift amount information corresponding to the lane line 5, on the shift amount information corresponding to the image heights 10 to 15, there will not occur a case where the shift amount of the lane point corresponding to the image height 10 is-2.5, the shift amount of the lane point corresponding to the image height 12 is-1, the shift amount of the lane point corresponding to the image height 13 is-1, the shift amount of the lane point corresponding to the image height 14 is-6, and the shift amount of the lane point corresponding to the image height 15 is 0, where the shift amount difference between the shift amount of the lane point corresponding to the image height 14 and the image height 15 is-6 and the shift amount-1.5 corresponding to the image height 12 is much larger than the shift amount difference between the shift amount-2.5 corresponding to the image height 10 and the shift amount-1.5 corresponding to the image height 12, and the shift amount-1.5 corresponding to the image height 12 and the shift amount-1.1 corresponding to the image height 13 The predicted offset information corresponding to the lane line 5 will have a jump-type change in the offset of the lane point corresponding to the image height 14, which is not reasonable in the offset information corresponding to the actual upper and lower frame lane lines, and the offset information corresponding to the actual upper and lower frame lane lines should be smoothly changed.

Therefore, the target neural network is trained by using the smoothing loss, so that the capability of predicting the smoothing of the target neural network can be improved, that is, the trained target neural network can adjust the predicted offset-6 at the lane point corresponding to the image height 14, and if the output offset after adjustment is-0.5, the smooth predicted offset information corresponding to each lane line shown in fig. 6 is obtained.

In one embodiment, the sample image of the previous frame of the sample identification image may also be obtained by:

and performing random translation and/or rotation operation on the sample identification image to obtain a previous frame sample image of the sample identification image.

Here, the random translation operation may be performed to translate the positions of the pixel points in the sample image, that is, to translate the positions of the pixel points corresponding to the lane lines in the sample identification image, so as to obtain an image different from the sample identification image. The random rotation operation can realize the rotation of the positions of the pixel points in the sample image, and an image different from the sample identification image can be obtained.

Fig. 7 is a schematic flowchart illustrating a process of training a target neural network according to an embodiment of the present disclosure. Fig. 7 shows a process of training a target neural network by using a plurality of sample recognition images, wherein the target neural network may determine, after performing image recognition on an nth frame sample recognition image of the plurality of sample recognition images, predicted thermodynamic diagram feature information corresponding to the nth frame sample recognition image, each predicted lane line corresponding to the sample recognition image, predicted offset information of each predicted lane line, and a prediction tracking result of each predicted lane line, and then may determine a thermodynamic diagram prediction loss value corresponding to the nth frame sample recognition image based on the predicted thermodynamic diagram feature information corresponding to the nth frame sample recognition image and an annotated thermodynamic diagram feature information corresponding to the nth frame sample recognition image, may determine a predicted thermodynamic diagram loss value corresponding to the target neural network in the nth frame sample recognition image based on the predicted lane line corresponding to the nth frame sample recognition image and an annotated lane line corresponding to the nth frame sample recognition image, determining a lane line prediction loss value corresponding to the N-th frame sample identification image of the target neural network, determining a shift amount information prediction loss value corresponding to the N-th frame sample identification image of the target neural network based on prediction shift amount information of each predicted lane line corresponding to the N-th frame sample identification image and labeling shift amount information corresponding to each predicted lane line, and determining a tracking loss value corresponding to the N-th frame sample identification image of the target neural network based on a prediction tracking result of each predicted lane line corresponding to the N-th frame sample identification image and a labeling tracking result corresponding to each predicted lane line, and similarly, according to the above steps, determining each loss value corresponding to the N + 1-th frame sample identification image. Then, performing iterative training on the target neural network by using each loss value corresponding to the sample identification image of the Nth frame and each loss value corresponding to the sample identification image of the (N + 1) th frame together to realize adjustment of the network parameter value of the target neural network; and obtaining the trained target neural network under the condition of meeting the training cutoff condition.

In fig. 7, if the previous frame image is the first frame image, the loss may be constructed without using the offset corresponding to the frame image, and only the loss of information other than the offset may be constructed to train the target neural network.

It will be understood by those skilled in the art that in the method of the present invention, the order of writing the steps does not imply a strict order of execution and any limitations on the implementation, and the specific order of execution of the steps should be determined by their function and possible inherent logic.

Based on the same inventive concept, the embodiment of the present disclosure further provides a lane line tracking device corresponding to the lane line tracking method, and as the principle of solving the problem of the device in the embodiment of the present disclosure is similar to that of the lane line tracking method in the embodiment of the present disclosure, the implementation of the device may refer to the implementation of the method, and repeated details are not repeated.

As shown in fig. 8, a schematic diagram of a lane line tracking apparatus provided in an embodiment of the present disclosure includes:

an obtaining module 801, configured to obtain a previous frame of image of an image to be identified, and obtain a first lane line by performing image identification on the previous frame of image;

a first determining module 802, configured to determine, based on the previous frame image and the image to be recognized, a second lane line in the image to be recognized, and offset information of the second lane line with respect to a first lane line closest to the second lane line;

the second determining module 803 is configured to determine, based on the offset information corresponding to each second lane line and each first lane line, a first lane line that matches at least one second lane line, and obtain a tracking result of the at least one second lane line.

In a possible implementation manner, the first determining module 802 is configured to determine a first image feature corresponding to the image to be recognized based on the previous frame image, the first thermodynamic diagram feature information corresponding to the previous frame image, and the image to be recognized;

In a possible implementation manner, the second determining module 803 is configured to determine, based on the second thermodynamic diagram feature information, an initial pixel point corresponding to each of the second lane lines;

In a possible implementation manner, the second determining module 803 is configured to determine, for each second lane line, an offset value corresponding to each initial pixel point in the second lane line based on offset information corresponding to the second lane line;

In a possible implementation manner, the second determining module 803 is configured to, for each second lane line, screen, based on the second thermodynamic diagram feature information, a pixel point to be matched from target pixel points of the second lane line;

In a possible embodiment, the second determining module 803 is further configured to, after determining the first lane line matching the second lane line, determine that the second lane line with the shortest distance to the first lane line matches the first lane line if a plurality of second lane lines match the same first lane line.

In a possible embodiment, the second determining module 803 is configured to, when the shortest second distance is smaller than the first preset value, take the first lane line corresponding to the shortest second distance as the lane line matching the second lane line.

In a possible embodiment, the second determining module 803 is further configured to take the second lane line as a new lane line if the shortest second distance is greater than the first preset value.

In a possible embodiment, the apparatus further comprises:

a deleting module 804, configured to determine, for each first lane line that is not successfully matched, the number of times that the first lane line is successfully unmatched after the tracking result is obtained; the times of the unmatched success are used for representing the times of the corresponding first lane line which is not matched with the second lane line in the continuous multi-frame images to be recognized;

In a possible embodiment, the apparatus further comprises:

a training module 805 configured to train the target neural network by:

In a possible implementation, the training module 805 is further configured to obtain a sample image of a frame before the sample identification image by:

In a possible implementation manner, the obtaining module 801 is configured to perform image recognition on the previous frame image and determine a second image feature corresponding to the previous frame image when the previous frame image is the first frame image;

In a possible implementation manner, the first determining module 802 is configured to perform downsampling processing on the image to be identified and the previous frame of image according to a preset sampling multiple, so as to obtain a new image to be identified and a new previous frame of image;

The description of the processing flow of each module in the device and the interaction flow between the modules may refer to the related description in the above method embodiments, and will not be described in detail here.

An embodiment of the present disclosure further provides a computer device, as shown in fig. 9, which is a schematic structural diagram of a computer device provided in an embodiment of the present disclosure, and the computer device includes:

a processor 91 and a memory 92; the memory 92 stores machine-readable instructions executable by the processor 91, the processor 91 being configured to execute the machine-readable instructions stored in the memory 92, the processor 91 performing the following steps when the machine-readable instructions are executed by the processor 91: s101: acquiring a previous frame image of an image to be identified, and carrying out image identification on the previous frame image to obtain a first lane line; s102: determining a second lane line in the image to be recognized and offset information of the second lane line relative to the closest first lane line based on the previous frame image and the image to be recognized, and S103: and determining the first lane line matched with the at least one second lane line based on the offset information corresponding to each second lane line and each first lane line, and obtaining the tracking result of the at least one second lane line.

The memory 92 includes a memory 921 and an external memory 922; the memory 921 is also referred to as an internal memory, and temporarily stores operation data in the processor 91 and data exchanged with an external memory 922 such as a hard disk, and the processor 91 exchanges data with the external memory 922 through the memory 921.

For the specific execution process of the instruction, reference may be made to the steps of the lane line tracking method in the embodiments of the present disclosure, and details are not described here.

The embodiments of the present disclosure also provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program performs the steps of the lane line tracking method described in the above method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

The computer program product of the lane line tracking method provided in the embodiments of the present disclosure includes a computer-readable storage medium storing a program code, where instructions included in the program code may be used to execute the steps of the lane line tracking method described in the above method embodiments, which may be referred to in the above method embodiments specifically, and are not described herein again.

The computer program product may be embodied in hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working process of the apparatus described above may refer to the corresponding process in the foregoing method embodiment, and is not described herein again. In the several embodiments provided in the present disclosure, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implementing, and for example, a plurality of units or components may be combined, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present disclosure. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

Finally, it should be noted that: the above-mentioned embodiments are merely specific embodiments of the present disclosure, which are used for illustrating the technical solutions of the present disclosure and not for limiting the same, and the scope of the present disclosure is not limited thereto, and although the present disclosure is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive of the technical solutions described in the foregoing embodiments or equivalent technical features thereof within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present disclosure, and should be construed as being included therein. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims

1. A lane line tracking method, comprising:

2. The method according to claim 1, wherein the determining a second lane line in the image to be recognized and offset information of the second lane line relative to a closest first lane line based on the previous frame image and the image to be recognized comprises:

3. The method according to claim 2, wherein the determining a first lane line matching at least one of the second lane lines based on the offset information corresponding to each of the second lane lines and each of the first lane lines to obtain a tracking result of at least one of the second lane lines comprises:

4. The method according to claim 3, wherein the determining a target pixel point corresponding to each second lane line based on the offset information corresponding to each second lane line and the initial pixel point corresponding to each second lane line respectively comprises:

5. The method of claim 3, wherein determining the first lane line matching at least one of the second lane lines based on the target pixel point corresponding to each of the second lane lines and the pixel point corresponding to each of the first lane lines comprises:

6. The method of claim 5, after determining the first lane line that matches the second lane line, further comprising:

7. The method of claim 5, wherein determining a first lane line matching the second lane line based on the determined second distance comprises:

8. The method of claim 7, wherein determining a first lane line matching the second lane line based on the determined second distance further comprises:

9. The method of claim 1, after obtaining the tracking result, further comprising:

10. The method according to any one of claims 1 to 9, wherein the lane line tracking method is performed by a target neural network trained using a plurality of sample recognition images.

11. The method of claim 10, wherein the target neural network is trained using the steps of:

12. The method of claim 11, wherein the predicted offset information comprises an offset of the predicted lane line in at least one image direction.

13. The method of claim 11, wherein the sample image of the frame preceding the sample identification image is obtained by:

14. The method of claim 1, wherein obtaining a first lane line from image recognition of the previous frame of image comprises:

15. The method according to claim 1, wherein the determining, based on the previous frame image and the image to be recognized, a second lane line in the image to be recognized and offset information corresponding to the second lane line comprises:

and determining a second lane line in the image to be recognized based on the new previous frame image and the new image to be recognized, and the offset information of the second lane line corresponding to the first lane line closest to the second lane line.

16. A lane line tracking apparatus, comprising:

17. A computer device, comprising: a processor, a memory storing machine readable instructions executable by the processor, the processor for executing the machine readable instructions stored in the memory, the processor performing the steps of the lane line tracking method of any one of claims 1 to 15 when the machine readable instructions are executed by the processor.

18. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a computer device, performs the steps of the lane line tracking method according to any one of claims 1 to 15.