WO2023093124A1 - Lane line tracking method and apparatus, and computer device, storage medium and computer program product - Google Patents

Abstract

Provided in the embodiments of the present disclosure are a lane line tracking method and apparatus, a computer device, a storage medium and a computer program product. The method comprises: acquiring the previous frame of image of an image to be subjected to recognition, and first lane lines, which are obtained by performing image recognition on the previous frame of image; on the basis of the previous frame of image and the image to be subjected to recognition, determining second lane lines in the image to be subjected to recognition, and offset information of each second lane line relative to the closest first lane line; and on the basis of the offset information corresponding to each second lane line, and each first lane line, determining a first lane line which matches at least one second lane line, so as to obtain a tracking result of the at least one second lane line.

Description

A lane line tracking method, device, computer equipment, storage medium and computer program product

Cross References to Related Applications

This disclosure is based on the Chinese patent application with the application number 202111433344.8, the application date is November 29, 2021, and the application name is "A lane line tracking method, device, computer equipment and storage medium", and requires the Chinese patent application Priority, the entire content of the Chinese patent application is hereby incorporated by reference into this disclosure.

technical field

The present disclosure relates to but not limited to the technical field of automatic driving, and in particular relates to a lane line tracking method, device, computer equipment, storage medium and computer program product.

Background technique

Lane line detection and tracking is a necessary process in automatic driving. The accuracy of detection and tracking results is related to the safety of automatic driving. However, the lane line detection and tracking technology in related technologies is not only complicated in the detection and tracking process, but The accuracy of the finalized tracking results is not high.

Contents of the invention

Embodiments of the present disclosure at least provide a lane line tracking method, device, computer equipment, storage medium, and computer program product.

An embodiment of the present disclosure provides a lane line tracking method, including:

Acquiring the previous frame image of the image to be recognized, and the first lane line obtained by performing image recognition on the previous frame image;

Based on the previous frame image and the image to be recognized, determine a second lane line in the image to be recognized, and offset information of the second lane line relative to the nearest first lane line;

Based on the offset information corresponding to each of the second lane lines and each of the first lane lines, determine a first lane line that matches at least one of the second lane lines, and obtain at least one of the second lanes Line tracking results.

An embodiment of the present disclosure also provides a lane line tracking device, including:

The acquiring part is configured to acquire a previous frame image of the image to be recognized, and a first lane line obtained by performing image recognition on the previous frame image;

The first determination part is configured to determine, based on the previous frame image and the image to be recognized, a second lane line in the image to be recognized, and a relative distance between the second lane line and the closest first lane Line offset information;

The second determining part is configured to determine a first lane line matching at least one of the second lane lines based on the offset information corresponding to each of the second lane lines and each of the first lane lines, A tracking result of at least one second lane line is obtained.

An embodiment of the present disclosure also provides a computer device, a processor, and a memory, where the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the machine-readable instructions stored in the memory, When the machine-readable instructions are executed by the processor, when the machine-readable instructions are executed by the processor, the above first aspect, or the steps in any possible implementation manner of the first aspect are executed.

An embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed, the above-mentioned first aspect, or any possible implementation manner in the first aspect is executed. in the steps.

An embodiment of the present disclosure provides a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, a part or part of the above-mentioned method is implemented. All steps.

In the embodiment of the present disclosure, in the process of detecting and tracking the second lane line in the image to be recognized, by combining the previous frame image, it can make the determination of the second lane line and the determination of the second lane line relative to the closest distance When the offset information of the first lane line can be combined with the image features of the previous frame image, it is beneficial to improve the accuracy of the determined second lane line and the offset information corresponding to the second lane line; then, based on The offset information with higher accuracy tracks the second lane line, which can improve the accuracy of the determined tracking result.

In order to make the above objects, features and advantages of the embodiments of the present disclosure more comprehensible, exemplary embodiments will be described in detail below together with the accompanying drawings.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present disclosure more clearly, the following will briefly introduce the accompanying drawings used in the embodiments. The accompanying drawings here are incorporated into the specification and constitute a part of the specification. The drawings show the embodiments consistent with the present disclosure, and are used together with the description to explain the technical solution of the present disclosure. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as limiting the scope. For those skilled in the art, they can also make From these drawings other related drawings are obtained.

FIG. 1 is a schematic diagram of an implementation flow of a lane line tracking method provided by an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of a recognized first lane line and a second lane line provided by an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of identifying a second lane line by performing image recognition on an image to be recognized by a reasoning module in a target neural network provided by an embodiment of the present disclosure;

FIG. 4 is a schematic diagram of an implementation process for image recognition of an image to be recognized by a reasoning module in a target neural network provided by an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of an implementation process for determining a tracking result corresponding to a second lane line by a tracking module in a target neural network provided by an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of output prediction offset information provided by an embodiment of the present disclosure;

FIG. 7 is a schematic flow diagram of training a target neural network provided by an embodiment of the present disclosure;

FIG. 8 is a schematic diagram of the composition and structure of a lane line tracking device provided by an embodiment of the present disclosure;

FIG. 9 is a schematic structural diagram of a computer device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only It is a part of the embodiments of the present disclosure, but not all of them. The components of the disclosed embodiments generally described and illustrated herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort shall fall within the protection scope of the present disclosure.

In addition, the terms "first", "second" and the like in the description and claims in the embodiments of the present disclosure and the above drawings are used to distinguish similar objects, and not necessarily used to describe a specific sequence or sequence. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments described herein can be practiced in sequences other than those illustrated or described herein.

"Plural or several" mentioned herein means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B may indicate: A exists alone, A and B exist simultaneously, and B exists independently. The character "/" generally indicates that the contextual objects are an "or" relationship.

It has been found through research that lane line detection and tracking are necessary processes in automatic driving, and the accuracy of detection and tracking results is related to the safety of automatic driving. However, most lane line detection and tracking technologies in related technologies first use neural networks Detect the lane lines, and then use lengthy traditional algorithms (such as Kalman filter and Hungarian algorithm) to post-process the detected lane lines, so as to realize the tracking of the lane lines.

In the process of lane line tracking using traditional algorithms, complex sensor models also need to be incorporated. This not only increases the complexity of the detection and tracking process, but also makes the detection results limited by the detection results of each sensor model, which greatly affects the accuracy of the lane line tracking results.

Based on the above studies, embodiments of the present disclosure provide a lane line tracking method, device, computer equipment, storage medium, and computer program product. During the process of detecting and tracking the second lane line in the image to be recognized, by combining the previous One frame of image can make it possible to combine the image features of the previous frame image when determining the second lane line and determining the offset information of the second lane line relative to the nearest first lane line, which is beneficial to improve the determination The accuracy of the second lane line and the offset information corresponding to the second lane line; then, tracking the second lane line based on the more accurate offset information can improve the accuracy of the determined tracking result.

The defects in the above solutions are all the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions to the above problems proposed by the embodiments of the present disclosure below should be Contributions made by the inventors to the disclosure during the course of the disclosure.

It should be noted that like numerals and letters denote similar items in the following figures, therefore, once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

In order to facilitate the understanding of the embodiments of the present disclosure, a lane line tracking method disclosed in the embodiments of the present disclosure is firstly introduced in detail. The execution subject of the lane line tracking method provided in the embodiments of the present disclosure is generally a computer device with certain computing power , in some possible implementation manners, the lane line tracking method may be implemented in a manner in which a processor invokes computer-readable instructions stored in a memory.

The lane line tracking method provided by the embodiment of the present disclosure will be described below by taking the execution subject as the target neural network as an example.

Fig. 1 is a schematic diagram of the implementation flow of a lane line tracking method provided by an embodiment of the present disclosure. As shown in Fig. 1, the method may include the following steps S101 to S103:

S101: Acquire a previous frame image of the image to be recognized, and a first lane line obtained by performing image recognition on the previous frame image.

Here, the image to be recognized may be an image including lane lines captured by the camera device installed on the target vehicle, and the previous frame image may be captured by the camera device or generated based on the image to be recognized, which is not limited here .

During implementation, after the image to be recognized is acquired, the image to be recognized may be input into the target neural network, and the target neural network is used to perform image recognition on the previous frame image to obtain each first lane line in the previous frame image.

S102: Based on the previous frame image and the image to be recognized, determine the second lane line in the image to be recognized, and the offset information of the second lane line relative to the closest first lane line.

FIG. 2 is a schematic diagram of an identified first lane line and a second lane line provided by an embodiment of the present disclosure, wherein the first lane line includes a first lane line 11 , a first lane line 12 , and a first lane line 13 and the first lane line 14 , the second lane line includes a second lane line 21 , a second lane line 22 , a second lane line 23 and a second lane line 24 .

The offset information is used to characterize the offset between the second lane line and the nearest first lane line. In some implementations, the offset information may correspond to each first lane point in the second lane line The positional offset information between the pixel point of and the pixel point corresponding to each second lane point in the nearest first lane line. Wherein, the lane point can be a key point on the lane line, the offset information can be the offset of each second lane line corresponding to the width direction of the image, and the target neural network predicts the lane point corresponding to each lane line For the offset, for the pixel corresponding to each lane point, the information of the pixel in the image height direction can be fixed, and the offset information of the pixel in the image width direction can be determined. In this way, when the target neural network performs subsequent lane line matching, it can only calculate the offset information in one direction (image width) to adjust the position of the lane line, and determine the previous lane line that matches the lane line. The lane line in the frame image reduces the amount of calculation, which is beneficial to improve the speed and efficiency of lane line tracking.

During implementation, the target neural network can use the convolutional layer to convolve the previous frame image and the image to be recognized, respectively, to obtain the feature maps corresponding to the two images, and combine the obtained two feature maps to obtain the image to be recognized The corresponding target feature map. Wherein, the feature map corresponding to each image obtained by convolution may include a thermal feature map, a depth feature map, and the like. Afterwards, based on the image features included in the obtained target feature map, the image features corresponding to each second lane line can be determined, and then each second lane line in the image to be recognized can be determined.

The target neural network can also determine the first lane line in the previous frame image according to the image features included in the target feature map; according to the determined second lane line and the first lane line, each second lane line can be determined respectively Corresponding to the first lane line with the closest distance, and further, the offset information of each second lane line relative to the first lane line with the closest distance can be obtained.

Among them, the offset information can be predicted by the offset branch in the target neural network.

S103: Based on the offset information corresponding to each second lane line and each first lane line, determine a first lane line matching at least one second lane line, and obtain a tracking result of at least one second lane line.

Here, the tracking result is a result representing whether there is a lane line matching the second lane line in the first lane line, that is, it can represent whether there is a second lane line identical to the first lane line in the image to be recognized

For each first lane line, there may be a piece of lane line identification information for identifying the identity of the lane line. For example, the lane line identification information may include a lane line number. During implementation, if there is a matching first lane line in the second lane line, the second lane line can be regarded as the same lane line as the first lane line, and the lane line number of the first lane line can be used as the second lane The lane line number of the line, and vice versa, a new lane line number can be generated for the second lane line.

For each second lane line, the offset information of the second lane line can be used to adjust the position of the lane line of the second lane line to obtain each adjusted second lane line, and then, the The second lane line is matched with the first lane line, and it is determined whether there is a first lane line matching the second lane line.

If so, the lane line identification information corresponding to the first lane line matching the second lane line can be used as the lane line identification information of the second lane line, and the lane line identification information can be used as the second lane line identification information tracking results. For example, the first lane line includes three first lane lines whose numbers are 1, 2, and 3 respectively. When it is determined that the lane line number of the first lane line matching a certain second lane line is 2, then it can be Let the lane line number 2 be the lane line number of the second lane line.

In addition, the second lane line may also be used to replace the first lane line matched with it stored in the local database. Wherein, the local database stores each identified lane line, the image corresponding to the lane line, image features and lane line identification information. In some implementations, all information related to the lane line can be stored in the local database. For example, the second lane line may be used to replace the matched first lane line, or each second lane point corresponding to the second lane line may be directly used to replace the first lane point corresponding to the first lane line.

If not, it may be determined that the second lane line is a newly recognized lane line, and lane line identification information corresponding to the lane line is generated as a tracking result of the second lane line. In addition, the second lane line can also be stored in a local database. Continuing the above example, when it is determined that there is no first lane line matching a certain second lane line, a new lane line number 4 may be generated for the second lane line.

In this way, first of all, only one target neural network is needed to realize the detection and tracking of lane lines, which improves the correlation between detection and tracking, and the combination of the two is conducive to improving the accuracy of detection. Moreover, the above-mentioned embodiment does not need to be combined with a multi-sensor model, and detection and tracking can be realized by using the acquired image to be recognized, which can not only reduce the complexity of detection and tracking, but also improve the universality of the lane line tracking method. In addition, the detection and tracking steps are performed in the same target neural network, which can reduce part of the post-processing process for the tracking process, thereby reducing the impact of the detection results of the sensor model on the tracking results and improving tracking accuracy. In addition, in the process of tracking the second lane line in the image to be recognized, by combining the previous frame image, it is possible to combine the image features of the previous frame image when determining the offset information corresponding to the second lane line , thus, it is beneficial to improve the accuracy of the determined offset information, and then, tracking the second lane line based on the more accurate offset information can improve the accuracy of the determined tracking result.

In some embodiments, S102 may be implemented according to the following steps:

S102-1: Based on the previous frame image, the first thermal map feature information corresponding to the previous frame image, and the image to be recognized, determine the first image feature corresponding to the image to be recognized.

Here, the heat map feature information can reflect the heat value of each pixel in the image, and the heat map feature information can be expressed in the form of a heat map. The first heat map feature information corresponds to the first heat map, and the pixels corresponding to different lane lines Points have different heating values, for example, for a road image, the heating value of each lane point in the lane line is higher than the heating value of other points on the road.

The feature information of the first heat map corresponding to the previous frame image may be obtained by performing image recognition on the previous frame image by the target neural network.

During implementation, the target neural network can perform convolution operations on each image after obtaining the previous frame image, the first heat map corresponding to the previous frame image, and the image to be recognized, to obtain the convolution results corresponding to each image, and then , the convolution results can be fused to obtain the initial feature map corresponding to the initial image features, and then the encoder is used to perform feature encoding on the initial image features to obtain the first feature map corresponding to the image to be recognized, wherein the first feature map includes The first image feature corresponding to the image to be recognized, the first image feature may be a feature vector.

S102-2: Based on the first image feature, determine the second thermal map feature information corresponding to the image to be recognized.

During implementation, the target neural network may perform further feature processing on the determined first feature map to obtain feature information of the second heat map. For example, a decoding operation may be performed on the first feature map to determine the thermal value corresponding to each pixel in the first feature map, and then to determine the second thermal map feature information corresponding to the image to be recognized.

S102-3: Based on the second heat map feature information and the first image feature, determine a second lane line in the image to be recognized and offset information corresponding to the second lane line.

In this step, the heat value of each pixel point in the second heat map can be determined first based on the feature information of the second heat map, and then the initial pixel point corresponding to the second lane point in the second lane line can be determined , and then, according to the determined position of each initial pixel point in the second heat map, feature clustering can be performed on the features corresponding to each second lane point, and the second lane line to which each second lane point belongs can be determined . During implementation, the feature embedding branch in the target neural network can be used to perform feature clustering on the features corresponding to each second lane point to determine each second lane line, wherein the feature embedding branch can be the embedding ( Embedding) layer.

Here, based on the previous frame image, the first thermal map feature information corresponding to the previous frame image, and the image to be recognized, the obtained first image features can not only contain the image features of the image to be recognized, but also can be combined with the previous frame image And its corresponding first heat map feature information improves the richness of feature information contained in the first image feature, and then, based on the first image feature containing rich feature information, accurate second heat map features can be obtained information, and because the heat value of the lane line is different from the heat value of other areas in the image where it is located, based on the feature information of the second heat map, the second lane line can be accurately determined, and then the first frame image containing the previous frame image can be used An image feature that can accurately determine offset information corresponding to the second lane line.

In some implementations, the maximum number of recognized lane lines can also be preset, and if the number of recognized lane lines in any frame of the image to be recognized is greater than the maximum number, abnormal prompt information can be generated. That is, during the driving process of the target vehicle, the number of lane lines on the road where it is located is limited. If the number of identified lane lines is too large, there may be a problem of detection error, which will affect the safety of automatic driving. Therefore, , by setting the maximum lane line and generating abnormal prompt information, the safety of automatic driving can be improved.

The target neural network can also determine the first lane lines in the previous frame image according to the feature processing of the first image features, and then determine the first lane lines and the second lane lines based on the first lane lines and the second lane lines. The distance between each second lane line, according to the above-mentioned determined distance, can determine the first lane line with the shortest distance corresponding to each second lane line. Furthermore, for each second lane line, offset information of the second lane line relative to the nearest first lane line can be determined. During implementation, the step of determining the second lane line, the offset information corresponding to the second lane line, and the second heat map feature information may be completed by an inference module in the target neural network. Figure 3 is a schematic diagram of the recognition of the second lane line by the reasoning module in a target neural network provided by an embodiment of the present disclosure for image recognition to determine the second lane line, as shown in Figure 3, the reasoning module can respectively correspond to the previous frame image The first thermal map 31, the previous frame image 32 and the image to be recognized 33 are convolved to obtain the convolution results corresponding to each image, and the convolution results are fused to obtain the initial feature map 34 corresponding to the initial image features, Utilize the encoder 35 to carry out feature encoding to the initial image feature again, obtain the first feature map corresponding to the image to be recognized (that is, the first feature map 36 corresponding to the first image feature), carry out further feature processing to the first feature map 36, Obtain the second heat map feature information, based on the second heat map feature information, can determine the second heat map 37 corresponding to the image to be recognized, based on the heat value of each pixel in the second heat map 37, can determine the second Lane lines 38; the reasoning module can also determine each first lane line in the previous frame image according to the feature processing of the first image features, and then can determine the relative distance of each second lane line 38 to the nearest first lane line. The offset information of the lane line, that is, the offset information 39 corresponding to the second lane line.

During implementation, the target neural network may include a reasoning module and a tracking module, wherein the tracking module is used to determine the first lane line that matches the second lane line in the image to be recognized, and the reasoning module is used to determine each acquired image to be recognized. Recognize the lane line corresponding to the image, the offset information of the lane line and the feature information of the heat map, and after determining the above information corresponding to the image to be recognized, input the above information into the tracking module to complete the second lane in the image to be recognized line tracking.

FIG. 4 is a schematic diagram of an implementation process of image recognition of an image to be recognized by an inference module in a target neural network provided by an embodiment of the present disclosure, wherein the image 41 used for recognition includes any frame captured by an automatic driving device (target vehicle) image, input the Nth frame of the image to be recognized 42 into the target neural network 43, the image feature 44 corresponding to the Nth frame of the image to be recognized can be obtained, and then the thermal map feature information 45 and the Nth frame of the Nth image to be recognized can be obtained. The offset information 46 of the lane line in the image to be recognized in the frame, the feature embedding branch 47 is used to perform feature clustering on the feature of the initial pixel point corresponding to the lane point (that is, the image feature 44), and determine the Nth frame to be processed currently. Recognize the lane line 48 in the image. In addition, in FIG. 4, the thermal map feature information 45 corresponding to the Nth frame of the image to be recognized is input to the target neural network 43 together with the Nth frame of the image to be recognized, for the N+1 frame Image recognition is performed on the image to be recognized.

In some embodiments, for the step of performing image recognition on the previous frame image, if the previous frame image is the first frame image, the previous frame image of the previous frame image and its corresponding thermal map feature information may not be used , directly perform image recognition on the previous frame image, and determine the second image feature corresponding to the previous frame image. Furthermore, the second image feature can be processed to determine the first thermal map feature information corresponding to the previous frame image, and then, based on the first thermal map feature information and the second image feature, the first thermal map feature information in the previous frame image can be determined. lane line. In this way, the recognition processing of the first frame image can be carried out directly based on the image, and the first lane line can be obtained without obtaining the corresponding previous frame image and the corresponding heat map feature information, which improves the flexibility of the target neural network processing. Regarding the step of determining the first lane line in the previous image frame based on the feature information of the first heat map and the feature of the second target image, reference may be made to the step of determining the second lane line in the above-mentioned embodiment.

And, when the previous frame image is the first frame image, since there is no previous frame image in the first frame image, there is no offset between the lane line in the first frame image and the lane line in the previous frame image, nor The lane line in the first frame image needs to be matched with the lane line in the previous frame image, therefore, the offset information corresponding to the first frame image output by the target neural network can be deleted.

In one embodiment, for S103, it can be implemented according to the following steps:

S103-1: Based on the feature information of the second heat map, determine the initial pixel points corresponding to each second lane line.

Here, the initial pixel is the pixel in the second heat map.

During implementation, based on the feature information of the second heat map, the heat value of each pixel in the second heat map can be determined, and then it can be determined whether each pixel belongs to a lane point, and the pixel belonging to the lane point Points are used as initial pixel points corresponding to each second lane line.

S103-2: Based on the offset information corresponding to each second lane line and the initial pixel point corresponding to each second lane line, respectively determine a target pixel point corresponding to each second lane line.

In this step, for each second lane line, the offset information corresponding to each second lane point in the second lane line can be used to determine the initial pixel point corresponding to each second lane point in the second lane line Adjust the position of each initial pixel to determine the target position corresponding to each initial pixel, and use the initial pixel at the target position as the target pixel corresponding to the second lane line.

In some embodiments, the target pixel points corresponding to each second lane line can be determined according to the following steps 1 to 2:

Step 1. For each second lane line, based on the offset information corresponding to the second lane line, determine the offset value corresponding to each initial pixel point in the second lane line.

Here, the offset information corresponding to the second lane line can represent the offset value corresponding to each second lane point in the second lane line. In this step, for each second lane line, the offset value of the initial pixel point corresponding to each second lane point in the second lane line may be determined according to the offset information corresponding to the second lane line.

Step 2: Determine the target pixel point corresponding to the second lane line based on the offset value corresponding to each initial pixel point.

Here, the offset value of each initial pixel point can be used to adjust the position of the initial pixel point to determine the target pixel point corresponding to each initial pixel point, that is, to obtain each first pixel point in the second lane line The target pixel corresponding to the second lane point. Furthermore, target pixel points corresponding to each second lane point in each second lane line may be determined based on the offset information corresponding to each second lane line and the corresponding initial pixel point respectively.

In this way, based on the offset information corresponding to the second lane line, the offset value of each initial pixel point in the second lane line can be obtained, and then the position of each initial pixel point can be adjusted by using the offset value, Get the exact target pixel corresponding to each initial pixel

S103-3: Based on the target pixel point corresponding to each second lane line and the pixel point corresponding to each first lane line, determine the first lane line matching at least one second lane line, and obtain at least one second lane line tracking results.

During implementation, for each second lane line and each first lane line, the position of the target pixel point corresponding to each second lane point of the second lane line, and each first lane in the first lane line can be The position of each pixel point corresponding to the point determines the initial distance between each target pixel point corresponding to the second lane line and each pixel point in the first lane line. Here, the initial distance can be the distance between the target pixel point at the same height and the pixel point in the first lane line, that is, the target pixel point corresponding to each initial distance and the corresponding distance between the first lane line Pixels are at the same height.

Furthermore, the target distance between the second lane line and the first lane line corresponding to each target pixel point can be determined according to the determined initial distance corresponding to each target pixel point, and then, the distance between each second lane line and the first lane line can be determined. Target distance between each first lane line.

Then, for each first lane line, the second lane line with the shortest target distance between it can be determined, and then, the second lane line can be used as the lane line matching the first lane line, and the Tracking results of the second lane line. Based on this, the tracking result of each second lane line can be determined.

Alternatively, for each second lane line and each first lane line, it may also be based on the characteristics of the target pixel points corresponding to each second lane point in the second lane line, and each first lane line in the first lane line The characteristics of each pixel point corresponding to the point determine the similarity between each first lane line and each second lane line, and then determine the tracking result of each second lane line based on the determined similarity.

Here, since the heat value corresponding to the second lane line is different from the heat value of other regions in the image to be recognized, based on the feature information of the second heat map, the heat value of each pixel can be determined, and then the second heat value can be determined. The initial pixel corresponding to the two-lane line. The offset information can reflect the prediction error when predicting the second lane line. Using the determined offset information to adjust the position of the initial pixel point can realize the adjustment of the predicted position of the second lane line and determine the second lane line. The exact position of the lane line in the image to be recognized, and then, using the target pixel point corresponding to the second lane line and the pixel point corresponding to each first lane line, the second lane line and each first lane line can be accurately determined Therefore, performing lane line matching based on the determined distance can accurately determine the tracking result corresponding to the second lane line.

In some embodiments, S103-3 may be implemented according to the following steps:

S103-3-1: For each second lane line, based on the feature information of the second heat map, select the pixel points to be matched from the target pixel points of the second lane line.

Here, the pixel points to be matched are pixel points screened out from the target pixel points and used to determine the first lane line matching the second lane line. During implementation, for each second lane line, the thermal value corresponding to each target pixel point in the second lane line may be determined first. For each target pixel point in the second lane line, its corresponding initial pixel point can be determined, and then based on the second heat map feature information, the heat value corresponding to the initial pixel point can be determined, and then the The heat value is used as the heat value corresponding to the target pixel. In this way, the heat value corresponding to each target pixel point in the second lane line can be determined.

Then, according to the thermal value corresponding to each target pixel point, each target pixel point can be sorted in descending order, and the target pixel point whose sorting order is greater than the preset order is used as the pixel point to be matched.

Or, for each target pixel point in each second lane line, the confidence degree corresponding to each target pixel point can be determined first, and then, the confidence degree greater than Preset the first target pixel of the reliability threshold, and then determine the heat value of each first target pixel based on the second heat map feature information, and sort the first target pixels based on the heat value, and put The first target pixel whose sort order is greater than the preset order is used as the pixel to be matched.

S103-3-2: Based on the pixel points corresponding to each first lane line, respectively determine a first distance between each to-be-matched pixel point and each first lane line.

In this step, for each pixel point to be matched in the second lane line and each first lane line, based on the position of the pixel point to be matched and the pixels corresponding to each first lane point in the first lane line The position of the point, from each pixel point corresponding to the first lane line, filter out the pixel point with the same height as the pixel point to be matched, and then determine the initial distance between the pixel point to be matched and the pixel point distance, and use the initial distance as the first distance from the first lane line. Furthermore, from each pixel point corresponding to the first lane line, the pixel point having the same height as each pixel point to be matched can be screened out, and the initial pixel point corresponding to each pixel point to be matched can be determined based on the screened out pixel points. Thus, the first distance between each to-be-matched pixel point and each first lane line can be obtained.

S103-3-3: Based on the first distance corresponding to each to-be-matched pixel point, determine a second distance between the second lane line and each first lane line.

Here, the second distance may be the Euclidean distance between the two lanes. For the second lane line and each first lane line, the second lane line can be determined according to the first distance between each pixel point to be matched in the second lane line corresponding to the pixel point in the first lane line. A second distance between the lane marking and the first lane marking. During implementation, the average value of the first distance corresponding to each pixel point to be matched in the second lane line can be used as the second distance, or the smallest first distance can be selected as the second distance, or according to each The confidence corresponding to the pixels to be matched determines the weight corresponding to each pixel to be matched, and according to the weight corresponding to each pixel to be matched and its corresponding first distance, the first distance is summed to obtain the second The distance is not limited here.

In this way, the second distance between each second lane line and each first lane line is determined through the filtered pixel points to be matched, without using each target pixel point in the second lane line for calculation, reducing The calculation amount of determining the second distance is reduced, and the speed of determining the second distance is increased, which is beneficial to improving the speed of determining the tracking result.

S103-3-4: Based on the determined second distance, determine a first lane line matching the second lane line.

In this step, for each second lane line, the first lane line with the shortest second distance to the second lane line can be determined based on the second distance between the second lane line and each first lane line. Lane line, the first lane line is used as the lane line matching the second lane line.

In some embodiments, when multiple second lane lines are matched to the same first lane line, that is, for any first lane line, when multiple second lane lines are matched to the first lane line In the case of , each second lane line in the plurality of second lane lines can be compared with the second distance corresponding to the first lane line; the shortest second distance is determined, and it is determined that the shortest second distance corresponds to The second lane line of is the lane line that finally matches the first lane line. In this way, the two determined matching lane lines can be in one-to-one correspondence, and there is no other matching lane line, thereby improving the rationality and accuracy of the tracking result.

In addition, for each second lane line in the plurality of second lane lines except the second lane line on the final match, the second shortest second distance corresponding to each second lane line can be determined, and then, using the second The two shortest second distances, and the shortest second distances corresponding to other second lane markings determine whether there is a matching first lane marking among the plurality of second lane markings. Therefore, the matching result of each second lane line is determined.

In some embodiments, for each second lane line, after determining the matching first lane line, the second distance between the second lane line and the first lane line and the first preset Set value for comparison. Here, the first preset value is used to characterize the maximum distance between the matched first lane line and the second lane line. If the second distance is greater than the maximum distance, even if the second The distance is the shortest, and the two lane lines are also considered as mismatched lane lines. In this way, further judgment is made on the shortest second distance and the determined shortest first distance by using the first preset value, which improves the rationality and accuracy of the determined matching result.

During implementation, when it is determined that the shortest second distance corresponding to the first lane line is less than the first preset value, the first lane line can be used as the final distance of the second lane line corresponding to the shortest second distance. Match lane lines.

In some embodiments, when it is determined that the shortest second distance corresponding to the first lane line is greater than the first preset value, the second lane line corresponding to the shortest second distance can be used as a new lane line, or That is, it can be determined that there is no lane line matching the second lane line in the first lane line, thereby improving the accuracy of the determined tracking result.

In some embodiments, after determining the tracking results corresponding to each second lane line, for each first lane line stored in the local database, based on the tracking results, it is possible to determine the corresponding tracking results of each first lane line. The number of successful matches. Here, the number of failed matches is used to represent the number of times that the first lane line fails to match the second lane line in a row.

Then, for the above-mentioned first lane line, the number of failed matches can be compared with the second preset value, and if it is determined that the number of failed matches is greater than the second preset value, it means that the first lane line has been repeated multiple times If the matching is not successful, it means that the second lane line has probably disappeared within the driving range of the target vehicle, then the first lane line can be deleted, thereby reducing the number of stored lane lines, reducing the storage space occupied, and at the same time improving detection efficiency.

If the number of failed matching successes is less than the second preset value, the first lane line may be continuously stored for the next matching.

In addition, after each first lane line is matched by the second lane line, the number of unmatched successes corresponding to the first lane line may be set as an initial value, where the initial value may be 0.

Fig. 5 is a schematic diagram of the implementation flow of the tracking module in the target neural network to determine the tracking result corresponding to the second lane line provided by the embodiment of the present disclosure, wherein the tracking module 51 acquires the lane line and the lane line input by the reasoning module After the offset information, it can first determine whether the lane line is the lane line corresponding to the first frame image (S501), that is, judge whether the image to be recognized corresponding to the lane line is the first frame image; For other lane lines matched by the lane line, the lane line identification information corresponding to each lane line can be directly generated (S502), and, when the image to be recognized is the first frame image, the reasoning module can only input the first frame image For the corresponding lane line, the offset information of the lane line is not input. The local database 52 is used to store the information of each lane line determined after the image recognition of the image to be recognized, which can be the lane point information on the lane line, the lane line number information, etc.; The lane line tracking method introduced in , determines the obtained tracking result of the lane line input by the reasoning module, as shown in Figure 5, can follow the steps S521 to S524 as follows to determine the obtained track result of the lane line input by the reasoning module : Step S521, based on the thermal map feature information corresponding to the image to be recognized in the Nth frame, determine the pixel points to be matched corresponding to each lane line; Step S522, determine each lane line corresponding to the image to be recognized in the Nth frame and the previous frame image The second distance between the corresponding lane lines; Step S523, determine the tracking result corresponding to each lane line corresponding to the N frame image to be recognized; Step S524, replace the previous frame with the lane line corresponding to the N frame image to be recognized The image corresponds to the lane line on the match.

In some embodiments, for S102, after the image to be recognized and the previous frame of image are acquired, the image to be recognized and the previous frame of image can also be down-sampled respectively by using preset sampling multiples to obtain the image corresponding to the image to be recognized, The sampled new image to be recognized, and the sampled new previous frame image corresponding to the previous frame image. For example, the image to be recognized and the previous frame image may be down-sampled by using a 4-fold down-sampling manner or an 8-fold down-sampling manner.

Then, the target neural network can be used to convolve the new image to be recognized, the new previous frame image, and the first heat map corresponding to the previous frame image to obtain the first image features corresponding to the new image to be recognized. Then, the second lane line in the image to be recognized and the offset information corresponding to the second lane line are determined by using the first image feature.

Here, since the first heat map is obtained by image recognition of the previous frame image, in the process of image recognition of the previous frame image, the downsampling of the previous frame image has been completed, and further, the obtained first A heat map is the first heat map corresponding to the sampled previous frame image, and the first heat map can be directly convolved without downsampling the first heat map.

In this way, through down-sampling, each pixel in the obtained new image (and the new previous frame image and the new image to be recognized) can represent the number of pixels corresponding to the sampling multiple information, and further, using the information of each pixel in the new image is beneficial to determine a more accurate second lane line and its corresponding offset information.

In some embodiments, since the lane line tracking method provided by the embodiments of the present disclosure is executed by the target neural network, the target neural network can be trained to improve prediction accuracy. Therefore, the implementation of the present disclosure also provides a method for identifying Steps to train the target neural network on images:

Step T1: Input the sample identification image, the first predicted heat map feature information corresponding to the sample image of the previous frame of the sample identification image, and the sample image of the previous frame into the target neural network, and process the input information through the target neural network to obtain the sample The second predicted heat map feature information corresponding to the recognition image, the predicted lane line corresponding to the sample recognition image, the predicted offset information corresponding to the predicted lane line, and the predicted tracking result corresponding to each predicted lane line.

Wherein, the multiple sample identification images may be images in the same video clip, and each sample image includes lane lines, or the multiple sample identification images may also be images including lane lines captured separately.

Exemplarily, the plurality of sample identification images may be images in a video clip containing visible lane lines captured by the autonomous vehicle.

The feature information of the first predicted heat map corresponding to the sample image of the previous frame may be obtained by image recognition of the sample image of the previous frame by the target neural network, and the predicted lane line may be the predicted lane line in the sample recognition image output by the target neural network.

During implementation, when performing image recognition on the sample recognition image, the sample recognition image, the first predicted heat map feature information corresponding to the previous frame sample image of the sample recognition image, and the previous frame sample image can be input into the target neural network, using The reasoning module in the target neural network processes the above-mentioned input information, determines the sample prediction image features corresponding to the sample recognition image, and then determines the second prediction heat map feature corresponding to the sample recognition image based on the sample prediction image features. Furthermore, the predicted lane line corresponding to the sample recognition image and the predicted offset information corresponding to the predicted lane line can be determined based on the second predicted heat map feature and the sample predicted image feature, and then the tracking module is used to determine the corresponding predictive tracking results.

Step T2: Based on the second predicted heat map feature information, the marked heat map feature information corresponding to the sample recognition image, the predicted lane line, the marked lane line corresponding to the sample recognition image, the predicted offset information, and the mark offset corresponding to the sample recognition image The amount information, the predicted tracking result corresponding to each predicted lane line and the labeled tracking result corresponding to each marked lane line are used to determine the loss value.

During implementation, the first predicted loss value corresponding to the heat map branch in the target neural network can be determined based on the second predicted heat map feature and the marked heat map feature information (that is, the true value) corresponding to the sample recognition image, based on the predicted lane line The labeled lane line corresponding to the sample recognition image determines the second prediction loss value of the feature embedding branch in the target neural network, and determines the target neural network based on the predicted offset information and the labeled offset information corresponding to the sample recognition image. The third prediction loss value corresponding to the offset prediction branch, and based on the prediction tracking result corresponding to each predicted lane line and the label tracking result corresponding to each standard lane line, determine the fourth prediction loss value corresponding to the target neural network. Afterwards, the first predicted loss value, the second predicted loss value, the third predicted loss value, and the fourth predicted loss value may be used as the corresponding loss value of the target neural network.

Step T3: Adjust the network parameter value of the target neural network according to the loss value until the preset training cut-off condition is met, and a trained target neural network is obtained.

Here, the preset training cut-off condition may be that the number of rounds of iterative training is greater than the preset number of rounds and/or the prediction accuracy of the trained target neural network reaches the preset accuracy.

During implementation, the target neural network can be iteratively trained according to the loss value, so as to realize the adjustment of the network parameter value of the target neural network; when it is determined that the preset training cut-off condition is satisfied, the training of the target neural network is completed, and the The network parameter value obtained at this time is used as the target network parameter value corresponding to the target neural network, and the target neural network that has been trained at this time is used as the trained target neural network.

In this way, using the second predicted heat map features, predicted lane lines, predicted offset information, and the predicted tracking results corresponding to each predicted lane line, the target loss values corresponding to all the above information can be determined respectively, and then the target loss value can be adjusted using the loss value The network parameter value of the neural network can improve the prediction accuracy of the target neural network, thereby improving the accurate training of each of the above information output by the target neural network.

In some embodiments, the predicted offset information includes the predicted offset of the lane line in at least one image direction.

Here, the predicted offset information may include an offset in the image width direction and/or an offset in the image height direction. In this way, by outputting the offset of the lane line in one direction, it is possible to effectively reduce the amount of calculation that the target neural network needs to process when outputting the offset information, which is conducive to improving the speed and efficiency of information processing.

During implementation, for each predicted lane line, the offset branch in the target neural network can fix a direction when determining the predicted offset corresponding to the predicted lane line, and output the predicted lane line compared with the previous frame image The predicted offset of the lane line in the other direction. For example, the image height direction is fixed, and for each predicted lane line, at each image height, its offset relative to the lane line in the previous frame image in the image width direction can be output.

FIG. 6 is a schematic diagram of an output prediction offset information provided by an embodiment of the present disclosure. Among them, X represents the offset in the image width direction, each image height in the Y image height direction, and the offset of each predicted lane line in the X direction can be gradually changed with the increase or decrease of the image height Form changes. In FIG. 6 , different offsets can be represented by different colors, and the offsets represented by different colors in the X direction can be shown in the color indicator bar 60. FIG. 6 shows the predicted lane line 1 , predicted offsets 611 , 612 , 613 , and 614 corresponding to predicted lane line 2 , predicted lane line 3 , and predicted lane line 4 , respectively.

In addition, the unit of the offset in the X direction can be related to the downsampling multiple, for example, when the preset sampling multiple is 8 times, the unit can be 8×offset; when the preset sampling multiple is 4 times, The unit can be 4×offset.

In addition, in the process of using the predicted offset information of each predicted lane line and the standard offset information corresponding to each predicted lane line to construct the target neural network for the predicted offset information, the labeled offset information and predicted The offset information constructs the smoothing loss for predicting the smoothing offset, and uses the smoothing loss to train the target neural network to reduce the process of outputting the predicted offset information of the lane line in the image to be recognized by the trained target neural network In , there is a problem that the offset corresponding to several lane points adjacent to the same lane line changes suddenly.

For example, for the predicted offset information corresponding to lane line 5, in the offset information corresponding to image height 10 to 15, the offset of the lane point corresponding to image height 10 is -2.5, and the offset of image height 12 corresponds to The offset of the lane point is -1.5, the offset of the lane point corresponding to the image height 13 is -1, the offset of the lane point corresponding to the image height 14 is -6, and the offset of the lane point corresponding to the image height 15 is The case where the offset is 0, here, since the offset of the lane point corresponding to the image height 14 is -6, the offset 0 corresponding to the image height 15, and the offset between -1.5 corresponding to the image height 12 The offset difference is much larger than the offset difference between the offset -2.5 corresponding to the image height 10 and the offset -1.5 corresponding to the image height 12, and the offset -1.5 corresponding to the image height 12 and The offset difference between the offset -1 corresponding to the image height 13, therefore, there will be a jump change in the predicted offset information corresponding to the lane line 5 in the offset of the lane point corresponding to the image height 14 , and this situation is unreasonable in the offset information corresponding to the real upper and lower frame lane lines, and the offset information corresponding to the real upper and lower frame lane lines should be smooth and gradual.

Therefore, using the smoothing loss to train the target neural network can improve the ability of the target neural network to predict smoothness, that is, the trained target neural network can predict the offset -6 at the lane point corresponding to the image height 14 After adjustment, for example, the adjusted output offset may be -0.5, so that the smooth prediction offset information corresponding to each lane line as shown in FIG. 6 is obtained.

In some embodiments, the sample image of the previous frame of the sample identification image may also be acquired by the following steps: performing random translation and/or rotation operations on the sample identification image to obtain the sample image of the previous frame of the sample identification image.

Here, the random translation operation can realize the translation of the positions of the pixels in the sample image, that is, complete the translation of the positions of the pixels corresponding to the lane lines in the sample recognition image, and obtain an image different from the sample recognition image. The random rotation operation can realize the rotation of the positions of the pixels in the sample image, and can also obtain an image different from the sample recognition image. In this way, through random translation and/or rotation operations, the position of the pixels in the sample recognition image can be changed, and the simulated previous frame image can be obtained. Therefore, only one frame of sample recognition image can be obtained to complete the target recognition. The training of neural network improves the flexibility of neural network training.

FIG. 7 is a schematic diagram of an implementation process for training a target neural network provided by an embodiment of the present disclosure. 7 shows the process of using multiple sample recognition images to train the target neural network, wherein, after the target neural network 70 performs image recognition on the Nth frame of sample recognition images in the multiple sample recognition images, it can determine the first The predicted heat map feature information 701 corresponding to N frames of sample recognition images, each predicted lane line 702 corresponding to the sample recognition image, the predicted offset information 703 of each predicted lane line, and the predicted tracking result 704 of each predicted lane line, and then, it can be Based on the predicted heat map feature information 701 corresponding to the sample recognition image of the Nth frame and the marked heat map feature information corresponding to the sample recognition image of the Nth frame, determine the heat map prediction loss value 711 corresponding to the sample recognition image of the Nth frame of the target neural network, Based on the predicted lane line 702 corresponding to the sample recognition image of the Nth frame and the marked lane line corresponding to the sample recognition image of the Nth frame, the lane line prediction loss value 712 corresponding to the sample recognition image of the Nth frame of the target neural network can be determined, which can be based on the The predicted offset information 703 of each predicted lane line corresponding to the N-frame sample recognition image and the label offset information corresponding to each predicted lane line, and determine the offset information prediction loss corresponding to the N-frame sample recognition image of the target neural network value 713, and based on the predicted tracking results 704 of each predicted lane line corresponding to the determined sample recognition image of the Nth frame and the label tracking results corresponding to each predicted lane line, determine the tracking corresponding to the sample recognition image of the Nth frame of the target neural network Loss value 714, similarly, according to the above steps, the predicted heat map feature information 721 corresponding to the N+1th frame sample recognition image, each predicted lane line 722 corresponding to the sample recognition image, and the predicted offset of each predicted lane line can be determined Information 723 and predicted tracking results 724 of each predicted lane line, as well as heat map predicted loss value 731 , lane line predicted loss value 732 , offset information predicted loss value 733 , and tracking loss value 734 . Afterwards, the target neural network 70 can be iteratively trained using the respective loss values corresponding to the Nth frame sample recognition image and the respective loss values corresponding to the N+1th frame sample recognition image, so as to realize the network parameter value of the target neural network. Adjustment; when the training cut-off condition is met, the trained target neural network is obtained.

In addition, in Figure 7, if the previous frame image is the first frame image, you can not use the offset corresponding to the frame image to construct the loss, but only construct the loss of information other than the offset to train the target neural network. For For images other than the first frame, it is necessary to construct the above-mentioned losses for training the target neural network.

Those skilled in the art can understand that in the above method, the writing order of each step does not imply a strict execution order and constitutes any limitation on the implementation process, and the specific execution order of each step should be determined by its function and possible internal logic.

Based on the same inventive concept, the embodiment of the present disclosure also provides a lane line tracking device corresponding to the lane line tracking method. Since the problem-solving principle of the device in the embodiment of the present disclosure is similar to that of the above-mentioned lane line tracking method in the embodiment of the present disclosure, therefore The implementation of the device can refer to the implementation of the method.

Fig. 8 is a schematic diagram of the composition and structure of a lane line tracking device provided by an embodiment of the present disclosure. As shown in Fig. 8, the device includes:

The acquisition part 801 is configured to acquire a previous frame image of the image to be recognized, and a first lane line obtained by performing image recognition on the previous frame image;

The first determining part 802 is configured to determine the second lane line in the image to be recognized based on the previous frame image and the image to be recognized, and the relative distance between the second lane line and the first Lane line offset information;

The second determining part 803 is configured to determine the first lane line matching at least one of the second lane lines based on the offset information corresponding to each of the second lane lines and each of the first lane lines , obtaining at least one tracking result of the second lane line.

In some implementations, the first determining part 802 is further configured to: determine the identified The first image feature corresponding to the image to be identified; based on the first image feature, determine the second thermal map feature information corresponding to the image to be identified; based on the second thermal map feature information and the first image feature , determining a second lane line in the image to be recognized, and offset information corresponding to the second lane line.

In some implementations, the second determining part 803 is further configured to: determine the initial pixel points corresponding to each of the second lane lines based on the second heat map feature information; The offset information corresponding to the two lane lines and the initial pixel points corresponding to each of the second lane lines respectively determine the target pixel points corresponding to each of the second lane lines; based on each of the second lane lines corresponding to The target pixel point and the pixel point corresponding to each of the first lane lines, determine the first lane line matching at least one of the second lane lines, and obtain a tracking result of at least one of the second lane lines.

In some implementations, the second determination part 803 is further configured to: for each second lane line, based on the offset information corresponding to the second lane line, determine the An offset value corresponding to each initial pixel point; determining a target pixel point corresponding to the second lane line based on the offset value corresponding to each initial pixel point.

In some implementations, the second determining part 803 is further configured to: for each second lane line, based on the second heat map feature information, filter the target pixel points of the second lane line Obtain the pixel points to be matched; based on the pixel points corresponding to each first lane line, respectively determine the first distance between each pixel point to be matched and each of the first lane lines; Determine the second distance between the second lane line and each of the first lane lines; based on the determined second distance, determine the first lane line matching the second lane line .

In some implementations, the second determining part 803 is further configured to: after determining the first lane line that matches the second lane line, when multiple second lane lines match the same first lane line, In the case of a lane line, it is determined that the second lane line with the shortest distance from the first lane line matches the first lane line.

In some implementations, the second determining part 803 is further configured to use the first lane line corresponding to the shortest second distance as the first lane line corresponding to the shortest second distance when the shortest second distance is less than the first preset value. The second lane line matches the lane line.

In some implementations, the second determining part 803 is further configured to use the second lane line as a new lane line when the shortest second distance is greater than a first preset value.

In some implementations, the device further includes: a deleting part 804 configured to: after obtaining the tracking result, for each of the first lane lines that are not successfully matched, determine the The number of unsuccessful matches; the number of unsuccessful matches is used to characterize the number of times that the corresponding first lane line does not match the second lane line in consecutive multiple frames of images to be recognized; when the number of successful mismatches is greater than the second preset value case, delete the first lane line.

In some implementations, the lane line tracking method is implemented by a target neural network, and the target neural network is trained by using multiple sample recognition images.

In some implementations, the device further includes: a training part 805 configured to: train the target neural network by adopting the following steps: The first predicted heat map feature information and the previous frame sample image are input into the target neural network, and the target neural network outputs the second predicted heat map feature information corresponding to the sample recognition image, and the sample recognition image corresponding to Predicted lane lines, predicted offset information corresponding to the predicted lane lines, and predicted tracking results corresponding to each of the predicted lane lines; Map feature information, the predicted lane line, the labeled lane line corresponding to the sample recognition image, the predicted offset information, the labeled offset information corresponding to the sample recognized image, and each predicted lane line Determine the loss value for the corresponding prediction tracking result and the label tracking result corresponding to each labeled lane line; adjust the network parameter value of the target neural network according to the loss value until the preset training cut-off condition is met, and the trained target is obtained Neural Networks.

In some implementations, the predicted offset information includes an offset of the predicted lane line in at least one image direction.

In some implementations, the training part 805 is further configured to acquire the sample image of the previous frame of the sample recognition image through the following steps: performing random translation and/or rotation operations on the sample recognition image to obtain the The image in the previous frame of the sample recognition image.

In some implementations, the acquisition part 801 is further configured to: if the previous frame image is the first frame image, perform image recognition on the previous frame image, and determine that the previous frame image corresponds to The second image feature; based on the second image feature, determine the first thermal map feature information corresponding to the previous frame image; based on the first thermal map feature information and the second image feature, determine the The first lane line in the previous image frame.

In some implementations, the first determining part 802 is further configured to: respectively perform downsampling processing on the image to be recognized and the image of the previous frame according to a preset sampling multiple to obtain a new image to be recognized and a new previous frame image; based on the new previous frame image and the new image to be recognized, determine the second lane line in the image to be recognized, and the second lane line relative to the nearest first Offset information of a lane line.

For the description of the processing flow of each part in the device and the interaction flow between each part, reference may be made to the relevant descriptions in the foregoing method embodiments.

It should be noted that, in the embodiments of the present disclosure and other embodiments, a "part" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course, it may also be a unit, and it may also be a module or a non-module of.

An embodiment of the present disclosure also provides a computer device, as shown in FIG. 9 , which is a schematic structural diagram of a computer device provided by an embodiment of the present disclosure, including:

Processor 91 and memory 92; the memory 92 stores machine-readable instructions executable by the processor 91, the processor 91 is used to execute the machine-readable instructions stored in the memory 92, and the machine-readable instructions are executed by the processor 91 During execution, the processor 91 performs the following steps: S101: Acquire the previous frame image of the image to be recognized, and the first lane line obtained by performing image recognition on the previous frame image; S102: Based on the previous frame image and the image to be recognized Recognize the image, determine the second lane line in the image to be recognized, and the offset information of the second lane line relative to the nearest first lane line and S103: Based on the offset information corresponding to each second lane line and For each first lane line, determine the first lane line matching the at least one second lane line, and obtain a tracking result of the at least one second lane line. Above-mentioned storer 92 comprises memory 921 and external memory 922; Here memory 921 is also called internal memory, is used for temporarily storing computing data in the processor 91, and the data exchanged with external memory 922 such as hard disk, processor 91 communicates with memory 921 through memory 921. The external memory 922 performs data exchange.

For the execution process of the above instructions, reference may be made to the steps of the lane line tracking method described in the embodiments of the present disclosure.

An embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the lane line tracking method described in the foregoing method embodiments are executed. Wherein, the storage medium may be a volatile or non-volatile computer-readable storage medium.

The computer program product of the lane line tracking method provided by the embodiments of the present disclosure includes a computer-readable storage medium storing program codes, and the instructions included in the program code can be used to execute the lane line tracking method described in the above method embodiments For details, please refer to the above method embodiment. The computer program product can be specifically realized by means of hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) etc. wait.

Those skilled in the art can clearly understand that for the convenience and brevity of description, for the specific working process of the device described above, reference can be made to the corresponding process in the foregoing method embodiments. In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods may be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined. Or some features can be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

Finally, it should be noted that: the above-described embodiments are only exemplary implementations of the present disclosure, and are used to illustrate the technical solutions of the present disclosure, rather than to limit them, and the scope of protection of the present disclosure is not limited thereto. Although referring to The foregoing embodiments have described the present disclosure in detail, and those of ordinary skill in the art should understand that any person familiar with the art within the technical scope of the present disclosure can still carry out the technical solutions described in the foregoing embodiments Modifications or changes can be easily thought of, or equivalent replacements for some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in this disclosure. within the scope of public protection.

Industrial Applicability

Embodiments of the present disclosure provide a lane line tracking method, device, computer equipment, storage medium, and computer program product, wherein the method includes: acquiring a previous frame image of the image to be recognized, and performing image recognition on the previous frame image The obtained first lane line; based on the previous frame image and the image to be recognized, determine the second lane line in the image to be recognized, and the offset information of the second lane line relative to the nearest first lane line; Based on the offset information corresponding to each second lane line and each first lane line, determine the first lane line matching the at least one second lane line, and obtain a tracking result of the at least one second lane line. Through the embodiments of the present disclosure, in the process of detecting and tracking the second lane line in the image to be recognized, the accuracy of the determined second lane line and the offset information corresponding to the second lane line can be improved, so that Improve the accuracy of determined trace results.

Claims (19)

1. A lane line tracking method, comprising:

   Acquiring the previous frame image of the image to be recognized, and the first lane line obtained by performing image recognition on the previous frame image;

   Based on the previous frame image and the image to be recognized, determine a second lane line in the image to be recognized, and offset information of the second lane line relative to the nearest first lane line;

   Based on the offset information corresponding to each of the second lane lines and each of the first lane lines, determine a first lane line that matches at least one of the second lane lines, and obtain at least one of the second lanes Line tracking results.
2. The method according to claim 1, wherein, based on the previous frame image and the image to be recognized, determining the second lane line in the image to be recognized, and the relative distance between the second lane line The offset information of the nearest first lane line, including:

   Based on the previous frame image, the first thermal map feature information corresponding to the previous frame image, and the image to be recognized, determine the first image feature corresponding to the image to be recognized;

   Based on the first image feature, determine the second thermal map feature information corresponding to the image to be recognized;

   Based on the second heat map feature information and the first image feature, determine a second lane line in the image to be recognized and offset information corresponding to the second lane line.
3. The method according to claim 2, wherein, based on the offset information corresponding to each of the second lane lines and each of the first lane lines, it is determined to match at least one of the second lane lines The first lane line, obtaining at least one tracking result of the second lane line, includes:

   Based on the feature information of the second heat map, determine the initial pixel points corresponding to each of the second lane lines;

   Based on the offset information corresponding to each of the second lane lines and the initial pixel points corresponding to each of the second lane lines, respectively determine the target pixel points corresponding to each of the second lane lines;

   Based on the target pixel points corresponding to each of the second lane lines and the pixel points corresponding to each of the first lane lines, determine the first lane line that matches at least one of the second lane lines, and obtain at least one of the Tracking results of the second lane line.
4. The method according to claim 3, wherein, based on the offset information corresponding to each of the second lane lines and the initial pixel points corresponding to each of the second lane lines, each of the second lane lines is determined respectively The target pixel corresponding to the two-lane line, including:

   For each second lane line, based on the offset information corresponding to the second lane line, determine an offset value corresponding to each initial pixel point in the second lane line;

   Based on the offset value corresponding to each of the initial pixel points, the target pixel point corresponding to the second lane line is determined.
5. The method according to claim 3 or 4, wherein, based on the target pixel points corresponding to each of the second lane lines and the pixel points corresponding to each of the first lane lines, it is determined that at least one of the second lane lines corresponds to The first lane line matched by the second lane line, including:

   For each second lane line, based on the second heat map feature information, select the pixel points to be matched from the target pixel points of the second lane line;

   Based on the pixel points corresponding to each first lane line, respectively determine a first distance between each of the pixel points to be matched and each of the first lane lines;

   determining a second distance between the second lane line and each of the first lane lines based on the first distance corresponding to each pixel point to be matched;

   Based on the determined second distance, a first lane line matching the second lane line is determined.
6. The method according to claim 5, wherein, after determining the first lane line matching the second lane line, further comprising:

   In the case that multiple second lane lines match the same first lane line, it is determined that the second lane line with the shortest distance to the first lane line matches the first lane line.
7. The method according to claim 5 or 6, wherein said determining the first lane line matching the second lane line based on the determined second distance comprises:

   If the shortest second distance is less than the first preset value, the first lane line corresponding to the shortest second distance is used as the lane line matching the second lane line.
8. The method according to claim 7, wherein said determining a first lane line matching said second lane line based on said determined second distance further comprises:

   If the shortest second distance is greater than the first preset value, the second lane line is used as a new lane line.
9. The method according to any one of claims 1 to 8, wherein, after obtaining the tracking result, further comprising:

   For each of the first lane lines that are not successfully matched, determine the number of times that the first lane lines are not matched successfully; The number of matching times on the second lane line;

   If the number of failed matching successes is greater than a second preset value, the first lane line is deleted.
10. The method according to any one of claims 1 to 9, wherein the lane line tracking method is executed by a target neural network, and the target neural network is trained by using a plurality of sample recognition images.
11. The method of claim 10, wherein the target neural network is trained using the steps of:

    Input the sample identification image, the first predicted heat map feature information corresponding to the sample image of the previous frame of the sample identification image, and the sample image of the previous frame into the target neural network, and the target neural network outputs the The second predicted heat map feature information corresponding to the sample identification image, the predicted lane line corresponding to the sample identification image, the predicted offset information corresponding to the predicted lane line, and the predicted tracking result corresponding to each predicted lane line;

    Based on the feature information of the second predicted heat map, the feature information of the marked heat map corresponding to the sample recognition image, the predicted lane line, the marked lane line corresponding to the sample recognition image, the predicted offset information, the The label offset information corresponding to the sample recognition image, and the predicted tracking results corresponding to each of the predicted lane lines and the labeled tracking results corresponding to each labeled lane line, to determine the loss value;

    Adjusting the network parameter value of the target neural network according to the loss value until the preset training cut-off condition is met, and a trained target neural network is obtained.
12. The method according to claim 11, wherein the predicted offset information comprises an offset of the predicted lane line in at least one image direction.
13. The method according to claim 11, wherein the sample image of the previous frame of the sample identification image is obtained by the following steps:

    A random translation and/or rotation operation is performed on the sample identification image to obtain a sample image of a previous frame of the sample identification image.
14. The method according to any one of claims 1 to 13, wherein obtaining the first lane line obtained by performing image recognition on the previous frame image comprises:

    When the previous frame image is the first frame image, perform image recognition on the previous frame image, and determine the second image feature corresponding to the previous frame image;

    Based on the second image feature, determine the first thermal map feature information corresponding to the previous frame image;

    Based on the first heat map feature information and the second image feature, determine the first lane line in the previous frame image.
15. The method according to any one of claims 1 to 14, wherein, based on the previous frame image and the image to be recognized, determining the second lane line in the image to be recognized, and the second lane line in the image to be recognized The offset information of the second lane line relative to the nearest first lane line, including:

    respectively performing downsampling processing on the image to be recognized and the image of the previous frame according to a preset sampling multiple to obtain a new image to be recognized and a new image of the previous frame;

    Based on the new previous frame image and the new image to be recognized, determine a second lane line in the image to be recognized, and the offset information of the second lane line corresponding to the closest first lane line .
16. A lane line tracking device, comprising:

    The acquiring part is configured to acquire a previous frame image of the image to be recognized, and a first lane line obtained by performing image recognition on the previous frame image;

    The first determination part is configured to determine, based on the previous frame image and the image to be recognized, a second lane line in the image to be recognized, and a relative distance between the second lane line and the closest first lane Line offset information;

    The second determining part is configured to determine a first lane line matching at least one of the second lane lines based on the offset information corresponding to each of the second lane lines and each of the first lane lines, A tracking result of at least one second lane line is obtained.
17. A computer device, comprising: a processor and a memory, the memory stores machine-readable instructions executable by the processor, the processor is configured to execute the machine-readable instructions stored in the memory, and the machine can When the read instruction is executed by the processor, the processor executes the steps of the lane line tracking method according to any one of claims 1 to 15.
18. A computer-readable storage medium, a computer program is stored on the computer-readable storage medium, and when the computer program is run by a computer device, the computer device executes the lane marking according to any one of claims 1 to 15 The steps of the trace method.
19. A computer program product, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, when the computer program is read and executed by a computer, the method described in any one of claims 1 to 15 is realized in the steps.

Images