CN116168173A - Lane line map generation method, device, electronic device and storage medium - Google Patents

Abstract

The present application relates to a lane line map generation method, device, electronic device, and storage medium, wherein the lane line map generation method includes: acquiring an original image collected by a vehicle-mounted camera; determining a corresponding mask image based on the original image, and masking The film image includes lane line attribute information; based on the mask image, the corresponding grid map is constructed; based on the mask image and the internal and external parameters of the vehicle camera, the lane line attribute information is filled into the grid map to obtain the filled grid Map; generate a local lane line map based on the filled grid map. Through this application, the problem in the prior art that the lane line map cannot be generated from the image information of the visual space is solved, and the lane line map is generated according to the image information of the visual space.

Description

Lane line map generation method, device, electronic device and storage medium

technical field

The present application relates to the technical field of driving assistance, in particular to a lane line map generation method, device, electronic device and storage medium.

Background technique

Lane map can provide important prior information for the field of intelligent driving assistance, for example, it can provide the basis for real-time positioning and motion planning of intelligent driving vehicles. Therefore, the generation of lane map has attracted a lot of attention from scholars in related fields.

Most of the current lane line maps are generated based on the point cloud data generated by lidar and combined with manual labeling. And the degree of wear and tear, resulting in low integrity of the map. Moreover, no effective solution has been proposed so far for generating lane line maps based on visual-spatial image information.

How to use the image information of visual space to generate lane line map is a problem that needs to be solved.

Contents of the invention

In this embodiment, a lane line map generation method, device, electronic device and storage medium are provided to solve the problem of how to use visual space image information to generate a lane line map in the related art.

In the first aspect, a method for generating a lane line map is provided in this embodiment, including:

Obtain the original image collected by the on-board camera;

Determine a corresponding mask image based on the original image, where the mask image includes lane line attribute information;

Constructing a corresponding grid map based on the mask image;

Filling the lane line attribute information into the grid map based on the mask image and the internal and external parameters of the vehicle camera to obtain a filled grid map;

Based on the filled grid map, a local lane line map is generated.

In some of these embodiments, the constructing a corresponding grid map based on the mask image includes:

Obtain the internal and external parameters of the vehicle camera;

Converting the mask image into the grid map based on extrinsic and extrinsic parameters of the onboard camera.

In some of these embodiments, based on the mask image and the internal and external parameters of the vehicle camera, the lane line attribute information is filled into the grid map, and the filled grid map is obtained, including :

Based on the internal and external parameters of the vehicle camera, project the grid map to the imaging plane of the vehicle camera to obtain a distortion-free map;

Obtain the mapping relationship between the undistorted image and the distorted image in the vehicle-mounted camera;

Obtaining a distortion map corresponding to the grid map based on the distortion-free map and the mapping relationship;

Based on the mask image and the distortion map, the lane line attribute information is filled into the grid map to obtain a filled grid map.

In some of these embodiments, the lane line attribute information includes a plurality of attribute categories, and after the local lane line map is generated based on the filled grid map, the method further includes:

Obtaining a plurality of local lane line maps corresponding to the vehicle-mounted camera in different poses;

determining an initial global lane line map based on a plurality of said local lane line maps;

Determine the total number of observations of the target grid, and the attribute category corresponding to each observation of the target grid, where the target grid is any grid in the initial global lane line map;

determining the target attribute category of the target grid based on the number of attribute observations of the target grid in each attribute category, the total number of observations, and a preset threshold;

Based on the target attribute categories of all grids in the initial global lane line map, a target global lane line map is generated.

In some of these embodiments, the determining the target attribute category of the target grid based on the number of attribute observations of the target grid in each attribute category, the total number of observations, and a preset threshold includes:

Based on the number of attribute observations of the target grid in each attribute category and the total number of observations, determine the category probability corresponding to the target grid in each attribute category;

Determine the target attribute category of the target grid based on the probability of each category and the preset threshold.

In some of these embodiments, after generating the target global lane line map based on the target attribute categories of all grids in the initial global lane line map, the method further includes:

Obtain the distribution parameters of the target global lane line map;

Based on the distribution parameters, the current number of attribute observations of lane lines in each attribute category in the target global lane line map, and the current total number of observations of lane lines in the target global lane line map, the target The global lane line map is updated.

In some of these embodiments, the current attribute observation times of the lane lines in each attribute category based on the distribution parameters, the lane lines in the target global lane line map, and the lane lines in the target global lane line map The current total number of observations of the target global lane line map is updated, including:

Based on the distribution parameters, the current number of attribute observations of lane lines in each attribute category in the target global lane line map, and the current total number of observation times of lane lines in the target global lane line map, the determined Describe the current probability distribution function of the target global lane line map;

Based on the current probability distribution function, the target global lane map is updated.

In the second aspect, in this embodiment, a lane line map generation device is provided, including:

The obtaining module is used to obtain the original image collected by the vehicle-mounted camera;

A determining module, configured to determine a corresponding mask image based on the original image, where the mask image includes lane line attribute information;

A building module for building a corresponding grid map based on the mask image;

A filling module, configured to fill the lane line attribute information into the grid map based on the mask image and internal and external parameters of the vehicle camera, to obtain a filled grid map;

A generating module, configured to generate a local lane line map based on the filled grid map.

In the third aspect, this embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and operable on the processor, and the processor executes the computer program The program implements the first aspect above and the lane line map generation method described in any embodiment of the first aspect.

In the fourth aspect, this embodiment provides a storage medium on which a computer program is stored, and when the program is executed by a processor, it realizes the first aspect and the lane marking described in any embodiment of the first aspect Map generation method.

Compared with related technologies, the lane line map generation method provided in this embodiment determines the corresponding mask image through the original image collected by the vehicle-mounted camera, wherein the mask image includes the attribute information of the lane line, and further, according to the mask The mask image is used to construct the corresponding grid map, and according to the mask image and the internal and external parameters of the vehicle camera, the lane line attribute information in the mask image is filled into the corresponding grid map to obtain the filled grid map, so that The filled grid map includes the attribute information of the lane line. Further, a local lane line map is generated according to the filled grid map, so that the lane line map is generated from the image information of the visual space collected by the vehicle camera, which avoids the The influence of weather, lane line material and other factors on the lidar leads to the problem of low integrity of the generated lane line map. The lane line map generated according to the visual space image information provided by this application has simple and clear algorithm logic and can Provide accurate prior information for real-time positioning and motion planning of intelligent driving vehicles.

The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below, so as to make other features, objects, and advantages of the application more comprehensible.

Description of drawings

The drawings described here are used to provide a further understanding of the application and constitute a part of the application. The schematic embodiments and descriptions of the application are used to explain the application and do not constitute an improper limitation to the application. In the attached picture:

FIG. 1 is a schematic diagram of an application scenario of a lane line map generation method provided by an embodiment of the present application;

Fig. 2 is a flow chart of a method for generating a lane line map provided in an embodiment of the present application;

Fig. 3 is a schematic diagram of an imaging plane and a perspective plane of a bird's-eye view provided by an embodiment of the present application;

Fig. 4 is the flow chart of an embodiment of a method for generating a lane line map according to an embodiment of the present application;

FIG. 5 is a schematic diagram of a lane line map in a park environment provided by an embodiment of the present application;

Fig. 6 is a structural block diagram of a lane line map generation device provided by an embodiment of the present application;

Fig. 7 is a schematic diagram of an internal structure of a computer device provided by an embodiment of the present application.

Detailed ways

In order to understand the purpose, technical solution and advantages of the present application more clearly, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

Unless otherwise defined, the technical terms or scientific terms involved in the application shall have the general meanings understood by those skilled in the technical field to which the application belongs. In this application, words like "a", "an", "an", "the", "these" and the like do not denote quantitative limitations, and they may be singular or plural. The terms "comprising", "comprising", "having" and any variants thereof referred to in this application are intended to cover non-exclusive inclusion; for example, processes, methods and The system, product, or apparatus is not limited to the steps or modules (units) listed, but may include steps or modules (units) that are not listed, or may include other steps or modules that are inherent to the process, method, product, or apparatus (unit). The terms "connected", "connected", "coupled" and the like referred to in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Plurality" referred to in this application means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships. For example, "A and/or B" may indicate: A exists alone, A and B exist simultaneously, and B exists independently. Usually, the character "/" indicates that the objects associated before and after are in an "or" relationship. The terms "first", "second", "third" and the like involved in this application are only for distinguishing similar objects, and do not represent a specific ordering of objects.

The lane line map generation method provided by the embodiment of the present application can be applied to the application scenario shown in FIG. 1 , and FIG. 1 is a schematic diagram of an application scenario of a lane line map generation method provided by the embodiment of the present application. Wherein, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that needs to be processed by the server 104 . The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. In the embodiment of this application, the terminal 102 can be a smart vehicle-mounted device, and the terminal 102 can also be various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers , smart TV, smart air conditioner, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

Lane map can provide important prior information for the field of intelligent driving assistance, for example, it can provide the basis for real-time positioning and motion planning of intelligent driving vehicles. Therefore, the generation of lane map has attracted a lot of attention from scholars in related fields.

Most of the current lane line maps are generated based on the point cloud data generated by lidar and combined with manual labeling. And the degree of wear and tear, resulting in low integrity of the map. Moreover, no effective solution has been proposed so far for generating lane line maps based on visual-spatial image information.

How to use the image information of visual space to generate lane line map is a problem that needs to be solved.

In this embodiment, a method for generating a lane line map is provided. FIG. 2 is a flowchart of a method for generating a lane line map provided in an embodiment of the present application. The execution subject of the method may be an electronic device, and optionally, an electronic The device may be a server or a terminal device, but the present application is not limited thereto. Specifically, as shown in Figure 2, the process includes the following steps:

Step S201, acquiring the original image collected by the vehicle-mounted camera.

Step S202, based on the original image, determine the corresponding mask image.

Wherein, the mask image includes lane attribute information.

Specifically, the lane attribute information may include solid lines, dashed lines, stop lines, and other four categories. More specifically, the lane attribute information may also include solid lines, dashed lines, stop lines, and other corresponding labels. For example, the label of a solid line is 1, the label of the dashed line is 2, the label of the stop line is 3, and the label of the others is 0.

Exemplarily, the original image of the map area to be generated is acquired by a vehicle-mounted camera, and the map area to be generated includes lane lines.

Specifically, the vehicle-mounted camera may be a vehicle-mounted monocular camera, or may be a vehicle-mounted binocular camera. In the embodiment of the present application, the vehicle-mounted camera is a vehicle-mounted monocular camera as an example for illustration, and no limitation is set here.

Further, according to the acquired original image, a mask image corresponding to the original image is determined using a deep learning algorithm. Specifically, the texture information of the lane lines in the original image can be obtained according to the deep learning algorithm of the transformer, and further, the corresponding mask image can be obtained according to the texture information of the lane lines, wherein the mask image can include the attributes of the lane lines information.

It should be noted that in the embodiment of this application, only the deep learning algorithm of transformer is used as an example for illustration. In practical applications, R-CNN algorithm, SPP-Net algorithm, Fast R-CNN algorithm and FPN algorithm can also be used One or more of other types of deep learning algorithms can also be used, which is not limited here.

Step S203, constructing a corresponding grid map based on the mask image.

Further, the corresponding grid map is constructed according to the mask image, so as to obtain the grid map under the bird's-eye view.

Step S204, based on the mask image and the internal and external parameters of the vehicle camera, fill the lane line attribute information into the grid map to obtain the filled grid map.

Exemplarily, based on the mask image and the internal and external parameters of the vehicle camera, the position conversion relationship between the mask image and the grid map is determined, and further, according to the conversion relationship, the lane line attribute information in the mask image is filled into the grid The corresponding position in the grid map, so as to obtain the filled grid map.

Step S205, generating a local lane line map based on the filled grid map.

Exemplarily, the filled grid map is determined as a local lane line map.

In the above implementation process, the corresponding mask image is determined through the original image collected by the vehicle-mounted camera, wherein the mask image includes the attribute information of the lane line, further, the corresponding grid map is constructed according to the mask image, and according to the mask Image and the internal and external parameters of the vehicle camera, fill the lane line attribute information in the mask image into the corresponding grid map, and obtain the filled grid map, so that the filled grid map includes the lane line attribute information , further, a local lane line map is generated according to the filled grid map, so that the lane line map is generated from the image information of the visual space collected by the vehicle camera, which avoids the influence of the weather, lane line material and other factors on the lidar , leading to the problem of low integrity of the generated lane line map. The lane line map generated according to the visual space image information provided by this application has simple and clear algorithm logic, which can provide accurate real-time positioning and motion planning for intelligent driving vehicles. Prior Information.

In some of the embodiments, constructing a corresponding grid map based on the mask image may include the following steps:

Step 1: Obtain the internal and external parameters of the vehicle camera.

Step 2: Convert the mask image to a raster map based on the extrinsic and extrinsic parameters of the on-board camera.

Exemplarily, the internal and external parameters of the vehicle-mounted camera are obtained. Further, according to the internal and external parameters of the vehicle-mounted camera, the position conversion relationship between the mask image and the bird's-eye view can be determined, and further, according to the position between the mask image and the bird's-eye view The conversion relationship converts the position information of the lane line in the mask image to the bird's-eye view, so as to obtain the grid map under the bird's-eye view.

In the above implementation process, the position conversion relationship between the mask image and the bird's-eye view is determined according to the internal and external parameters of the vehicle camera, and then according to the position conversion relationship between the mask image and the bird's-eye view, and the set grid map The size and resolution determine the grid map under the bird's-eye view, thus realizing the construction of the grid map.

In some of these embodiments, based on the mask image and the internal and external parameters of the vehicle camera, filling the lane line attribute information into the grid map to obtain the filled grid map may include the following steps:

Step 1: Based on the internal and external parameters of the vehicle camera, project the grid map to the imaging plane of the vehicle camera to obtain a distortion-free map.

Step 2: Obtain the mapping relationship between the undistorted image and the distorted image in the vehicle camera.

Step 3: Obtain the distortion map corresponding to the grid map based on the undistorted map and the mapping relationship.

Step 4: Based on the mask image and the distortion map, fill the lane line attribute information into the grid map to obtain the filled grid map.

Because the image formed by the optical system on the object will be distorted relative to the object itself, that is, distortion, the direct reason is that the magnification of the edge part of the lens is different from that of the center part. The distortion of the lens cannot be eliminated, but the corresponding distorted image can be determined based on the undistorted image through the mapping relationship between the undistorted image and the distorted image in the vehicle camera, or the corresponding undistorted image can be determined based on the distorted image.

Exemplarily, according to the internal and external parameters of the vehicle camera, each grid in the grid map is projected to the imaging plane of the vehicle camera, so as to obtain the projected map, and the map obtained by the projection method is undistorted, and also That is to say, the projected map is also a distortion-free map.

Fig. 3 is a schematic diagram of an imaging plane and a perspective plane of a bird's-eye view provided by an embodiment of the present application. As shown in Fig. 3 , the coordinate system of the vehicle-mounted camera is OXYZ, the imaging plane is the plane where uν is located, and the perspective plane of the bird's-eye view is Fig. The plane where the thick line in 3 is located is the plane where the grid map is located.

Since the mask image is obtained from the original image with distortion, the mask image is distorted. In order to accurately determine the position correspondence between the mask image and the raster map, it is necessary to determine the map correspondence after projection. distorted image.

Specifically, the mapping relationship between the undistorted image and the distorted image in the vehicle camera can be obtained, and according to the undistorted map and the mapping relationship, the distorted image corresponding to the projected map can be determined, which is the distorted map corresponding to the grid map .

Further, since both the mask image and the distortion map are distorted, and the position of the lane line in the mask image is in one-to-one correspondence with the position of the lane line in the distortion map, the lane line in the mask image can be The attribute information of the line corresponds to the distorted image, and the distorted image is obtained by position conversion according to the grid map. Therefore, the attribute information of the lane line corresponding to the position of the mask image in the grid map can be determined, and the mask image The lane line attribute information of each position in the image is filled into the grid map, so as to obtain the filled grid map.

In the above implementation process, according to the internal and external parameters of the vehicle camera, the grid map is projected to the imaging plane of the vehicle camera to obtain a distortion-free map, so as to obtain a distortion-free map on the same plane as the mask image. Further, considering the vehicle-mounted The influence of the distortion of the camera on the position, and the mask image is a distorted image, then according to the mapping relationship between the undistorted image and the distorted image in the vehicle camera, the distorted image corresponding to the undistorted image is determined, so as to obtain the mask image The distorted map on the same plane, so that the lane line attribute information of the corresponding position in the distorted map can be determined according to the corresponding relationship between the mask image and the distorted map position, and the distorted map corresponds to the position of the grid map, and further, the grid map can be determined The attribute information of the lane line at the corresponding position, and fill the attribute information of the lane line at each position into the grid map, so as to obtain the filled grid map.

In some of these embodiments, obtaining the mapping relationship between the undistorted image and the distorted image in the vehicle-mounted camera may include the following steps:

Step 1: Obtain the distortion parameters of the vehicle camera.

Step 2: Determine the mapping relationship between the undistorted image and the distorted image according to the distortion parameters and the original image.

Exemplarily, the vehicle-mounted camera can be calibrated using a checkerboard to obtain the distortion parameters of the vehicle-mounted camera. Specifically, the distortion parameters of the vehicle-mounted camera can be: (k ₁ , k ₂ , p ₁ , p ₂ , k ₃ ), where, ( k ₁ , k ₂ , k ₃ ) are radial distortion parameters, and (p ₁ , p ₂ ) are tangential distortion parameters.

Further, the undistorted position corresponding to each position in the original image is determined according to the distortion parameters, so as to determine the mapping relationship between the undistorted image and the distorted image.

In the above implementation process, according to the distortion parameters of the vehicle camera and the original image, the mapping relationship between the undistorted image and the distorted image is determined, so as to facilitate the position conversion between the undistorted image and the distorted image according to the mapping relationship.

In some of these embodiments, the lane line attribute information includes a plurality of attribute categories, and after the local lane line map is generated based on the filled grid map, the method may further include the following steps:

Step 1: Obtain multiple local lane line maps corresponding to vehicle cameras in different poses.

Step 2: Based on multiple local lane line maps, determine an initial global lane line map.

Step 3: Determine the total number of observations of the target grid and the attribute category corresponding to each observation of the target grid. The target grid is any grid in the initial global lane line map.

Step 4: Determine the target attribute category of the target grid based on the number of attribute observations of the target grid in each attribute category, the total number of observations, and a preset threshold.

Step 5: Generate the target global lane map based on the target attribute categories of all grids in the initial global lane map.

Exemplarily, since the local lane lines are obtained by the vehicle-mounted camera, when the vehicle-mounted camera acquires the image, it will obtain it according to the set frequency. For example, the frequency of the vehicle-mounted camera is generally 20hz to 30hz, that is, 20 to 30 For the secondary image, 20 to 30 local lane line maps will be obtained correspondingly. Specifically, one local lane line map can be regarded as one observation, that is, each observation completes one observation of each grid in the local lane line image.

Furthermore, due to the limited size of images acquired by the vehicle-mounted camera, in order to obtain the global lane line map, the vehicle-mounted camera can acquire images in different attitudes, and then obtain local lane line maps corresponding to different attitudes.

Further, the positions of each grid in the multiple local lane line maps are combined to obtain an initial global lane line map. Specifically, the size of the initial global lane line map can be determined according to the operating environment of the vehicle. As an example , the length of the global lane map in this application is L, the width is W, and the grid resolution is δa.

Since there may be repeated observation positions in multiple local lane line maps, in the initial global lane line map, a grid may exist in several local lane line maps, or may only exist in one local lane line map Exist in , that is to say, in the initial global lane map, the total number of observations of each grid may be the same or different, and the lane attribute information may include multiple attribute categories. Specifically, the lane attribute information may include real Lines, dashed lines, stop lines, and other four attribute categories. In the initial global lane line map, each observation of each grid will get a corresponding attribute category.

Further, the total number of observations of the target grid is determined, the total number of observations is the total number of times the target grid is observed in multiple local lane line maps, and the lane line attribute category corresponding to each observation of the target grid, Among them, the target grid is any grid in the initial global lane line map.

Further, determine the number of occurrences of each attribute category of the target grid in multiple attribute categories, that is, the number of attribute observations, and further determine the target grid according to the number of attribute observations, the total number of observations, and the preset threshold The target attribute category for the raster.

Further, the target attribute categories of all grids in the initial global lane line map are determined, and the initial global lane line map in which all target attribute categories are determined is determined as the target global lane line map.

In the above implementation process, the initial global lane line map is constructed according to the local lane line maps in different attitudes of the vehicle camera, so that the initial global lane line map includes the position information of all lane lines. Further, according to each The total number of times a grid is observed, the attribute category of each observation and the preset threshold value determine the target attribute category of each grid, and then determine the target attribute category of each grid in the initial global lane line map, and The initial global grid map whose target attribute categories of all grids are determined is determined as the target global grid map, thereby realizing the generation of the global grid map.

In some of the embodiments, determining the target attribute category of the target grid based on the number of attribute observations of the target grid in each attribute category, the total number of observations, and a preset threshold may include the following steps:

Step 1: Based on the number of attribute observations of the target grid in each attribute category and the total number of observations, determine the category probability corresponding to the target grid in each attribute category.

Step 2: Determine the target attribute category of the target raster based on the probability of each category and the preset threshold.

For example, according to the number of attribute observations of the target grid in each attribute category and the total number of observations, the category probability corresponding to the target grid in each attribute category can be determined, and further, according to the attributes of each attribute category Probability, determine the target attribute grid c _i of the target grid.

Specifically, if the target grid is the i-th grid c _i in the initial global lane line map, its number of observations on solid lines, dashed lines, stop lines and other attributes are recorded as n ₁ , n ₂ , n ₃ and n ₄ , the total number of observations is n _i , and the category probability of grid c _i in each attribute category is counted separately, that is, the category probability of grid c _i is a solid line is P ₁ =n ₁ /n _i , and grid c _i The category probability of the dotted line is P ₂ =n ₂ /n _i , the category probability of the grid c _i is the stop line is P ₃ =n ₃ /n _i , and the grid c _i is the other category probability of P ₄ =n ₄ /n _i , to determine the maximum category probability, if the size of each category probability of grid _ci is P ₁ <P ₃ <P ₄ <P ₂ , then the maximum category probability of grid _ci is P ₂ .

Further, the target attribute category of the target grid is determined according to the maximum category probability and the preset threshold, and the target attribute category of the target grid is determined according to the size between the maximum category probability and the preset threshold.

Specifically, it is assumed that the preset threshold is P _r , if P _r ≤ _{P 2} , then the target attribute category of grid _ci is a dotted line, and if P _r >P ₂ , then grid _ci is a non-lane line attribute.

In the above implementation process, according to the number of attribute observations of the target grid in each attribute category and the total number of observations, determine the category probability corresponding to the target grid in each attribute category, and further determine the target according to the probability of each category and the preset threshold The target attribute category of the raster, so as to accurately determine the target attribute category of the target raster.

In some of these embodiments, after the target global lane line map is generated based on the target attribute categories of all grids in the initial global lane line map, the method may further include the following steps:

Step 1: Obtain the distribution parameters of the target global lane map.

Step 2: Based on the distribution parameters, the current number of attribute observations of the lane lines in each attribute category in the target global lane line map, and the current total number of observations of the lane lines in the target global lane line map, the target global lane line map is renew.

Exemplarily, after the target global lane line map is generated, the target global map can be deployed to an actual vehicle for application, and during the application process, the target global map can also be updated. Specifically, the method can also include: Obtain the distribution parameters of the target global lane map. Specifically, the distribution parameters can be preset distribution parameters. Further, according to the distribution parameters and the application process of the target lane map, each grid in each attribute category The current number of attribute observations and the total number of current observations are used to update the target global lane line map.

In the above implementation process, the target global lane map is updated according to the distribution parameters of the target global map, the current accumulated attribute observation times and the total number of observations of each grid in the target global lane map, so that the target global lane map can be In the practical application of the global lane marking map, the real-time update improves the accuracy of the global lane marking map.

In some of these embodiments, based on the distribution parameters, the current number of attribute observations of lane lines in each attribute category in the target global lane line map, and the current total number of observation times of lane lines in the target global lane line map, the target The update of the global lane line map may include the following steps:

Step 1: Based on the distribution parameters, the current number of attribute observations of lane lines in each attribute category in the target global lane line map, and the current total number of observations of lane lines in the target global lane line map, determine the current value of the target global lane line map. The probability distribution function of .

Step 2: Based on the current probability distribution function, update the target global lane map.

Exemplarily, the Beta distribution has important applications in statistics as the density function of the conjugate prior distribution of the Bernoulli distribution and the Binomial distribution. Therefore, in the embodiment of this application, the Beta distribution function, Beta(α, β), can be obtained according to the distribution parameters α and β, and in the actual application process, any grid in the target global lane line map is recorded at each The current number of attribute observations α ₀ and the total number of current observations β ₀ in the attribute category, thus constructing the current probability distribution function Beta(α+α ₀ , β+β ₀ ) of the target global lane line map.

Further, the target global lane line map may be updated according to the current probability distribution function Beta(α+α ₀ , β+β ₀ ).

In the above implementation process, the target global lane map can be determined according to the distribution parameters, the current attribute observation times α ₀ in each attribute category of any grid in the target global lane map, and the current total number of observations β ₀ The current probability distribution function realizes the quantification of lane line attribute information, and updates the target global lane line map according to the current probability distribution function, thereby realizing the real-time maintenance and update of the target global lane line map.

In this embodiment, an embodiment of a method for generating a lane line map is also provided. Fig. 4 is a flowchart of an embodiment of a method for generating a lane line map according to an embodiment of the present application. As shown in Fig. 4, the process includes the following steps:

In step S401, a lane line mask image is obtained based on the original image acquired by the vehicle-mounted camera.

For example, the original image is obtained based on the vehicle-mounted monocular camera, and the mask image including the lane line attribute information is obtained according to the texture information of the original image through the deep learning method, and the dense pixel-level segmentation is completed.

In recent years, the transformer deep learning algorithm has played an increasingly important role in vision tasks. In the embodiment of the present application, the vision-based transformer deep learning algorithm is used to obtain the segmentation result of the original image. The lane line attribute information in the mask image obtained by image segmentation includes four types of lane line solid line, dashed line, stop line and other categories.

Step S402, based on the mask image, determine a grid map.

Build a local grid map with preset grid map size and resolution.

Specifically, in the embodiment of this application, it is necessary to convert the mask image of the original image into a grid map under the perspective of a bird's-eye view. In the current vehicle body coordinate system, through the preset grid map size and resolution, to generate the corresponding local grid map.

Step S403, based on the mask image and the internal and external parameters of the vehicle camera, fill in the lane line attribute information in the grid map to obtain a local lane line map.

Specifically, according to the internal and external parameters of the vehicle camera, the center of each grid in the constructed local grid map is projected to the imaging plane, so as to obtain a distortion-free map on the same plane as the mask image.

Further, the mapping relationship between the undistorted image and the original image is obtained. Specifically, the vehicle-mounted camera can be calibrated with a checkerboard grid to obtain the distortion parameters of the vehicle-mounted camera. Specifically, the distortion parameters of the vehicle-mounted camera can be: (k ₁ , k ₂ , p ₁ , p ₂ , k ₃ ), where (k ₁ , k ₂ , k ₃ ) is the radial distortion parameter, (p ₁ , p ₂ ) is the tangential distortion parameter, and then the original image is determined according to the distortion parameter Each position corresponds to the undistorted position, so as to determine the mapping relationship between the undistorted image and the original image.

Further, through the mapping relationship between the undistorted image and the original image, the undistorted map on the same plane as the mask image is transformed to obtain the distorted image on the same plane as the mask image, so that the distorted image and the mask The positions in the membrane image are in one-to-one correspondence, while the distorted image is obtained by performing position conversion according to the grid map.

Furthermore, the attribute information of lane lines in the mask image can be filled into the grid map, so as to obtain a local lane line map including lane line information.

Step S404, constructing a global lane line map based on multiple local lane line maps under different attitudes of the vehicle-mounted camera, and determining the lane line attribute information of each grid in the global lane line map.

Specifically, according to the size of the working environment of the vehicle, the length of the global lane line map is set to L, the width is set to W, and the grid resolution is δa, and it is constructed based on multiple local lane line maps under different attitudes of the vehicle camera Global lane line map.

Further, the number of attribute observations and the total number of observations of each grid on each attribute category in the global lane line map are counted, and the ratio of the number of attribute observations on each attribute category to the total number of observations is determined as each The category probability of the attribute category, furthermore, the value with the largest category probability is reserved.

Further, the value with the highest probability and the preset probability threshold are determined. If the value with the highest probability is greater than or equal to the preset probability threshold, the attribute category corresponding to the maximum probability is the target attribute category of the grid, thereby determining Lane line attribute information of each grid in the global lane line map.

Apply the final global lane map to the vehicle.

Step S405, based on the distribution parameters, the number of observations of each grid in each attribute category in the global lane marking map, and the total number of observations, the global lane marking map is updated.

During the application process of the global lane marking map, the number of attribute observations and the total number of observations currently acquired in real time for each grid in the global lane marking map are recorded, and the global lane marking map is updated in combination with preset distribution parameters.

Specifically, given the parameters α and β, the distribution Beta(α, β) is obtained. During the update process, the number of grid attribute observations and the total number of observations of the current lane line are respectively recorded as α ₀ , β ₀ , so the new The distribution of Beta(α+α ₀ , β+β ₀ ). On this basis, the mean value of the new Beta distribution can be calculated to update and maintain the attributes of the corresponding grid.

In the above implementation process, the lane line mask image is obtained from the original image based on the deep learning method; then a local grid map is constructed to obtain the lane line information from the perspective of the bird's-eye view; and then combined with the real-time pose information of the current camera to construct Lane line statistical grid map, and determine the attributes of each grid according to the preset probability threshold to generate a global lane line map; finally, in the actual application process, the global lane line map is updated and maintained by introducing Beta distribution. FIG. 5 is a schematic diagram of a lane line map in a park environment provided by an embodiment of the present application.

It should be noted that although the steps in the flow charts involved in the above-mentioned embodiments are displayed sequentially according to the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flow charts involved in the above-mentioned embodiments may include multiple steps or stages, and these steps or stages are not necessarily executed at the same time, but may be performed at different times For execution, the execution order of these steps or stages is not necessarily performed sequentially, but may be executed in turn or alternately with other steps or at least a part of steps or stages in other steps.

This embodiment also provides a lane line map generation device, which is used to implement the above embodiments and preferred implementation modes, and what has been described will not be repeated. The terms "module", "unit", "subunit" and the like used hereinafter may be a combination of software and/or hardware that realizes a predetermined function. Although the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

Fig. 6 is a structural block diagram of a lane line map generation device provided by an embodiment of the present application. As shown in Fig. 6, the device includes:

The obtaining module 601 is used to obtain the original image collected by the vehicle-mounted camera.

The determining module 602 is configured to determine a corresponding mask image based on the original image, where the mask image includes lane line attribute information.

A construction module 603, configured to construct a corresponding grid map based on the mask image.

The filling module 604 is configured to fill the lane line attribute information into the grid map based on the mask image and the internal and external parameters of the vehicle camera to obtain the filled grid map.

A generating module 605, configured to generate a local lane line map based on the filled grid map.

In some of these embodiments, the construction module 603 is specifically used for:

Obtain the internal and external parameters of the vehicle camera;

Convert the mask image to a raster map based on the extrinsic and extrinsic parameters of the on-board camera.

In some of these embodiments, the filling module 604 is specifically used for:

Based on the internal and external parameters of the vehicle camera, the grid map is projected to the imaging plane of the vehicle camera to obtain a distortion-free map;

Obtain the mapping relationship between the undistorted image and the distorted image in the vehicle camera;

Based on the undistorted map and the mapping relationship, the distortion map corresponding to the grid map is obtained;

Based on the mask image and the distortion map, the lane line attribute information is filled into the grid map to obtain the filled grid map.

In some of these embodiments, the lane line attribute information includes multiple attribute categories, and the generation module 605 is also used for:

Obtain multiple local lane line maps corresponding to vehicle cameras in different poses;

Determining an initial global lane line map based on multiple local lane line maps;

Determine the total number of observations of the target grid and the attribute category corresponding to each observation of the target grid. The target grid is any grid in the initial global lane line map;

Determine the target attribute category of the target grid based on the number of attribute observations of the target grid in each attribute category, the total number of observations, and a preset threshold;

Based on the target attribute categories of all rasters in the initial global lane line map, a target global lane line map is generated.

In some of these embodiments, the generation module 605 is specifically used to:

Based on the number of attribute observations of the target grid in each attribute category and the total number of observations, determine the category probability corresponding to the target grid in each attribute category;

Determine the target attribute category of the target grid based on the probabilities of each category and the preset threshold.

In some of these embodiments, the generating module 605 is also used for:

Obtain the distribution parameters of the target global lane line map;

Based on the distribution parameters, the current number of attribute observations of the lanes in each attribute category in the target global lane map, and the current total number of observations of the lanes in the target global lane map, the target global lane map is updated.

In some of these embodiments, the generation module 605 is specifically configured to: based on the distribution parameter, the current number of attribute observations of lane lines in each attribute category in the target global lane line map, and the number of lane lines in the target global lane line map The current total number of observations determines the current probability distribution function of the target global lane line map;

Based on the current probability distribution function, the target global lane line map is updated.

It should be noted that each of the above-mentioned modules may be a function module or a program module, and may be realized by software or by hardware. For the modules implemented by hardware, the above modules may be located in the same processor; or the above modules may be located in different processors in any combination.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure may be as shown in FIG. 7 . Fig. 7 is a schematic diagram of an internal structure of a computer device provided by an embodiment of the present application. The computer device includes a processor, memory and a network interface connected by a system bus. Wherein, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs and databases. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store the data acquired by the vehicle-mounted camera. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a lane line map generation method is realized.

Those skilled in the art can understand that the structure shown in Figure 7 is only a block diagram of a part of the structure related to the solution of this application, and does not constitute a limitation to the computer equipment on which the solution of this application is applied. The specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.

In one embodiment, an electronic device is also provided, including a memory and a processor, where a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the foregoing method embodiments are implemented.

It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all Information and data authorized by the user or fully authorized by all parties.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above-mentioned embodiments can be completed by instructing related hardware through computer programs, and the computer programs can be stored in a non-volatile computer-readable memory In the medium, when the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any reference to storage, database or other media used in the various embodiments provided in the present application may include at least one of non-volatile and volatile storage. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive variable memory (ReRAM), magnetic variable memory (Magnetoresistive Random Access Memory, MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (Phase Change Memory, PCM), graphene memory, etc. The volatile memory may include a random access memory (Random Access Memory, RAM) or an external cache memory and the like. As an illustration but not a limitation, the RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM). The databases involved in the various embodiments provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a blockchain-based distributed database, etc., but is not limited thereto. The processors involved in the various embodiments provided by this application can be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., and are not limited to this.

The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered to be within the range described in this specification.

The above-mentioned embodiments only express several implementation modes of the present application, and the description thereof is relatively specific and detailed, but should not be construed as limiting the protection scope of the patent. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be determined by the appended claims.

Claims (10)

1. A lane line map generation method, characterized by comprising:

acquiring an original image acquired by a vehicle-mounted camera;

determining a corresponding mask image based on the original image, wherein the mask image comprises lane line attribute information;

constructing a corresponding grid map based on the mask image;

filling the lane line attribute information into the grid map based on the mask image and the internal and external parameters of the vehicle-mounted camera to obtain a filled grid map;

And generating a local lane line map based on the filled grid map.

2. The lane line map generation method according to claim 1, wherein the constructing a corresponding grid map based on the mask image includes:

acquiring internal and external parameters of the vehicle-mounted camera;

and converting the mask image into the grid map based on the internal and external parameters of the vehicle-mounted camera.

3. The lane line map generation method according to claim 1, wherein the filling the lane line attribute information into the grid map based on the mask image and the internal and external parameters of the in-vehicle camera to obtain a filled grid map comprises:

based on the internal and external parameters of the vehicle-mounted camera, projecting the grid map to an imaging plane of the vehicle-mounted camera to obtain a distortion-free map;

acquiring a mapping relation between an undistorted image and a distorted image in the vehicle-mounted camera;

obtaining a distortion map corresponding to the grid map based on the undistorted map and the mapping relation;

and filling the lane line attribute information into the grid map based on the mask image and the distortion map to obtain a filled grid map.

4. The lane-marking map generating method according to claim 1, wherein the lane-marking attribute information includes a plurality of attribute categories, and after the generating a local lane-marking map based on the filled grid map, the method further comprises:

acquiring a plurality of local lane line maps corresponding to the vehicle-mounted camera under different poses;

determining an initial global lane line map based on a plurality of the local lane line maps;

determining the total number of observations of a target grid and attribute categories corresponding to each observation of the target grid, wherein the target grid is any grid in the initial global lane line map;

determining a target attribute category of the target grid based on the attribute observation times of the target grid in each attribute category, the total observation times and a preset threshold;

and generating a target global lane line map based on target attribute categories of all grids in the initial global lane line map.

5. The lane-line map generation method according to claim 4, wherein the determining the target attribute category of the target grid based on the number of attribute observations of the target grid in each attribute category, the total number of observations, and a preset threshold value includes:

Determining class probability corresponding to each attribute class of the target grid based on the attribute observation times of the target grid in each attribute class and the observation total times;

and determining the target attribute category of the target grid based on the category probabilities and the preset threshold.

6. The lane-line map generation method according to claim 4, wherein after generating a target global lane-line map based on target attribute categories of all grids in the initial global lane-line map, the method further comprises:

acquiring distribution parameters of the target global lane line map;

updating the target global lane line map based on the distribution parameters, the current attribute observation times of the lane lines in the target global lane line map in each attribute category and the current total observation times of the lane lines in the target global lane line map.

7. The lane-line map generation method according to claim 6, wherein the updating the target global lane-line map based on the distribution parameter, a current number of attribute observations of lane lines in the target global lane-line map in each attribute category, and a current total number of observations of lane lines in the target global lane-line map, comprises:

Determining a current probability distribution function of the target global lane line map based on the distribution parameters, the current attribute observation times of the lane lines in the target global lane line map in each attribute category and the current total observation times of the lane lines in the target global lane line map;

and updating the target global lane line map based on the current probability distribution function.

8. A lane line map generation apparatus, comprising:

the acquisition module is used for acquiring an original image acquired by the vehicle-mounted camera;

the determining module is used for determining a corresponding mask image based on the original image, wherein the mask image comprises lane line attribute information;

the construction module is used for constructing a corresponding grid map based on the mask image;

the filling module is used for filling the lane line attribute information into the grid map based on the mask image and the internal and external parameters of the vehicle-mounted camera to obtain a filled grid map;

and the generation module is used for generating a local lane line map based on the filled grid map.

9. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, the processor being arranged to run the computer program to perform the lane line map generation method of any one of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the lane line map generation method of any one of claims 1 to 7.

Images