CN115638788A

CN115638788A - Semantic vector map construction method, computer equipment and storage medium

Info

Publication number: CN115638788A
Application number: CN202211660619.6A
Authority: CN
Inventors: 请求不公布姓名
Original assignee: Anhui Weilai Zhijia Technology Co Ltd
Current assignee: Anhui Weilai Zhijia Technology Co Ltd
Priority date: 2022-12-23
Filing date: 2022-12-23
Publication date: 2023-01-24
Anticipated expiration: 2042-12-23
Also published as: CN115638788B

Abstract

The invention relates to the technical field of automatic driving, in particular to a construction method of a semantic vector map, computer equipment and a storage medium, and aims to solve the problems of improving the map construction efficiency of the semantic vector map and reducing the map construction cost. For the purpose, the method provided by the invention comprises the steps of establishing a point cloud map according to a sensor data frame acquired by a sensor on a vehicle; semantic vector perception of map elements is respectively carried out on the sensor data frame and the point cloud map through a first semantic vector perception model and a second semantic vector perception model so as to obtain first semantic vector information and second semantic vector information; and establishing a semantic vector map based on the point cloud map and according to the first and/or second semantic vector information. By the method, the semantic vector information of the map elements can be automatically acquired through the semantic vector perception model, and the semantic vector map is automatically established according to the point cloud map and the acquired semantic vector information, so that the map construction cost can be reduced and the map construction efficiency can be improved on the premise of ensuring the map construction quality.

Description

Semantic vector map construction method, computer equipment and storage medium

Technical Field

The invention relates to the technical field of automatic driving, in particular to a semantic vector map construction method, computer equipment and a storage medium.

Background

After a point cloud map is established by using data acquired by a camera or a laser radar and the like, the semantic vector map can be obtained only by vectorizing and marking the elements such as lane lines, parking spaces and the like on the point cloud map and generating topological information among the elements. In order to improve the accuracy of vectorization labeling and topology information generation, the conventional semantic vector map construction method mainly manually labels each element on a point cloud map manually in a vector manner and manually generates topology information among the elements. Because the method depends heavily on manual map building, the method brings higher manual map building cost and also influences the building efficiency of the semantic vector map.

Accordingly, there is a need in the art for a new solution to the above problems.

Disclosure of Invention

In order to overcome the above-mentioned drawbacks, the present invention is proposed to provide a semantic vector map construction method, a computer device, and a storage medium that solve or at least partially solve the technical problem of how to improve the mapping efficiency and reduce the mapping cost of the semantic vector map.

In a first aspect, a method for constructing a semantic vector map is provided, the method comprising:

establishing a point cloud map according to a sensor data frame acquired by a sensor on a vehicle;

performing semantic vector perception of map elements on the sensor data frame through a first semantic vector perception model to obtain first semantic vector information of the map elements;

performing semantic vector perception of map elements on the point cloud map through a second semantic vector perception model to acquire second semantic vector information of the map elements;

and establishing a semantic vector map based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information.

In a technical solution of the method for constructing a semantic vector map, "establishing a semantic vector map based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information" specifically includes:

generating map mother database data which accord with the specification of a preset map mother database based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information;

obtaining a map product specification of a semantic vector map to be established;

converting the map database data into map product data meeting the specification of the map product;

and establishing a semantic vector map according to the map product data.

In one technical solution of the above method for constructing a semantic vector map, after the step of "establishing a semantic vector map based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information", the method further includes:

and performing quality inspection on the semantic vector map to obtain the semantic vector map with qualified quality inspection.

In one technical solution of the method for constructing a semantic vector map, after the step of performing quality inspection on the semantic vector map to obtain a semantic vector map qualified in quality inspection, the method further includes:

respectively acquiring model training data of the first semantic vector perception model and the second semantic vector perception model according to the semantic vector map qualified in quality inspection;

and performing model training on the first semantic vector perception model and the second semantic vector perception model respectively by adopting the model training data so as to optimize the first semantic vector perception model and the second semantic vector perception model.

In one technical solution of the method for constructing a semantic vector map, the step of "respectively obtaining model training data of the first semantic vector perceptual model and the second semantic vector perceptual model according to the semantic vector map qualified by quality inspection" specifically includes:

generating map mother base training data which accord with the specification of a preset map mother base according to the semantic vector map qualified in quality inspection;

and respectively acquiring model training data of the first semantic vector perception model and the second semantic vector perception model according to the map mother base training data.

In one technical solution of the construction method of the semantic vector map, the data coordinate system of the training data of the map mother base is a global coordinate system, and the step of respectively obtaining the model training data of the first semantic vector perceptual model and the model training data of the second semantic vector perceptual model according to the training data of the map mother base specifically includes obtaining the model training data of the first semantic vector perceptual model in the following manner:

converting the map mother library training data from the global coordinate system to a sensor coordinate system of the sensor to obtain the map mother library training data in the sensor coordinate system;

and obtaining model training data of the first semantic vector perception model according to the map master library training data in the sensor coordinate system.

In one embodiment of the construction method of the semantic vector map, "converting the map mother library training data from the global coordinate system to the sensor coordinate system of the sensor to obtain the map mother library training data in the sensor coordinate system" includes:

acquiring the acquisition time of a sensor data frame corresponding to each map mother library training data;

acquiring the pose of the vehicle at the acquisition time of each sensor data frame, wherein the pose is the pose converted from a vehicle body coordinate system to a global coordinate system;

and respectively converting the training data of each map database from the global coordinate system to the vehicle body coordinate system and then from the vehicle body coordinate system to the sensor coordinate system according to the pose.

In one technical solution of the construction method of the semantic vector map, the step of "acquiring the pose of the vehicle at the acquisition time of each sensor data frame" specifically includes:

forming a dense track of the vehicle according to the pose of the vehicle at each IMU data acquisition time;

and respectively acquiring the pose of the vehicle at the acquisition time of each sensor data frame according to the dense track of the vehicle.

In one technical solution of the construction method of the semantic vector map, the step of performing semantic vector sensing of the map element on the sensor data frame through the first semantic vector sensing model to obtain the first semantic vector information of the map element specifically includes:

respectively sensing semantic vectors of map elements for each sensor data frame acquired by the sensor through a first semantic vector sensing model to acquire semantic vector information of the map elements corresponding to each sensor data frame;

and fusing semantic vector information of the map elements corresponding to each sensor data frame to acquire first semantic vector information of the map elements.

In one technical solution of the method for constructing a semantic vector map, "performing semantic vector sensing on a map element on the sensor data frame through a first semantic vector sensing model to acquire first semantic vector information of the map element" further includes:

acquiring a first semantic vector perception model corresponding to each sensor;

respectively sensing semantic vectors of map elements on sensor data frames acquired by each sensor through a first semantic vector sensing model corresponding to each sensor to acquire first semantic vector information of the map elements corresponding to each sensor;

and fusing the first semantic vector information of the map elements corresponding to each sensor to obtain the final first semantic vector information.

fusing the first semantic vector information and the second semantic vector information to obtain third semantic vector information of the map elements;

and establishing a semantic vector map according to the point cloud map and the third semantic vector information.

In one technical solution of the above method for constructing a semantic vector map, "establishing a semantic vector map based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information" further includes:

establishing an initial semantic vector map based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information;

obtaining an artificial labeling map obtained by performing map element semantic vector labeling on the point cloud map in an artificial labeling mode;

and establishing a final semantic vector map according to the initial semantic vector map and/or the artificial labeling map.

In a technical solution of the construction method of the semantic vector map, the step of "establishing a final semantic vector map according to the initial semantic vector map and/or the artificial annotation map" specifically includes:

if the current map building mode is an automatic mode, building a final semantic vector map according to the initial semantic vector map;

if the current map building mode is an artificial mode, building a final semantic vector map according to the artificial labeled map;

and if the current map building mode is a mixed mode, building a final semantic vector map according to the initial semantic vector map and the manual labeling map.

In a second aspect, there is provided a computer device comprising a processor and a storage means, the storage means being adapted to store a plurality of program codes, the program codes being adapted to be loaded and run by the processor to perform the method of constructing a semantic vector map according to any one of the above-mentioned aspects of the method of constructing a semantic vector map.

In a third aspect, there is provided a computer readable storage medium having stored therein a plurality of program codes adapted to be loaded and executed by a processor to execute the method of constructing a semantic vector map according to any one of the above-mentioned aspects.

One or more technical schemes of the invention at least have one or more of the following beneficial effects:

in the technical scheme for constructing the semantic vector map, a point cloud map can be established according to a sensor data frame acquired by a sensor on a vehicle; semantic vector perception of map elements is carried out on the sensor data frame through a first semantic vector perception model so as to obtain first semantic vector information of the map elements; performing semantic vector perception on map elements on the point cloud map through a second semantic vector perception model to obtain second semantic vector information of the map elements; and finally, establishing a semantic vector map based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information. Through the embodiment, the semantic vector information of the map elements can be automatically acquired through the semantic vector perception model, and then the semantic vector map can be automatically established according to the point cloud map and the acquired semantic vector information, so that the problems that in the prior art, manual map building is seriously relied on, the map building cost is high, and the map building efficiency is low are solved.

Furthermore, in the technical scheme of the invention, model training data of the first semantic vector perception model and the second semantic vector perception model can be respectively obtained according to the qualified semantic vector map of the quality inspection; and then model training can be carried out on the first semantic vector perception model and the second semantic vector perception model respectively by adopting model training data so as to optimize the first semantic vector perception model and the second semantic vector perception model. Model training data are obtained according to the semantic vector map qualified in quality inspection, and model training is carried out on the semantic vector perception model, so that the perception capability of the model can be improved, and the accuracy of automatic map building is further improved. In addition, the established semantic vector map is reused to obtain model training data, extra manual resources are not occupied, and the obtaining cost of the model training data can be reduced.

Further, in the technical scheme of implementing the invention, besides establishing the initial semantic vector map based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information, an artificial labeling map obtained by performing map element semantic vector labeling on the point cloud map in an artificial labeling mode can be obtained, and then establishing the final semantic vector map according to the initial semantic vector map and/or the artificial labeling map. Furthermore, a mapping mode of the semantic vector map can be set, and the mapping mode can comprise an automatic mode, a manual mode and a mixed mode. If the current map building mode is an automatic mode, a final semantic vector map can be built according to the initial semantic vector map; if the current map building mode is an artificial mode, a final semantic vector map can be built according to an artificial labeled map; if the current map building mode is a mixed mode, a final semantic vector map can be built according to the initial semantic vector map and the manual labeling map.

Through the embodiment, the mapping mode can be flexibly set according to the actual mapping requirement, and then the semantic vector map is established by adopting different mapping modes. For example, according to the perception capability of the semantic vector perception model, a semantic vector map can be built by adopting a manual mode, a mixed mode and an automatic mode step by step.

Drawings

The present disclosure will become more readily understood with reference to the accompanying drawings. As is readily understood by those skilled in the art: these drawings are for illustrative purposes only and are not intended to constitute a limitation on the scope of the present invention. Wherein:

FIG. 1 is a flow chart illustrating the main steps of a semantic vector map construction method according to an embodiment of the present invention;

FIG. 2 is a schematic flow chart of a point cloud map building method according to an embodiment of the invention;

FIG. 3 is a main structural diagram of a factor graph according to an embodiment of the present invention;

FIG. 4 is a flow chart illustrating the main steps of a method for obtaining first semantic vector information according to one embodiment of the present invention;

FIG. 5 is a flow chart illustrating the main steps of a method for obtaining first semantic vector information according to another embodiment of the invention;

FIG. 6 is a flow chart illustrating the main steps of a method for building a semantic vector map based on a point cloud map and based on semantic vector information according to an embodiment of the present invention;

FIG. 7 is a flow chart illustrating the main steps of a method for building a semantic vector map based on a point cloud map and based on semantic vector information according to another embodiment of the present invention;

FIG. 8 is a flow diagram illustrating the main steps of a method for optimizing a semantic vector perception model according to an embodiment of the present invention;

FIG. 9 is a schematic diagram of coordinate system conversion of a map corpus training data, in accordance with one embodiment of the present invention;

fig. 10 is a block diagram illustrating a main structure of a semantic vector map constructing apparatus according to an embodiment of the present invention;

fig. 11 is a main configuration diagram of a computer apparatus according to an embodiment of the present invention.

Detailed Description

Some embodiments of the invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only for explaining the technical principle of the present invention, and are not intended to limit the scope of the present invention.

In the description of the present invention, a "processor" may include hardware, software, or a combination of both. The processor may be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and/or signal processing functionality. The processor may be implemented in software, hardware, or a combination thereof. Computer readable storage media include any suitable medium that can store program code such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random-access memory, and the like. The term "a and/or B" denotes all possible combinations of a and B, such as a alone, B alone or a and B.

An embodiment of the construction method of the semantic vector map is explained below.

Referring to fig. 1, fig. 1 is a flow chart illustrating main steps of a semantic vector map construction method according to an embodiment of the present invention. As shown in fig. 1, the method for constructing a semantic vector map in the embodiment of the present invention mainly includes the following steps S101 to S104.

Step S101: and establishing a point cloud map according to the sensor data frame acquired by the sensor on the vehicle.

The sensor includes but is not limited to an image acquisition device and a laser radar, and the sensor data frame includes but is not limited to an image frame acquired by the image acquisition device and a point cloud frame acquired by the laser radar. The point cloud map is a three-dimensional map established by using three-dimensional point cloud data acquired by using an image frame or a point cloud frame, and a map coordinate system of the map can be a global coordinate system.

The method for creating a point cloud map will be briefly described with reference to fig. 2, wherein the sensor 1 and the sensor 2 in fig. 2 may be laser radars with different field angles.

After the sensor data frames acquired by the sensor 1 and the sensor 2 are acquired, the key frame is selected according to the pose of each sensor data frame, and then a factor graph is constructed according to the key frame and IMU/GNSS data. After the factor graph is constructed, the pose of each key frame can be optimized based on the factor graph, namely, the pose of the key frame is solved and the pose of the key frame is obtained. After the pose of the key frame is obtained, each key frame can be spliced (point cloud map splicing) according to the pose of each key frame, so that a point cloud map is obtained. In addition, sensor data belonging to dynamic objects can be removed, and/or sensor data belonging to objects in a preset region of no interest on the ground can be removed, and the point cloud map after the sensor data are removed is used as a final point cloud map.

The key frame refers to a sensor data frame with a pose variation larger than a preset threshold compared with a previous sensor data frame. A person skilled in the art may flexibly set a specific value of the preset threshold according to an actual requirement, which is not specifically limited in the embodiment of the present invention. The pose variation is greater than the preset threshold, which means that at least one of the position variation and the pose variation is greater than the corresponding preset threshold.

In the embodiment of the present invention, a conventional factor graph construction method in the field of automatic driving technology may be adopted to construct a factor graph according to a key frame and IMU (Inertial Measurement Unit)/GNSS (Global Navigation Satellite System) data. For example, referring to fig. 3, the factor graph may include N factor nodes, an IMU pre-integration constraint term, an inter-frame matching constraint term, a loop detection constraint term, and an absolute pose constraint term (not shown in fig. 3), with each keyframe corresponding to each factor node. The IMU pre-integration constraint term is used for constraining the actual relative pose between two adjacent key frames by using the pose obtained by IMU pre-integration; the inter-frame matching constraint item is used for constraining the actual relative pose between two adjacent key frames by using the optimal relative pose between the two adjacent key frames. The loop detection constraint item is used for constraining the actual relative pose between two key frames by using the optimal relative pose between the two key frames which can form a loop relation; the absolute pose constraint item is used for constraining the actual pose of the key frame by using the absolute pose obtained by the GNSS. As shown in fig. 3, the key frame 2 and the key frame N can form a loop relationship, and therefore a loop detection constraint term is set between the key frame 2 and the key frame N. It should be noted that, in the embodiment of the present invention, a conventional factor graph constraint item construction method in the field of automatic driving technology may be adopted to construct the above constraint items. In addition, a conventional method for optimizing the pose based on the factor graph in the technical field of automatic driving may also be adopted, and the pose of each key frame is optimized according to the factor graph, which is not specifically limited in the embodiment of the present invention.

Step S102: and performing semantic vector perception on the map elements on the sensor data frame through the first semantic vector perception model to acquire first semantic vector information of the map elements.

The first semantic vector perception model is a pre-trained model capable of perceiving semantic vector information of the map elements from the sensor data frames, wherein the semantic vector information includes semantic information (such as lane lines) and vector information (such as positions and directions) of the map elements. After the model is obtained, the sensor data frame is input into the model, and the model can output first semantic vector information of the map element.

Map elements include, but are not limited to, lane lines, parking spaces, and the like.

Step S103: and performing semantic vector perception of map elements on the point cloud map through a second semantic vector perception model to acquire second semantic vector information of the map elements.

Similar to the first semantic vector perception model, the second semantic vector perception model is also a pre-trained model capable of sensing map element semantic vector information from the point cloud map. After obtaining the model, inputting the point cloud map into the model, and the model can output second semantic vector information of map elements.

Step S104: and establishing a semantic vector map based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information.

After obtaining the semantic vector information (the first semantic vector information and/or the second semantic vector information) of each map element, the semantic vector information of each map element may be labeled on the point cloud map, so as to obtain the semantic vector map. Specifically, when only the first semantic vector information is acquired, the semantic vector information of the map elements can be labeled on the point cloud map according to the first semantic vector information; when only the second semantic vector information is acquired, the semantic vector information of the map elements can be marked on the point cloud map according to the second semantic vector information; when the first semantic vector information and the second semantic vector information are obtained simultaneously, the first semantic vector information and the second semantic vector information can be fused to obtain third semantic vector information of map elements, and then a semantic vector map is established according to the point cloud map and the third semantic vector information, namely the semantic vector information of the map elements is marked on the point cloud map according to the third semantic vector information. In some embodiments, when the first semantic vector information and the second semantic vector information are fused, confidence degrees of the sensor and the point cloud map may be set in advance, and then the first semantic vector information and the second semantic vector information are fused according to the confidence degrees of the sensor and the point cloud map to obtain third semantic vector information. The skilled person can flexibly set the confidence level of the sensor and the point cloud map according to the actual requirement, for example, the confidence level can be set according to the error of the sensor and the point cloud map, which is not specifically limited in the present invention. In addition, in the embodiment of the present invention, a conventional method of performing data fusion by using confidence may be adopted to perform fusion on the first semantic vector information and the second semantic vector information, for example, for the same map element, if the first semantic vector information is inconsistent with the second semantic vector information, the semantic vector information with higher confidence is regarded as the third semantic vector information, and this is not specifically limited in the embodiment of the present invention.

Based on the method described in the above steps S101 to S104, the semantic vector information of the map elements can be automatically obtained by using the semantic vector perception model, and then the semantic vector map is automatically established according to the point cloud map and the semantic vector information, so that the degree of dependence on manual map establishment is reduced, thereby reducing the map establishment cost of the semantic vector map and improving the map establishment efficiency.

The following describes step S102 and step S104.

1. Step S102 will be explained.

When a point cloud map is built, multiple sensor data frames are usually spliced together to form the point cloud map, and map elements contained in different sensor data frames may be different, so that semantic vector information contained in different sensor data frames may be different. Further, even if the same map element is included, the vector information of the map element may be different. In order to accurately sense the first semantic vector information of the map element from the sensor data frame, the first semantic vector information of the map element may be acquired through steps S1021 to S1022 shown in fig. 4 in some embodiments.

Step S1021: and respectively sensing the semantic vector of the map element on each sensor data frame acquired by the sensor through the first semantic vector sensing model so as to acquire the semantic vector information of the map element corresponding to each sensor data frame.

Specifically, each sensor data frame is respectively input into a first semantic vector perception model, and semantic vector information contained in each sensor data frame is respectively obtained through the model.

Step S1022: and fusing semantic vector information of the map elements corresponding to each sensor data frame to acquire first semantic vector information of the map elements. That is, the first semantic vector information is a set of semantic vector information for a plurality of frames of sensor data.

Specifically, each sensor data frame may include a plurality of different map elements, so that when semantic vector information of different sensor data frames is fused, semantic vector information belonging to the same map element on different sensor data frames may be acquired, and then the semantic vector information belonging to the same map element is fused to form first semantic vector information of the map element.

When semantic vector information is fused, the semantic vector information of a plurality of sensor data frames can be directly combined together to form first semantic vector information, the semantic vector information of each sensor data frame can be optimized through factor graph optimization or Kalman Filtering and the like to obtain more accurate semantic vector information, and then the semantic vector information is combined together to form the first semantic vector information.

The method of factor graph optimization and kalman filtering is briefly described below.

1. Factor graph optimization

In the embodiment of the invention, a factor graph can be established according to the sensor data frames, factor nodes on the factor graph correspond to each sensor data frame one by one, constraint items are also set on the factor graph, and semantic vector information of two adjacent sensor data frames is constrained through the constraint items, or semantic vector information of two sensor data frames forming a loop relation is constrained, and the like. Those skilled in the art can flexibly set the specific content of the constraint item according to actual requirements, and the embodiment of the present invention is not particularly limited thereto. In addition, in the embodiment of the present invention, a conventional method for optimizing data based on a factor graph in the field of automatic driving technology may be adopted, and semantic vector information of each sensor data frame is optimized based on the factor graph, which is not specifically limited in the embodiment of the present invention.

2. Kalman filtering

In the embodiment of the invention, a semantic vector information estimation model can be established based on a Kalman filtering theory, the semantic vector information of each sensor data frame obtained by a first semantic vector perception model is used as an observed quantity, and the optimal semantic vector information is estimated based on the semantic vector information estimation model and according to the observed quantity. A person skilled in the art may use a conventional kalman filtering method in the field of the automatic driving technology to establish a semantic vector information estimation model and estimate to obtain optimal semantic vector information based on the semantic vector information estimation model and according to the observed quantity, which is not specifically limited in the embodiment of the present invention.

Based on the steps S1021 to S1022, accurate first semantic vector information can be acquired, which is favorable for establishing and acquiring an accurate semantic vector map.

Further, in practical applications, a semantic vector map may be built by using sensor data frames acquired by a plurality of sensors, for example, a semantic vector map may be built by using point cloud frames acquired by two laser radars with different field angles. In some embodiments, a first semantic vector perception model applicable to a plurality of sensors may be trained in advance, and semantic vector perception of map elements is performed on sensor data frames acquired by each sensor through the first semantic vector perception model. Since a plurality of different sensors are used, the sensing ability of a part of the sensors may be sacrificed in order to ensure the overall sensing ability of the plurality of sensors. In other embodiments, in order to ensure the sensing capability of each sensor, a first semantic vector sensing model may be set for each sensor, and then the sensor data frame acquired by each sensor may be sensed according to the first semantic vector sensing model corresponding to each sensor.

Specifically, referring to fig. 5, in the present embodiment, the first semantic vector information of the map element may be acquired through the following steps S201 to S203.

Step S201: and acquiring a first semantic vector perception model corresponding to each sensor. Before the first semantic vector information is acquired, a model for semantic vector information perception is trained for each sensor, and the first semantic vector perception model corresponding to each sensor is called when the first semantic vector information needs to be acquired.

Step S202: and respectively sensing semantic vectors of map elements on the sensor data frames acquired by each sensor through the first semantic vector sensing model corresponding to each sensor so as to acquire first semantic vector information of the map elements corresponding to each sensor.

For each sensor, the method of the foregoing steps S1021 to S1022 may be adopted to acquire the first semantic vector information of the map element.

Step S203: and fusing the first semantic vector information of the map elements corresponding to each sensor to obtain the final first semantic vector information.

When the first semantic vector information of different sensors is fused, the confidence of each sensor can be preset, and then the first semantic vector information of each sensor is fused according to the confidence of each sensor to obtain the final first semantic vector information.

The confidence of each sensor can be flexibly set by a person skilled in the art according to actual requirements, for example, the confidence can be set according to the error of the sensor, and the invention is not particularly limited thereto. In addition, in the embodiment of the present invention, a conventional method of performing data fusion by using confidence degrees may be adopted to perform fusion on the first semantic vector information of different sensors, for example, for the same map element, if the first semantic vector information of different sensors is inconsistent, the semantic vector information with a higher confidence degree is taken as the reference, and is used as the final first semantic vector information, which is also not specifically limited in the embodiment of the present invention.

Based on the method described in the above steps S201 to S203, the first semantic vector information of the map elements can be accurately obtained under the condition that the semantic vector map is established by using the sensor data frames acquired by the plurality of sensors, thereby facilitating establishment of the accurate semantic vector map.

2. Step S104 will be explained.

In practical application, according to different map building requirements, even aiming at the same area, semantic vector maps meeting different map product specifications can be built. In order to facilitate building semantic vector maps of different map product specifications, in some embodiments, the semantic vector maps may be built through the following steps S1041 to S1044 shown in fig. 6.

Step S1041: and generating map mother database data which accords with the specification of a preset map mother database based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information.

Step S1042: and obtaining the map product specification of the semantic vector map to be established.

Step S1043: and converting the map mother database data into map product data meeting the specification of the map product.

Step S1044: and establishing a semantic vector map according to the map product data.

The preset map mother library specification is a basic description mode of a physical world aiming at the characteristics identified by different sensors; the map product specification is a specific description of the physical world according to the design requirements of the mapping project or the map product demand side. According to the design requirements of map building projects or map product demand sides, specification adjustment is carried out on the map master database data, and map product data can be obtained.

Based on the method described in the above steps S1041 to S1044, map master library data that is not limited by the design requirement of the map building project or the map product demand side can be generated, and when a semantic vector map of a certain map product specification needs to be built, map product data that meets the map product specification can be obtained only by performing specification adjustment on the map master library data, so that the semantic vector map that meets the map product specification is quickly built.

In the embodiment of the invention, the semantic vector map can be automatically sensed and established by the method of the method embodiment, and the semantic vector map can also be established by combining manual means to different degrees. Specifically, referring to fig. 7, in some embodiments of the above step S104, the semantic vector map may be built through the following steps S301 to S303.

Step S301: and establishing an initial semantic vector map based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information. That is, the semantic vector map automatically created by the method described in the foregoing method embodiment is used as the initial semantic vector map.

Step S302: and obtaining the manual labeling map obtained by performing map element semantic vector labeling on the point cloud map in a manual labeling mode.

Step S303: and establishing a final semantic vector map according to the initial semantic vector map and/or the artificial labeling map.

Specifically, according to different mapping requirements, a final semantic vector map can be established only according to an initial semantic vector map or a manual labeling map, or a final semantic vector map can be established according to the initial semantic vector map and the manual labeling map at the same time.

Further, in order to determine which way to establish the final semantic vector map, three mapping modes including an automatic mode, a manual mode and a mixed mode may be set. If the current mapping mode is an automatic mode, establishing a final semantic vector map according to the initial semantic vector map; if the current map building mode is a manual mode, building a final semantic vector map according to a manual labeled map; and if the current map building mode is a mixed mode, building a final semantic vector map according to the initial semantic vector map and the artificial labeling map. Any mapping mode can be flexibly selected by a person skilled in the art according to actual requirements. In some preferred embodiments, the mapping mode may be selected according to the level of the perceptual capability of the first and second semantic vector perceptual models. Specifically, when the perception capability of the model is low, an artificial mode can be adopted for establishing the graph; when the perception capability of the model is improved to a certain extent but the capability requirement of the automatic mode is not met, a mixed mode can be adopted for establishing the graph; when the perception capability of the model is further improved and meets the capability requirement of the automatic mode, the automatic mode can be selected for mapping. Further, in order to finely adjust the participation degree of the artificial drawing, under the condition that the mixed mode is determined to be adopted, the mixed mode can be divided into an artificial cooperation mode and an artificial bottom-finding mode again according to the perception capability of the model. When the perception capability of the model is low, an artificial cooperation mode is adopted, in the mode, an initial semantic vector map can be established in an automatic mode aiming at most map elements, an artificial annotation map can be established in an artificial mode aiming at a small part of map elements, and then a final semantic vector map is obtained according to the initial semantic vector map and the artificial annotation map; and when the perception capability of the model is higher, an artificial bottom-holding mode is adopted, in the mode, an initial semantic vector map can be established by adopting an automatic mode aiming at most map elements, an artificial labeling map can be established by adopting an artificial mode aiming at the rest extremely few map elements, and then a final semantic vector map is obtained according to the initial semantic vector map and the artificial labeling map.

The above is the description of step S104.

In the embodiment of the construction method of the semantic vector map provided by the invention, after the semantic vector map is established by the method, the accuracy of the map is improved in order to avoid errors of the semantic vector map, and the semantic vector map can be subjected to quality inspection to obtain the semantic vector map qualified in quality inspection, so that the quality of the map is ensured.

Similar to the selection of the mapping mode in the foregoing method embodiment, in the embodiment of the present invention, different quality inspection methods may be selected to perform quality inspection on the semantic vector map according to the level of the sensing capability of the first and second semantic vector sensing models. Specifically, when the perception capability of the model is low, a manual quality inspection mode can be adopted to respectively perform quality inspection on semantic vector information of each map element on the semantic vector map; when the perception capability of the model is improved to a certain extent but the capability requirement of an automatic mode in the mapping mode is not met, manual work and a semantic vector perception model can be adopted for carrying out cross inspection on most map elements; for the rest of map elements, manual quality inspection can be adopted; when the perception capability of the model is further improved and the capability requirement of the automatic mode is met, the map elements can be sampled and inspected manually because the accuracy of the map is high.

Further, in order to improve the perception capability of the first and second semantic vector perception models, in the embodiment of the method for constructing the semantic vector map provided by the invention, model training data can be obtained according to the semantic vector map qualified in quality inspection, and model training is performed on the semantic vector perception model according to the model training data so as to optimize the model. By the method, the perception capability of the model can be gradually improved in the process of establishing different semantic vector maps by utilizing the first semantic vector perception model and the second semantic vector perception model, and finally the capability requirement of the automatic mode in the mapping mode is met.

Specifically, referring to fig. 8, after the semantic vector map is built and quality-checked in the steps S101 to S104 to obtain the semantic vector map with qualified quality-check, the first and second semantic vector perception models may be optimized in the following steps S106 to S107.

Step S106: and respectively acquiring model training data of the first semantic vector perception model and the second semantic vector perception model according to the semantic vector map qualified in quality inspection.

Specifically, the semantic vector map is obtained from a point cloud map, and the point cloud map is obtained by splicing a plurality of sensor data frames. Therefore, the sensor data frame on the semantic vector map can be used as a training sample, the semantic vector information contained in the sensor data frame can be used as a sample label, and the training sample and the sample label thereof can be used as model training data.

Step S107: model training is carried out on the first semantic vector perception model and the second semantic vector perception model respectively by adopting model training data so as to optimize the first semantic vector perception model and the second semantic vector perception model.

In the embodiment of the present invention, a conventional model training method in the technical field of machine learning may be adopted to perform model training on the first semantic vector perception model and the second semantic vector perception model, which is not specifically limited in the embodiment of the present invention. For example, taking the first semantic vector perceptual model as an example, the model training data may be input to the first semantic vector perceptual model, the loss value of the model is calculated through forward propagation, the parameter gradient of the model parameter is calculated according to the loss value, and the model parameter is updated according to the parameter gradient through backward propagation until the first semantic vector perceptual model satisfies the convergence condition, and then the training is stopped.

Based on the steps S106 to S107, the sensing capabilities of the first and second semantic vector sensing models can be continuously improved, and the mapping accuracy of the semantic vector map can be further improved. Meanwhile, the established semantic vector map is reused to obtain model training data, and extra manual resources are not occupied, so that the obtaining cost of the model training data can be reduced.

The above step S106 will be further explained.

As can be seen from step S104 in the foregoing method embodiment, the map product specification of the semantic vector map is a specific description manner of the physical world according to the design requirements of the mapping project or the map product demand side, which cannot be applied to sensors with different identification characteristics. Therefore, when the model training data of the first semantic vector perception model and the second semantic vector perception model are obtained according to the semantic vector map qualified in quality inspection, the map master library training data meeting the preset map master library specification can be generated according to the semantic vector map qualified in quality inspection, and then the model training data of the first semantic vector perception model and the model training data of the second semantic vector perception model are obtained according to the map master library training data. The preset map mother library specification is a basic description mode of the physical world aiming at different sensor identification characteristics, so that the map mother library training data can be suitable for the sensors with different identification characteristics.

The following further describes a method for obtaining model training data of the first semantic vector perceptual model and the second semantic vector perceptual model, respectively.

1. Method for obtaining model training data of first semantic vector perception model

In the embodiment of the invention, the data coordinate system of the training data of the map mother library is a global coordinate system. Because the coordinate system of the sensor data frame acquired by the sensor is a sensor coordinate system and is inconsistent with the data coordinate system of the map database training data, the map database training data needs to be converted from the global coordinate system to the sensor coordinate system of the sensor to acquire the map database training data in the sensor coordinate system, and then the model training data of the first semantic vector perception model is acquired according to the map database training data in the sensor coordinate system.

In a preferred embodiment provided by the present invention, the map library training data in the sensor coordinate system may be obtained by converting the map library training data into the sensor coordinate system using the pose of the vehicle (the pose of the vehicle converted from the vehicle body coordinate system to the global coordinate system). Specifically, in the present embodiment, the map mother library training data in the sensor coordinate system may be acquired through the following steps 11 to 13.

Step 11: and acquiring the acquisition time of the sensor data frame corresponding to the training data of each map mother library. Specifically, the acquisition time may be obtained according to a timestamp of each sensor data frame.

Step 12: and acquiring the pose of the vehicle at the acquisition time of each sensor data frame, wherein the pose is the pose converted from the vehicle body coordinate system to the global coordinate system.

In practical application, a semantic vector map may be built by using sensor data frames acquired by a plurality of sensors, for example, a semantic vector map may be built by using point cloud frames acquired by two laser radars with different field angles. As the data frame acquisition frequencies of different sensors are possibly different, in order to ensure that the pose of the vehicle at the acquisition time of each sensor data frame can be obtained, a dense track of the vehicle can be formed according to the pose of the vehicle at each IMU data acquisition time, and then the pose of the vehicle at the acquisition time of each sensor data frame is respectively obtained according to the dense track of the vehicle.

Referring again to fig. 2, in the process of building a point cloud map from the sensor data frames, the IMU may be utilized to obtain dense trajectories of the vehicle. Specifically, after the key frame pose is obtained, the key frame acquisition time can be obtained according to the time stamp of the key frame, and then the key frame pose at each key frame acquisition time is obtained. Furthermore, each data acquisition time of the IMU, namely the IMU acquisition time, can be acquired according to the data acquisition frequency of the IMU. And then, based on the IMU acquisition time and according to the key frame acquisition time and the corresponding key frame pose, performing time interpolation calculation to obtain a pose calculation result of each IMU acquisition time, and taking the pose calculation result as the pose of the vehicle at the IMU acquisition time. And finally, forming a dense track of the vehicle according to the poses of the vehicle at the time of acquiring the IMUs.

Step 13: and respectively converting the training data of each map database from the global coordinate system to the vehicle body coordinate system and then from the vehicle body coordinate system to the sensor coordinate system according to the pose.

After the map master library training data are converted into the vehicle body coordinate system, the map master library training data in the vehicle body coordinate system can be obtained. And then, according to the pose converted from the vehicle body coordinate system to the sensor coordinate system, converting the map database training data in the vehicle body coordinate system to the sensor coordinate system again to obtain the map database training data in the sensor coordinate system.

Referring to fig. 9, fig. 9 illustrates a conversion process for converting the map corpus training data from the global coordinate system to the sensor coordinate system of the sensor in one embodiment, wherein,

a global coordinate system is represented, and,

a coordinate system of the vehicle body is represented,

representing a cartesian coordinate system with the sensor as the origin,

which represents the coordinate system of the sensor and,

is a map database training data in a global coordinate system, the coordinates of the map database training data in the global coordinate system are

. Suppose, a master library of maps trains data

The acquisition time of the corresponding sensor data frame is

In a

The position and posture of the vehicle at any time are

Vehicle body coordinateThe pose of the system is converted into a Cartesian coordinate system with the sensor as the origin

The attitude model of the sensor is

。

As shown in FIG. 9, the training data of the map master library is obtained

Coordinates in a global coordinate system

Then, firstly, the

From a global coordinate system

Conversion to a vehicle body coordinate system

(origin of coordinate system is

The X-axis and the Y-axis are respectively

And

) Then will be

From the vehicle body coordinate system

Conversion to Cartesian coordinate system with sensor as origin

(origin of coordinates is

The X-axis and the Y-axis are respectively

And

) And finally will

From a Cartesian coordinate system

Conversion to sensor coordinate system

. Wherein the content of the first and second substances,

the position is indicated by a position indication,

indicating an angle. The above process can be expressed as

。

2. Method for obtaining model training data of second semantic vector perception model

In the embodiment of the invention, the data coordinate system of the training data of the map master library is a global coordinate system, and the map coordinate system of the point cloud map is also the global coordinate system, so that the training data of the map master library does not need to be converted by the coordinate system, and the model training data of the second semantic vector perception model can be directly obtained according to the training data of the map master library.

It should be noted that, although the foregoing embodiments describe each step in a specific sequence, those skilled in the art will understand that, in order to achieve the effect of the present invention, different steps do not necessarily need to be executed in such a sequence, and they may be executed simultaneously (in parallel) or in other sequences, and these changes are all within the protection scope of the present invention.

It will be understood by those skilled in the art that all or part of the flow of the method according to the above-described embodiment may be implemented by a computer program, which may be stored in a computer-readable storage medium and used to implement the steps of the above-described embodiments of the method when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying said computer program code, media, usb disk, removable hard disk, magnetic diskette, optical disk, computer memory, read-only memory, random access memory, electrical carrier wave signals, telecommunication signals, software distribution media, etc. It should be noted that the computer readable storage medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable storage media that does not include electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.

Furthermore, the invention also provides a device for constructing the semantic vector map.

Referring to fig. 10, fig. 10 is a main structural block diagram of a semantic vector map building apparatus according to an embodiment of the present invention. As shown in fig. 10, the semantic vector map constructing device in the embodiment of the present invention mainly includes a geometric map constructing module, a sensing module, a fusion vector map constructing module, a specification converting module, an automatic map constructing module, a manual map constructing module, a map quality inspection module, and a space-time back projection module. In some embodiments, one or more of a geometry mapping module, a perception module, a fusion vector mapping module, a specification conversion module, an automated mapping module, a manual mapping module, a map quality inspection module, and a spatiotemporal back-projection module may be combined together into one module.

The geometric mapping module may be configured to build a point cloud map from frames of sensor data acquired by sensors on the vehicle, and to form dense tracks of the vehicle from the pose of the vehicle at each IMU data acquisition time; the perception module may be configured to perform semantic vector perception of the map element on the sensor data frame through a first semantic vector perception model to obtain first semantic vector information of the map element, and perform semantic vector perception of the map element on the point cloud map through a second semantic vector perception model to obtain second semantic vector information of the map element; the fusion vector mapping module may be configured to build a semantic vector map based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information. The specification conversion module may be configured to convert the map master database data into map product data conforming to the specification of the map product after generating the map master database data conforming to the specification of the preset map master database based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information, and convert the map product training data into the map master database training data conforming to the specification of the preset map master database after obtaining the map product training data conforming to the specification of the map product according to the quality inspection-qualified semantic vector map. The automated mapping module may be configured to build a semantic vector map based on the map product data. The manual map building module can be configured to perform map element semantic vector marking on the point cloud map by adopting a manual marking mode and acquire a manual marking map obtained by adopting the manual marking mode; the map quality inspection module is configured to perform quality inspection on the semantic vector map so as to obtain the semantic vector map qualified in quality inspection; the spatio-temporal back projection module is configured to convert the map master training data from the global coordinate system to a sensor coordinate system of the sensor to obtain the map master training data in the sensor coordinate system.

When the map product is delivered through the device, model training data of the first semantic vector perception model and the second semantic vector perception model can be continuously generated without occupying extra manual operation resources, the perception capability of the first semantic vector perception model and the perception capability of the second semantic vector perception model are improved, the generation efficiency of the manual map building module can be further improved, the delivered map product and the obtained model training data are continuously increased in unit time, and the whole device is rapidly evolved from relying on manual map building to high automation.

The above semantic vector map construction apparatus is used to implement the semantic vector map construction method embodiments shown in fig. 1 to 9, and the technical principles, solved technical problems, and generated technical effects of the two are similar, and it can be clearly understood by those skilled in the art that, for convenience and simplicity of description, the specific working process and related description of the semantic vector map construction apparatus may refer to the contents described in the semantic vector map construction method embodiments, and are not repeated here.

Furthermore, the invention also provides computer equipment.

Referring to FIG. 11, FIG. 11 is a schematic diagram of the principal structure of one embodiment of a computer apparatus according to the present invention. As shown in fig. 11, the computer device in the embodiment of the present invention mainly includes a storage device and a processor, the storage device may be configured to store a program for executing the semantic vector map construction method of the above-described method embodiment, and the processor may be configured to execute a program in the storage device, the program including, but not limited to, a program for executing the semantic vector map construction method of the above-described method embodiment. For convenience of explanation, only the parts related to the embodiments of the present invention are shown, and specific technical details are not disclosed.

The computer device in the embodiment of the present invention may be a control apparatus device formed including various electronic devices. In some possible implementations, a computer device may include multiple storage devices and multiple processors. And the program executing the method for constructing the semantic vector map according to the above method embodiment may be divided into multiple segments of subroutines, each of which may be loaded and executed by a processor to perform different steps of the method for constructing the semantic vector map according to the above method embodiment. Specifically, each piece of sub program may be stored in a different storage device, and each processor may be configured to execute a program in one or more storage devices to implement the method for constructing a semantic vector map of the above method embodiment together, that is, each processor executes different steps of the method for constructing a semantic vector map of the above method embodiment to implement the method for constructing a semantic vector map of the above method embodiment together.

The multiple processors may be processors disposed on the same device, for example, the computer device may be a high-performance device composed of multiple processors, and the multiple processors may be processors configured on the high-performance device. In addition, the multiple processors may also be processors disposed on different devices, for example, the computer device may be a server cluster, and the multiple processors may be processors on different servers in the server cluster.

Further, the invention also provides a computer readable storage medium.

In an embodiment of a computer-readable storage medium according to the present invention, the computer-readable storage medium may be configured to store a program for executing the method for constructing a semantic vector map of the above-described method embodiment, and the program may be loaded and executed by a processor to implement the above-described method for constructing a semantic vector map. For convenience of explanation, only the parts related to the embodiments of the present invention are shown, and details of the specific techniques are not disclosed. The computer readable storage medium may be a storage device formed by including various electronic devices, and optionally, the computer readable storage medium is a non-transitory computer readable storage medium in the embodiment of the present invention.

So far, the technical solution of the present invention has been described in conjunction with one embodiment shown in the accompanying drawings, but it is easily understood by those skilled in the art that the scope of the present invention is obviously not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can fall into the protection scope of the invention.

Claims

1. A semantic vector map construction method is characterized by comprising the following steps:

performing semantic vector perception of map elements on the sensor data frame through a first semantic vector perception model to acquire first semantic vector information of the map elements;

2. The method for constructing a semantic vector map according to claim 1, wherein the step of establishing a semantic vector map based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information specifically comprises:

generating map mother database data which accords with the specification of a preset map mother database based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information;

and establishing a semantic vector map according to the map product data.

3. The method for constructing a semantic vector map according to claim 1, wherein after the step of establishing a semantic vector map based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information, the method further comprises:

4. The method for constructing the semantic vector map according to claim 3, wherein after the step of performing quality inspection on the semantic vector map to obtain the semantic vector map qualified by quality inspection, the method further comprises:

5. The method for constructing the semantic vector map according to claim 4, wherein the step of respectively obtaining model training data of the first semantic vector perception model and the second semantic vector perception model according to the semantic vector map qualified in quality inspection specifically comprises:

generating map mother base training data which accord with the specification of a preset map mother base according to the semantic vector map qualified by quality inspection;

6. The method according to claim 5, wherein the data coordinate system of the training data of the map corpus is a global coordinate system, and the step of obtaining the model training data of the first semantic vector perceptual model and the second semantic vector perceptual model respectively according to the training data of the map corpus specifically comprises obtaining the model training data of the first semantic vector perceptual model in the following manner:

converting the map database training data from the global coordinate system to a sensor coordinate system of the sensor to obtain the map database training data in the sensor coordinate system;

and obtaining model training data of the first semantic vector perception model according to the map mother base training data in the sensor coordinate system.

7. The method for constructing the semantic vector map according to claim 6, wherein the step of converting the map mother library training data from the global coordinate system to the sensor coordinate system of the sensor to obtain the map mother library training data in the sensor coordinate system comprises:

and respectively converting the training data of each map database from a global coordinate system to a vehicle body coordinate system and then from the vehicle body coordinate system to a sensor coordinate system according to the pose.

8. The method for constructing the semantic vector map according to claim 7, wherein the step of acquiring the pose of the vehicle at the acquisition time of each sensor data frame specifically comprises:

9. The method for constructing the semantic vector map according to claim 1, wherein the step of performing semantic vector sensing of the map element on the sensor data frame through the first semantic vector sensing model to obtain the first semantic vector information of the map element specifically comprises:

10. The method for constructing the semantic vector map according to claim 1, wherein the step of performing semantic vector sensing of the map element on the sensor data frame through the first semantic vector sensing model to obtain the first semantic vector information of the map element further comprises:

11. The method for constructing a semantic vector map according to claim 1, wherein the step of establishing a semantic vector map based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information specifically comprises:

12. The method for constructing a semantic vector map according to claim 1, wherein the step of establishing a semantic vector map based on the point cloud map and according to the first semantic vector information and/or the second semantic vector information further comprises:

obtaining a manual marking map obtained by marking map element semantic vectors of the point cloud map in a manual marking mode;

13. The method for constructing a semantic vector map according to claim 12, wherein the step of establishing a final semantic vector map according to the initial semantic vector map and/or the artificial labeling map specifically comprises:

if the current mapping mode is an automatic mode, establishing a final semantic vector map according to the initial semantic vector map;

if the current map building mode is a manual mode, building a final semantic vector map according to the manual labeled map;

14. A computer device comprising a processor and a storage means adapted to store a plurality of program codes, characterized in that said program codes are adapted to be loaded and run by said processor to perform the method of construction of a semantic vector map according to any one of claims 1 to 13.

15. A computer readable storage medium having stored therein a plurality of program codes, characterized in that said program codes are adapted to be loaded and run by a processor to perform the method of construction of a semantic vector map according to any one of claims 1 to 13.