WO2022041706A1

WO2022041706A1 - Positioning method, positioning system, and vehicle

Info

Publication number: WO2022041706A1
Application number: PCT/CN2021/082388
Authority: WO
Inventors: 薛常亮; 俞彦辉; 王化友; 周彦兴; 温丰
Original assignee: 华为技术有限公司
Priority date: 2020-08-28
Filing date: 2021-03-23
Publication date: 2022-03-03
Also published as: CN114200481A

Abstract

A positioning method, a positioning system, and a vehicle. The positioning method comprises: generating N sampling points C₁-C_N around an initial posture; respectively determining, on the basis of matching of laser features, first weights of the current estimation postures corresponding to the sampling points; determining, on the basis of matching of visual semantic information and pavement semantic information, second weights of the current estimation postures corresponding to the sampling points; and then performing weighted averaging on the current estimation postures according to the first weights and the second weights to obtain an actual posture of the vehicle. Accordingly, a laser positioning technique based on a laser feature and a visual positioning technique based on a feature are fused, and positioning efficiency is improved. Moreover, by coding and storing the laser feature and the visual semantic information in the same positioning map, fusion of map data is achieved, and data overhead and calculation overhead generated in a positioning process are reduced. The positioning system corresponds to the positioning method. The vehicle comprises the positioning system.

Description

A positioning method, positioning system and vehicle

This application claims the priority of the Chinese patent application with the application number 202010884916.3 and the invention titled "A Positioning Method, Positioning System and Vehicle", which was submitted to the State Intellectual Property Office on August 28, 2020, the entire contents of which are incorporated by reference in this application.

technical field

The present application relates to the technical field of automatic driving, and in particular, to a positioning method, a positioning system and a vehicle.

Background technique

Autonomous driving needs to address three core questions of driving: where? (vehicle location); where to go? (determine the destination); how to get there? (route plan). Among them, positioning technology is mainly used to solve the problem of "where?", and is one of the key technologies that are essential to realize autonomous driving.

Signal-based positioning technology uses satellite signals or 5G signals to achieve vehicle positioning, so it has the ability to global positioning and is currently the most widely used positioning technology. However, since the GNSS satellite signal is easily affected by the occlusion of tall buildings and mountains, the signal-based positioning technology cannot provide accurate positioning when the vehicle is driving in cities, tunnels and other road conditions, so it cannot meet the needs of full-scene autonomous driving. . To solve this problem, the current research direction in the field of autonomous driving is to integrate signal-based positioning technology, dead reckoning-based positioning technology, and feature matching-based positioning technology to make up for the shortcomings of signal-based positioning technology.

At present, for the positioning technology based on environmental feature matching, the laser positioning technology based on laser feature matching supplements or replaces the signal-based positioning technology, or the visual positioning technology based on visual feature matching supplements or replaces the signal-based positioning technology. An independent technical route, the map data that laser positioning and visual positioning rely on are stored separately, and the algorithms run independently, resulting in excessive data storage overhead and computational overhead, high requirements for the vehicle's control unit ECU and other hardware systems, and low efficiency.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a positioning method, a positioning system, and a vehicle, which can integrate a laser positioning technology based on laser feature matching and a visual positioning technology based on visual feature matching, so as to improve positioning efficiency and reduce the occurrence of data and computational overhead.

In a first aspect, an embodiment of the present application provides a positioning method, the method includes: determining an initial pose of a vehicle, and generating N sampling points C ₁ to C _N around the initial pose, where N is a positive integer; The current predicted pose P extracts the first laser feature and at least one visual semantic information from the positioning map; for any sampling point C _n , n is a positive integer, n≤N, according to its corresponding current estimated pose P _n , the extracted The first laser feature of the self-localization map is matched with the second laser feature to determine the first weight of the current estimated pose P _n , and the second laser feature is extracted from the point cloud data collected by the lidar; and, for At any sampling point C _n , according to the current estimated pose P _n , at least one visual semantic information is matched with at least one pavement semantic information to determine the second weight of the current estimated pose P _n , and the at least one pavement semantic information is derived from It is extracted from the image data collected by the camera; according to the current estimated pose P ₁ -PN of _N sampling points C ₁ _-CN and its first weight and second weight, the current estimated pose P ₁ - P is calculated The weighted average of _N is taken as the current pose of the vehicle; wherein, the positioning map includes multiple positioning pictures spliced with each other, the positioning pictures include color channels, the encoding of the first laser feature and the encoding and storage of visual semantic information in the color channel.

The technical solutions provided by the embodiments of the present application can respectively determine the first weight of the currently estimated pose based on the matching of laser features, and determine the current estimated pose based on the matching of visual semantic information and road semantic information (ie, matching of visual features). The second weight, and then calculate the weighted average of the current estimated pose according to the first weight and the second weight, and use the weighted average as the current pose of the vehicle, thereby realizing the laser positioning based on laser feature matching. The technology is integrated with the visual positioning technology based on visual feature matching, which improves the positioning efficiency. In addition, the technical solutions provided by the embodiments of the present application encode and store laser features and visual semantic information in the same positioning map, which realizes the fusion of map data and reduces the data overhead and computing overhead generated in the positioning process.

In an optional implementation manner, the positioning picture includes a first color channel, and the first color channel is used to store coding of visual semantic information. In this way, the method provided by the embodiment of the present application can decode the first color channel to obtain visual semantic information.

In an optional implementation, the positioning picture further includes a second color channel, where the second color channel is used to store the code of the first laser feature. In this way, the method provided by the embodiment of the present application can decode the second color channel to obtain the first laser feature.

In an optional implementation manner, the encoding of the visual semantic information includes at least one of a marker bit, a type encoding, and a brightness encoding; the marker bit is used to represent the type of the road marking, and the type encoding is used to represent the content of the road marking, Luma information encoding is used to represent the luma information of a picture. In this way, through the coding of the visual semantic information, it is possible to determine what kind of road signs the visual semantic information contains, such as white dotted lines, white solid lines, straight signs, etc., so as to facilitate matching with the road semantic information.

In an optional implementation, at least one piece of visual semantic information is extracted by the following steps: extracting a local positioning map from the positioning map according to the current predicted pose, the local positioning map includes M positioning pictures, and the M positioning pictures include the current prediction The first picture where the pose is located, and M-1 second pictures near the first picture, where M is a positive integer greater than 1; at least one visual semantic information is extracted from the local positioning map. In this way, the visual semantic information can include information such as road signs around the current predicted pose, so as to be matched with the road semantic information in the image data around the vehicle collected by the camera to determine the second weight.

In an optional implementation manner, for any sampling point C _n , according to the current estimated pose P _n , at least one piece of visual semantic information is matched with at least one piece of road semantic information, so as to determine the first position of the current estimated pose P _n Two weights, including: determining at least one valid pavement semantic information from at least one pavement semantic information, and the _number of pixels of each valid pavement semantic information is within a preset range; Projecting into the coordinate system of the local positioning map; determining the semantic relationship between at least one valid pavement semantic information and at least one visual semantic information; performing semantic matching on each pair of semantically related valid pavement semantic information and visual semantic information, according to the semantic matching As a result, the second weight is determined. In this way, some incomplete or oversized falsely detected visual semantic information can be filtered out by determining the effective pavement semantic information, so as to improve the efficiency of semantic matching and reduce the calculation amount of semantic matching.

In an optional implementation manner, determining the semantic relationship between at least one valid pavement semantic information and at least one visual semantic information includes: calculating the semantic weight of any valid pavement semantic information a _i and the arbitrary visual semantic information b _j Semantic weight; according to the difference between the semantic weight of the effective pavement semantic information a _i and the semantic weight of the visual semantic information b _j , the semantic correlation degree of the effective pavement semantic information a _i and the visual semantic information b _j is determined; when the semantic correlation When the degree is less than the preset first threshold, it is determined that the effective pavement semantic information a _i and the visual semantic information b _j have a semantic relationship. In this way, when the semantic relevance is less than the first threshold, it means that the semantic weight of the effective pavement semantic information a _i and the semantic weight of the visual semantic information b _j are relatively close, which further means that the effective pavement semantic information a _i and the visual semantic information b _j have Semantic association.

In an optional implementation manner, semantic matching is performed on each pair of semantically associated valid pavement semantic information and visual semantic information, and the second weight is determined according to the semantic matching result, including: separately calculating the valid pavement semantic information for each pair of semantic associations. The matching distance between the pavement semantic information and the visual semantic information; the weighted summation of the calculated matching distances is used to obtain the total matching distance; the second weight is determined according to the total matching distance. In this way, the second weight can reflect the closeness of the current estimated pose P _n to the real pose of the vehicle.

In an optional implementation manner, according to the current estimated poses P ₁ -PN of the _N sampling points C ₁ _-CN and their first weights and second weights, the current estimated pose P ₁ - The weighted average of P _N , and the weighted average value is taken as the current pose of the vehicle, including: using the first weight of the current estimated pose P ₁ -P _N to weight the current estimated pose P ₁ -P _N and averaging, Obtain the first weighted average; use the second weight of the current estimated pose _P ₁ -PN to weight the current estimated pose _P ₁ -PN to obtain the second weighted average; The second weighted average is weighted and averaged to obtain a weighted average, and the weighted average is used as the current pose of the vehicle. In this way, by using the first weight and the second weight to weight the current estimated pose, the fusion of the laser positioning technology based on laser feature matching and the visual positioning technology based on visual feature matching is realized.

In an optional implementation manner, the pavement semantic information includes: a pixel block containing at least one pavement mark, the number of pixels in the pixel block, and the type of pavement mark to which each pixel belongs. In this way, the type, size, etc. of the pavement markings can be determined through the pavement semantic information, so as to facilitate matching with the pavement semantic information.

In an optional implementation manner, the current predicted pose is determined by the following steps: determining the relative pose of the vehicle generated between the current time t and the first historical time _t -1 according to the odometer data; The predicted pose corresponding to the first historical time t-1 is added to the relative pose to obtain the current predicted pose.

In an optional implementation manner, when there is a preset proportion of the first weight or the second weight is lower than the second threshold, the N sampling points C ₁ -C _N are regenerated. In this way, the divergence phenomenon of sampling points with the increase of positioning times can be eliminated.

In a second aspect, an embodiment of the present application provides a positioning system, including: a GNSS/INS combination module, a control unit, a memory, a lidar, and a camera installed on a vehicle; and a GNSS/INS combination module for determining the initial position of the vehicle attitude; memory, used to store the positioning map, the positioning map includes multiple positioning pictures spliced with each other, the positioning picture includes a color channel, and the color channel stores the encoding of the first laser feature and the encoding of visual semantic information; lidar, used for collecting point cloud data, where the point cloud data includes the second laser feature; a camera for collecting image data, where the image data includes at least one pavement semantic information; a control unit for generating N sampling points C ₁ to C _N around the initial pose , N is a positive integer; the control unit is also used to extract the first laser feature and at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle; the control unit is also used for any sampling point C _n , n is A positive integer, n≤N, according to its corresponding current estimated pose P _n , the first laser feature extracted from the localization map and the second laser feature are matched to determine the first weight of the current estimated pose P _n , The second laser feature is extracted from the point cloud data collected by the lidar; the control unit is further configured to, for any sampling point C _n , associate the at least one visual semantic information with the at least one pavement semantic information according to the current estimated pose P _n . Matching is performed to determine the second weight of the current estimated pose _P _n , and at least _one road semantic information is extracted from the image data collected by the camera; the control unit is further configured to The current estimated pose _P ₁ -PN and its first weight and second weight are calculated, and the weighted average of the current estimated pose _P ₁ -PN is calculated, and the weighted average is used as the current pose of the vehicle.

In an optional implementation manner, the positioning picture includes a first color channel, and the first color channel is used to store coding of visual semantic information. In this way, the positioning system provided by the embodiment of the present application can decode the first color channel to obtain visual semantic information.

In an optional implementation, the positioning picture further includes a second color channel, where the second color channel is used to store the code of the first laser feature. In this way, the positioning system provided in this embodiment of the present application can decode the second color channel to obtain the first laser feature.

In an optional implementation manner, when the control unit is used to extract at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle: the control unit is specifically configured to extract from the positioning map according to the current predicted pose Local positioning map, the local positioning map includes M positioning pictures, the M positioning pictures include the first picture where the current predicted pose is located, and M-1 second pictures near the first picture, where M is a positive integer greater than 1; The control unit is further used for extracting at least one visual semantic information from the local localization map. In this way, the visual semantic information can include information such as road signs around the current predicted pose, so as to be matched with the road semantic information in the image data around the vehicle collected by the camera to determine the semantic relationship and the second weight.

In an optional implementation manner, when the control unit is configured to match at least one visual semantic information with at least one pavement semantic information according to the current estimated pose P _n for any sampling point C _n to determine the current estimated position When the second weight of the pose P _n is: the control unit, specifically configured to determine at least one valid road semantic information from the at least one road semantic information, and the number of pixels of each valid road semantic information is within a preset range; the control unit, also is used for projecting at least one valid pavement semantic information into the coordinate system of the local positioning map according to the current estimated pose P _n ; the control unit is further adapted to determine the semantic association relationship between at least one valid pavement semantic information and at least one visual semantic information; The control unit is further configured to perform semantic matching on each pair of semantically associated valid pavement semantic information and visual semantic information, and determine the second weight according to the semantic matching result. In this way, some incomplete or oversized falsely detected visual semantic information can be filtered out by determining the effective pavement semantic information, so as to improve the efficiency of semantic matching and reduce the calculation amount of semantic matching.

In an optional implementation manner, when the control unit is used to determine the semantic relationship between at least one valid pavement semantic information and at least one visual semantic information: the control unit is specifically adapted to calculate the semantics of any valid pavement semantic information a _i weight and the semantic weight of any visual semantic information b _j ; the control unit is also used to determine the effective pavement semantic information according to the difference between the semantic weight of the effective pavement semantic information a _i and the semantic weight of the visual semantic information b _j The semantic correlation degree of a _i and the visual semantic information b _j ; the control unit is further configured to determine that the effective pavement semantic information a _i and the visual semantic information b _j have semantic correlation when the semantic correlation degree is less than the preset first threshold. In this way, when the semantic relevance is less than the first threshold, it means that the semantic weight of the effective pavement semantic information a _i and the semantic weight of the visual semantic information b _j are relatively close, which further means that the effective pavement semantic information a _i and the visual semantic information b _j have Semantic association.

In an optional implementation manner, when the control unit is used to perform semantic matching on each pair of semantically associated valid pavement semantic information and visual semantic information, and determine the second weight according to the semantic matching result: the control unit, specifically using It is used to calculate the matching distance of each pair of semantically associated effective pavement semantic information and visual semantic information respectively; the control unit is also used to calculate the weighted sum of each matching distance to obtain the total matching distance; the control unit is also used to The total matching distance determines the second weight. In this way, the second weight can reflect the closeness of the current predicted pose P _n to the actual pose of the vehicle.

In an optional implementation manner, when the control unit is used to calculate the current estimated poses _P 1 -PN and their first weights and second weights according to the current estimated poses P ₁ -PN of the _N sampling points C ₁ -CN The weighted average of the poses P ₁ - P _N , when the weighted average is used as the current pose of the vehicle: the control unit is specifically configured to use the first weight of the current estimated pose _P ₁ -PN to determine the current estimated pose. The poses P ₁ -PN are weighted and averaged to obtain a first weighted average value; the control unit is further configured to use the second weight of the current estimated pose _P ₁ _-PN to weight the current estimated pose _P ₁ -PN averaging to obtain a second weighted average value; the control unit is further configured to perform a weighted average of the first weighted average value and the second weighted average value to obtain a weighted average value, and use the weighted average value as the current pose of the vehicle. In this way, by using the first weight and the second weight to weight the current estimated pose, the fusion of the laser positioning technology based on laser feature matching and the visual positioning technology based on visual feature matching is realized.

In an optional implementation manner, the positioning system further includes an odometer; the control unit is further configured to determine the relative pose of the vehicle between the current time t and the first historical time t-1 according to the odometer data; control The unit is also used to add the predicted pose corresponding to the sampling point C _n at the first historical time t-1 and the relative pose to obtain the current predicted pose.

In an optional implementation manner, the control unit is further configured to regenerate the N sampling points C ₁ -C _N when there is a preset ratio of the first weight or the second weight is lower than the second threshold. In this way, the divergence phenomenon of sampling points with the increase of positioning times can be eliminated.

In a third aspect, an embodiment of the present application provides a vehicle, and the vehicle includes the positioning system provided by the second aspect of the embodiment of the present application and various implementations thereof.

In a fourth aspect, embodiments of the present application further provide a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium runs on a computer, the computer executes the methods of the above aspects and their respective implementations.

In a fifth aspect, the embodiments of the present application further provide a computer program product including instructions, which, when run on a computer, enables the computer to execute the methods of the above aspects and their respective implementations.

In a sixth aspect, an embodiment of the present application further provides a chip system, where the chip system includes a processor for supporting the above-mentioned apparatus or system to implement the functions involved in the above-mentioned aspects, for example, generating or processing the functions involved in the above-mentioned method. information.

Description of drawings

Fig. 1 is the schematic diagram of point cloud type positioning map;

Fig. 2 is the schematic diagram of picture type positioning map;

Figure 3 is a schematic diagram of the amount of pixels of a sparse feature map, a semi-dense map and a dense map;

4 is a configuration diagram of a positioning system module of an autonomous vehicle;

5 is a schematic diagram of a laser map;

Fig. 6 is a coding format diagram of visual semantic information provided by an embodiment of the present application;

7 is a schematic diagram of 8-bit encoding of visual semantic information corresponding to pixels in different regions on the positioning map;

8 is a hardware framework diagram of a positioning system for implementing a positioning method provided by an embodiment of the present application;

9 is a flowchart of a positioning method provided by an embodiment of the present application;

10 is a block diagram of data flow involved in the positioning method provided by the embodiment of the present application;

Figure 11 exemplarily provides a scheme for generating sampling points;

Figure 12 is a schematic diagram of an odometer coordinate system;

13 is a schematic diagram of a process for determining a first weight provided by an embodiment of the present application;

14 is a schematic diagram of a process for determining a second weight provided by an embodiment of the present application;

15 is a flowchart of semantic matching provided by an embodiment of the present application;

FIG. 16 is a flowchart of step S201 of the positioning method provided by the embodiment of the present application;

17 is a schematic diagram of calculating the pixel area of road semantic information provided by an embodiment of the present application;

FIG. 18 is a flowchart of step S203 of the positioning method provided by the embodiment of the present application;

FIG. 19 is a flowchart of step S204 of the positioning method provided by the embodiment of the present application;

20 is a schematic diagram of a matching distance provided by an embodiment of the present application;

21 is a flowchart of step S502 of the positioning method provided by the embodiment of the present application;

FIG. 22 is a flowchart of step S105 of the positioning method provided by the embodiment of the present application;

FIG. 23 is a block diagram of software modules of a positioning system provided by an embodiment of the present application.

detailed description

Autonomous vehicles (autonomous vehicles or self-driving automobiles), also known as unmanned driving, computer driving, etc. Autonomous driving can sense its surrounding environment with technologies such as sensors, global navigation satellite system (GNSS) and machine vision, and determine its own position, plan navigation routes, update map information, and avoid obstacles based on the sensing data. The ultimate goal is to drive the vehicle autonomously with little or no human activity.

Generally speaking, autonomous driving needs to solve three core problems of driving: where? (vehicle location); where to go? (determine the destination); how to get there? (route plan). Among them, positioning technology is mainly used to solve the problem of "where?", and is one of the key technologies that are essential to realize autonomous driving.

At present, depending on the sensors they rely on, the positioning technologies for autonomous driving mainly include the following three types:

1. Signal-based positioning technology.

This technology is mainly based on satellite signals or 5G signals to achieve vehicle positioning. The current mainstream solution is to install a global navigation satellite system (GNSS) receiver in the vehicle to receive multiple GNSS satellite signals, and use the received satellite signals to calculate the global position of the vehicle in the space environment, and It is also possible to use GNSS ground stations to cooperate with GNSS satellites to improve positioning accuracy. Common GNSS systems such as: Beidou navigation satellite system (BDS), global positioning system (global positioning system, GPS) and so on.

2. Positioning technology based on dead reckoning.

The technology requires the vehicle to be equipped with sensors such as an inertial measurement unit (IMU) and wheel speedometers. Among them, the IMU can measure the angular velocity and acceleration of the vehicle, and the wheel speedometer can measure the rotational speed of the wheels. Based on the sensor data, after the initial position of the vehicle is determined, the current pose (position and attitude) of the vehicle can be estimated according to the dynamic equation of the vehicle.

3. Positioning technology based on environmental feature matching.

At present, this technology mainly includes two forms: laser positioning and visual positioning. Based on the laser sensor and the visual sensor, the environmental information around the vehicle is obtained in real time, and the obtained environmental information is processed and matched with the pre-stored positioning map to determine the vehicle's location. pose. It is understandable that the implementation of this technology needs to build a positioning map in advance, and the positioning map also has different construction methods according to the different sensors used.

When using laser positioning, the positioning map is mainly cloud-like and picture-like.

①Point cloud type: Use laser sensors to collect point cloud data, and then filter the point cloud data to remove noise. Finally, the processed point cloud data are spliced and superimposed together to form a point cloud map as shown in Figure 1.

②Picture format: Perform grid processing on the point cloud data spliced in ①, and encode each grid as a pixel, so as to convert the stitched point cloud data into the positioning of the picture format as shown in Figure 2 map. The laser map in the picture format can be a single-channel grayscale picture or a three-channel color picture. The image-based positioning map occupies a small storage space, which can solve the problem of high storage resource overhead of the point cloud map.

When visual positioning is used, according to the way and number of pixels selected from the original image collected by the visual sensor during construction, the positioning map (visual map) is shown in Figure 3, which can include sparse feature map, semi-dense map and dense map three At present, it is mainly saved in the form of point cloud.

① Sparse feature map: The pixels corresponding to the feature points in the original image are saved to the map, so the number of pixels is the least.

②Semi-dense map: Save some pixels in the original image, such as pixels with gradients, into the map, so the number of pixels is medium.

③Dense map: Save all the pixels in the original image to the map, so the number of pixels is the largest.

Signal-based positioning technology is currently the most widely used positioning technology due to its global positioning capability. However, since the GNSS satellite signal is easily affected by the occlusion of tall buildings and mountains, the signal-based positioning technology cannot provide accurate positioning when the vehicle is driving in cities, tunnels and other road conditions, so it cannot meet the needs of full-scene autonomous driving. . To solve this problem, the current research direction in the field of autonomous driving is to integrate signal-based positioning technology, dead reckoning-based positioning technology, and feature matching-based positioning technology to make up for the shortcomings of signal-based positioning technology.

At present, for the positioning technology based on environmental feature matching, the laser positioning technology using laser feature matching supplements or replaces the signal-based positioning technology, and the visual positioning technology using visual feature matching supplements or replaces the signal-based positioning technology. There are currently two independent technical routes for fusion positioning. Since the technical routes are independent of each other, the map data of laser positioning and visual positioning are stored separately, and the algorithms are independent of each other, resulting in excessive data storage and calculation overhead, high requirements for the vehicle's control unit ECU and other hardware systems, and low efficiency.

In order to solve the problems existing in the prior art, the embodiments of the present application provide a positioning method.

The technical solutions of the embodiments of the present application can be applied to various vehicles using automatic driving technology or positioning technology, including but not limited to various vehicles: such as vehicles (cars), ships, trains, subways, airplanes, etc., and Various robots, such as service robots, transport robots, automated guided vehicles (AGVs), unmanned ground vehicles (UGVs), etc., as well as various construction machinery, such as tunnel boring machines.

The hardware environment implemented by the technical solutions of the embodiments of the present application will be described below by taking a vehicle as an example.

As shown in FIG. 4 , the vehicle is configured with the following modules: a lidar LiDAR 110 , a camera 120 , a GNSS/inertial navigation system (INS) combined module 130 , and a control unit 140 . in:

The lidar 110 is used to collect the distance information of the elements around the environment (for example: vehicles, obstacles, pedestrians, street signs, etc.) from the vehicle. The lidar 110 can perform omnidirectional scanning of 360 degrees around the environment, or can only scan environmental information within a partial range (eg, 180 degrees) in front of the front of the vehicle.

The camera 120 is used to collect image information around the vehicle. The camera 120 may have one (that is, a monocular camera) or multiple (that is, a multi-eye camera), and may capture a 360-degree panoramic image of the surroundings, or may only capture an image of a partial range in front of the front of the vehicle.

The GNSS/INS combination module 130 may include devices such as a GNSS receiver and an inertial measurement unit IMU, and is used to implement fusion positioning based on satellite signals and IMU.

The control unit 140 may be the core computing unit of the electronic system of the entire autonomous vehicle, such as: a mobile data center (mobile data center, MDC), an electronic control unit (electronic control unit, ECU), etc., for processing other various modules to generate data, and generate vehicle control information based on the processing results.

In addition, in order to realize other assisted driving functions of the vehicle, the vehicle can also be configured with the following modules:

The ultrasonic sensor 150 is used for short-distance ranging, for example, it is turned on when assisting in parking, so as to provide short-distance warning information and the like.

Millimeter-wave radar 160, used for long-distance ranging, due to the strong anti-jamming ability of millimeter-wave and the ability to penetrate fog, smoke and dust, the millimeter-wave radar can work around the clock, for example, to assist in obstacle detection in bad weather Object ranging.

It can be understood that the hardware environment illustrated by the embodiments of the present application does not constitute a specific limitation on the technical solutions of the embodiments of the present application. In other embodiments of the present application, the hardware environment implemented by the technical solutions of the embodiments of the present application may include more or less components than shown in the figure, or combine some components, or separate some components, or different Component placement. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

In the embodiments of the present application, in order to realize the fusion of laser positioning and visual positioning technologies and reduce storage overhead and computing overhead, the basic idea is: a positioning map for laser positioning (hereinafter referred to as a laser map) and a positioning map for visual positioning (hereinafter referred to as a visual positioning map) The map data in the map) is fused into a positioning map, so that when the laser positioning and visual positioning technologies are applied at the same time, the data required for these two positioning methods can be obtained from the fused positioning map, thereby reducing the amount of data storage. Overhead and computational overhead, improve positioning efficiency.

The following describes in detail how the embodiments of the present application obtain the positioning map with reference to some examples.

The basic idea of obtaining the positioning map in the embodiment of the present application is to integrate and add the visual semantic information of the road markings to the laser map, so as to obtain the positioning map. Among them, the road signs include information used to assist in determining the position and lane of the vehicle, such as: single yellow line, double yellow line, white dotted line, white solid line, straight sign, left turn sign, right turn sign, U-turn sign, etc. To reduce storage overhead, the embodiment of the present application preferably uses a laser map in a picture format. Laser maps are composed of grayscale images with single color channel or color images with multiple color channels.

In one embodiment, as shown in FIG. 5, the laser map is implemented as a three-channel color image, including, for example, an R channel (red channel), a G channel (green channel), and a B channel (blue channel) ), and some laser map feature information can also be added to each channel. As an optional implementation, the R channel can be added with laser features, such as line features, corner features, gradient features, height features, etc. of various elements in the map; the G channel can be added with the brightness information of the laser map; The B channel can add relative height information, such as the height of the pixel relative to the ground, etc.

In addition, in addition to the above-mentioned RGB channels, the laser map may also include more channels, such as an alpha channel, etc., which is not limited in this embodiment of the present application. In some other implementation manners, the laser map may also be implemented by other color modes, such as RYYB format, etc. In this case, the laser map may correspondingly include four channels.

In one embodiment, each channel in the pixel of the laser map may contain a certain number of bits of information bits, such as 8 bits (8 bits of information), 16 bits, 32 bits, and so on. Therefore, feature information such as the above-mentioned laser feature, brightness information, and relative height information can be encoded and represented in the bits of the pixel. For example, the brightness information can be represented by 1 bit encoding, for example, a bit value of 1 indicates that there is brightness information, and 0 indicates that there is no brightness information.

In this embodiment of the present application, in order to fuse the visual map and the laser map into one map, the visual semantic information usually located in the visual map is fused into the laser map. In the specific implementation, the pixels that will contain visual semantic information in the laser map can be determined according to the corresponding positions of the content of road signs and road traffic signs in the laser map, and then the visual semantic information is encoded and stored in one or some of these pixels. in the bit information of these channels.

In one embodiment, the visual semantic information may be represented in an 8-bit binary encoding format, occupying 8 bits of the channel where it is located. FIG. 6 is an example diagram of an 8-bit encoding format for visual semantic information. As shown in Figure 6, the 8-bit encoding format can be composed of at least one of three parts: marker bit, type encoding and luminance encoding, wherein the marker bit can be used to indicate the type of road marking; the type encoding is used to indicate the road marking content; brightness information coding can be used to represent the brightness information of the picture.

In an implementation manner, the flag bit occupies 1 bit as shown in FIG. 6 , for example, the first bit of the 8-bit encoding, or other bits. At this time, the flag bit can have two values, 0 and 1, representing at most two types. For example, if the visual semantic information is divided into text elements and graphic reticle elements, then 0 can represent text elements, 1 can represent graphic reticle elements, or 1 represents text elements, and 0 represents graphic markers. Line element. For example, if the visual semantic information is divided into pavement marking elements and other marking elements, then 0 may represent pavement marking elements, 1 may represent other marking elements, or 1 may represent pavement marking elements, and 0 may represent other marking elements. Logo class element.

In some other implementations, the flag bit can occupy more than 1 bit, such as 2 bits, 3 bits, etc., so that more types of visual semantic information can be represented, for example, 2 bits can represent at most 4 types, 3 bits can represent up to 8 kinds. In specific practice, those skilled in the art can determine the length of the flag bit according to actual classification requirements, and the embodiment of the present application does not specifically limit the length of the flag bit.

In an implementation manner, the type code occupies 6 bits as shown in FIG. 6 , for example, 6 consecutive bits following the flag bit, or other bits. At this time, the value range of the type code can be from 000000 to 111111, which can represent up to ²⁶ content types. Exemplarily: 000000 can represent road, 000001 can represent road signs, etc; road signs etc. In this embodiment of the present application, the type encoding is a necessary part of the 8-bit encoding of the visual semantic information.

In some other implementations, the type code can occupy more than 6 bits, such as 7 bits, so that more content types can be represented; it can also be less than 6 bits, such as 5 bits, 4 bits, etc., so that in In the case of being able to represent all the required road signs and/or road traffic signs, reduce the bit length of the type code to reduce the data overhead and allow more bit information to be reserved in the 8-bit code to represent other information. In specific practice, those skilled in the art can determine the length of the type code according to the number of road signs and/or road traffic signs to be distinguished, and the length of the type code is not specifically limited in this embodiment of the present application.

In one implementation, the luma encoding occupies 1 bit as shown in FIG. 6 , such as the last bit of the 8-bit encoding, or other bits. At this time, the flag bit can have two values of 0 and 1, 0 means no brightness information, 1 means there is brightness information. Taking the lane line as an example, its brightness is higher than that of the road surface, so its brightness information can be 1; taking the road marking as an example, it may include the white line part and the road surface part, then the brightness information of the white line part is It can be 1, and the brightness information of the road surface can be 0.

In some other implementations, the luminance information may occupy more than 1 bit, such as 2 bits, 3 bits, etc., so that the luminance can be represented in more subdivisions. In specific practice, those skilled in the art can determine the length of the brightness information according to the actual classification requirements, and the embodiments of the present application do not specifically limit the length of the brightness information.

In one embodiment, the binary encoding of the visual semantic information may only include a part of the flag bit, the type encoding and the luminance encoding, for example, only the type encoding and the luminance encoding, or only the flag bit and the type encoding, or only the type encoding. Type encoding. In addition, the binary encoding of the visual semantic information may also be other encoding formats other than 8-bit encoding formats, such as encoding greater than 8 bits, such as 16-bit encoding, or encoding less than 8 bits, such as 4-bit encoding, etc., this application implements The example does not limit this.

In one embodiment, the encoded visual semantic information can be stored in the G channel of the pixel. In this way, when the G channel of the pixel contains 8 bits of information, the 8 bits of information can sequentially include flag bits, type encoding and luma encoding.

FIG. 7 is a schematic diagram of 8-bit encoding of visual semantic information corresponding to pixels of different regions on the positioning map. As shown in Figure 6, according to the coding rules of the above example, the pixels in area ① correspond to the road surface, and the flag bit is 0, the type code is 000000, and the brightness code is 0, so the visual semantic information is 00000000; the pixels in area ② correspond to straight signs , its flag bit is 1, the type code is 000011, and the brightness code is 1, so the visual semantic information is 10000111; the pixel in area ③ corresponds to the white solid line mark, its flag bit is 1, the type code is 000010, and the brightness code is 1. So the visual semantic information is 10000101.

It can be understood that, in the embodiment of the present application, the visual semantic information originally located in the visual map is encoded and stored in the pixel channel of the laser map, so as to realize the fusion of the laser map and the visual map, and obtain a visual map feature and a laser map at the same time. The location map of the feature reduces the storage overhead of the map.

The technical solutions of the positioning method provided by the embodiments of the present application will be specifically described below.

FIG. 8 is a hardware framework diagram of a positioning system for implementing a positioning method provided by an embodiment of the present application. As shown in FIG. 8 , the positioning system may include a control unit 140 , a GNSS/INS combination module 130 , a wheel speedometer 170 , an odometer 180 , a lidar 110 , a camera 120 , a memory 190 , and the like. Among them, the GNSS/INS combination module, wheel speedometer, odometer, lidar, camera and other modules are used to collect data respectively, and send the data to the control unit for processing. data, program instructions for execution by the control unit, and data generated by the control unit during data processing.

Based on the hardware structure shown in FIG. 8 , taking the target to be positioned as a vehicle as an example, the step flow of the positioning method provided by the embodiment of the present application will be described in detail. It can be understood that, in addition to the vehicle, the method of the embodiment of the present application The positioning target can also be ships, trains and other vehicles, various robots, and construction machinery.

FIG. 9 is a flowchart of a positioning method provided by an embodiment of the present application, and FIG. 10 is a block diagram of data flow involved in the positioning method. As shown in FIG. 9 and FIG. 10 , the positioning method can be implemented through the following steps S101-S109:

Step S101 , determine the initial pose of the vehicle, and generate N sampling points C ₁ -C _N around the initial pose, where N is a positive integer.

Wherein, the initial posture may include the initial position and initial posture of the vehicle.

In the specific implementation, the control unit can obtain the data collected by the GNSS/INS combination module. Then, the control unit may determine the initial position of the vehicle according to the antenna signal of the GNSS. Generally speaking, the initial position of the vehicle may be a global position. In addition, the control unit can also determine the initial attitude of the vehicle according to information such as the angular velocity and acceleration of the vehicle measured by the inertial measurement unit IMU of the INS module. Generally speaking, the initial attitude of the vehicle can be determined from the initial heading angle, pitch angle and roll angle of the vehicle. One or more parameters of , since the heading angle is mainly used in vehicle positioning and navigation, the initial attitude of the vehicle can also only include the heading angle.

In one embodiment, after determining the initial pose of the vehicle, the control unit may further generate N sampling points C ₁ within a certain range near the vehicle and within a certain range near the heading angle of the vehicle with the initial position of the vehicle as the center ~ _CN .

Figure 11 exemplarily provides a scheme for generating sample points. As shown in FIG. 11 , the control unit can determine a circular range with a radius of R=5 meters with the initial position of the vehicle as the center, and, with the direction of the heading angle yaw (the vehicle’s forward direction) as the center, select a left and right deviation of 2° The fan-shaped range, that is, yaw±2°, and then discrete N=1000 sampling points are generated in the overlapping area of the circular range and the fan-shaped range (ie, the gray shaded area in FIG. 11 ). In some implementations, the 1000 sampling points may be generated in a uniform distribution manner, so that the distribution of the 1000 sampling points in the distribution area thereof is relatively uniform. It can be understood that after the sampling point is selected, the initial pose of the sampling point is also determined according to the initial pose of the vehicle. In other implementation manners, the 1000 sampling points may also be generated in a non-uniform manner, such as a normal distribution, which is not limited in this embodiment of the present application.

It should be supplemented that the embodiment of the present application selects a large number of sampling points around the initial pose of the vehicle, which can realize multiple sampling and multiple calculations of the vehicle pose, and improve the positioning accuracy in combination with the sampling point filtering technology. In a specific implementation, the control unit may perform steps S102 to S108 for each sampling point respectively.

Step S102, extracting the first laser feature and at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle.

It should be noted here that since the pose of the vehicle is constantly changing during the driving process, various positioning methods are required to be able to locate the vehicle in real time, so as to facilitate the automatic driving system to realize real-time path planning and navigation functions. For the purpose of real-time positioning, the control unit can periodically position the vehicle, and the positioning behavior of each period can be called a positioning frame. In the specific description, in order to distinguish the positioning frames at different times, if the current time is set as t, the positioning frame corresponding to the current time t can be called the current frame, the previous positioning frame of the current frame is called the first historical frame, and the first historical frame is called the first historical frame. The time of one historical frame is recorded as the first historical time t-1.

It can be understood that, during the driving process of the vehicle, each time the control unit locates the vehicle, a predicted pose of the vehicle will be obtained. For ease of description, in the embodiment of the present application, the obtained vehicle pose at the first historical time t-1 is referred to as the first historical pose. Then, according to the first historical pose and the odometer parameters, it can be predicted that the vehicle is at the current time. The current predicted pose P t of _t . It should be noted that, if the first historical moment t-1 is the initial moment, then the first historical pose is the initial pose of the vehicle.

Based on the above definition, this embodiment of the present application may determine the current predicted pose of the vehicle in the following manner:

Step a, obtain the initial pose of the vehicle.

Among them, as mentioned above, the initial pose may be the pose determined according to the data collected by the GNSS/INS combination module when the positioning method is initialized and executed. Step a is only used to execute when the method is initialized.

Step b: Determine the relative pose of the vehicle between the current time t and the first historical time t-1 according to the odometer data.

In an implementation manner, the embodiment of the present application uses the mileage coordinate system to obtain the relative pose.

Figure 12 is a schematic diagram of an odometer coordinate system. As shown in Figure 12, the odometer coordinate system can take the initial pose of the vehicle as the origin Odom, and take the forward direction of the vehicle body in the initial pose as the X-axis direction, so as to be perpendicular to the forward direction of the vehicle body and point to the vehicle The direction of the left side of the body is taken as the Y-axis direction.

The local pose of the vehicle in the odometer coordinate system can be calculated by the odometer according to the measurement data of the wheel speedometer and the inertial measurement unit IMU. When calculating the local pose, the odometry can adopt the following kinematic models:

S=V*Δt (1)

Among them, S represents the movement distance of the vehicle relative to the initial pose, V represents the movement speed of the vehicle, and Δt represents the movement time of the vehicle;

Represents the change value of the heading angle of the vehicle relative to the initial pose, and ω represents the angular velocity of the vehicle.

According to the above motion model (1) (2), we can get:

x ₀ =V*cos(yaw)*Δt (3)

y ₀ =V*sin(yaw)*Δt (4)

Among them, x ₀ is the X-axis coordinate value when the vehicle is moving, y ₀ is the Y-axis coordinate value when the vehicle is moving, and yaw is the heading angle of the vehicle. In the odometer coordinate system, the value of yaw is

According to the above motion model (1) (2) and formula (3) (4), the local pose expressed by the parameters of the first local coordinate system at any time when the vehicle is moving can be obtained. For example, the local pose may include ( x ₀ , y ₀ , yaw).

It should be noted here that the local pose only represents the pose of the vehicle in the odometer coordinate system, and does not represent the absolute pose of the vehicle in the space environment.

It should be added that, in addition to the odometer coordinate system, other coordinate systems can also be used in this embodiment of the present application to obtain the relative pose, such as the GNSS coordinate system, the IMU coordinate system, and the ground projection coordinate system of the rear wheel axle of the vehicle. The application embodiments do not limit this.

Based on the odometer coordinate system, the odometer can send the local pose of the vehicle at the current time t and the local pose of the first historical time t-1 to the control unit. Then, the control unit can calculate the relative pose of the vehicle between the current time t and the first historical time t-1 according to the local pose of the vehicle at the current time t and the local pose of the first historical time t-1. The specific calculation method is as formula (5):

ΔP=o _t -o _t-1 (5)

Among them, ΔP is the relative pose of the vehicle between the current time t and the first historical time t-1, o _t is the local pose of the vehicle at the current time t, and o _t-1 is the vehicle at the first historical time t. -1 for the local pose.

Step c, adding the first historical pose corresponding to the vehicle at the first historical time t-1 and the relative pose to obtain the current predicted pose P _t of the vehicle.

As the following formula (6)

P _t =P _t-1 +ΔP (6)

Among them, P _t-1 is the first historical pose corresponding to the vehicle at the first historical time t-1.

It can be understood that the current estimated pose P _n of the sampling point C _n is also estimated through the above step c, namely:

P _n =P _n(t-1) +ΔP

Among them, P _n(t-1) is the estimated pose corresponding to the sampling point C _n at the first historical time t-1.

Further, the control unit can obtain an area near the current predicted pose P _t from the positioning map according to the current predicted pose P _t of the vehicle. Extract the first laser feature from the map.

In one embodiment, the positioning map may be formed by splicing a large number of pictures of preset sizes, and each picture corresponds to a range of a specified size in the spatial environment. For example, each picture of the positioning map is a square picture of equal length and width, and each picture corresponds to a square range of 100 meters long and 100 meters wide.

In one embodiment, when the positioning map is composed of a large number of pictures, the control unit may obtain the picture where the current predicted pose P _t is located and at least one nearby picture from the positioning map as the local positioning map. For example, as shown in Fig. 13 or 14, the control unit can obtain a total of 9 pictures of 3 × 3 where the current predicted pose P _t is located and nearby, if the corresponding range of each picture is 100m × 100m, then the local positioning map It includes an area of 300m×300m near the current predicted pose P _t .

Based on the above extracted picture of the positioning map, the control unit may extract the first laser feature from the channel of the picture in which the laser feature is stored, for example, extract the first laser feature from the R channel.

In addition, the control unit may extract at least one visual semantic information from the channel storing the visual semantic information in the local location map, for example, decode the G channel data of the picture to extract at least one visual semantic information in the G channel.

Step S103 , for any sampling point C _n , where n is a positive integer, n≤N, according to its corresponding current estimated pose P _n , the first laser feature extracted from the self-positioning map is matched with the second laser feature to determine: The first weight of the currently estimated pose P _n .

The second laser feature can be extracted from the point cloud data collected by the lidar.

In specific implementation, the control unit may sample the point cloud data collected by the lidar at the current time t to obtain the second laser feature, and the sampling method may be determined according to the specific form of the positioning map. For example: when the positioning map is a sparse feature map, the control unit can perform sparse feature sampling on the point cloud data; when the positioning map is a semi-dense map, the control unit can perform semi-dense feature sampling on the point cloud data; when the positioning map is dense When mapping, the control unit can perform dense feature sampling on the point cloud data. In this way, matching of the first laser feature with the second laser feature is facilitated.

Next, the control unit projects the laser features into the coordinate system of the local positioning map according to the current estimated pose P _n . For different sampling points C _n , since their current estimated poses P _n are different, the above laser features are based on After the current estimated poses P _n of different sampling points C _n are projected, they will correspond to different coordinate distributions in the local positioning map.

Next, the control unit may match the first laser feature and the second laser feature based on the coordinate distribution of the first laser feature and the second laser in the local positioning map, and calculate the difference between the first laser feature and the second laser feature The matching distance of , and the first weight of the current estimated pose P _n is determined according to the matching distance

Among them, the matching distance represents the closeness between the actual pose of the vehicle determined based on the laser feature and the current estimated pose _Pn of the sampling point _Cn , and the higher the closeness is, the higher the first weight is.

The larger, the lower the degree of proximity, the first weight

the smaller.

In some embodiments, the matching distance may be a cosine distance or an Euler distance, and the embodiments of the present application do not limit the algorithm used to obtain the matching distance. For example: when the matching distance is a cosine distance, the value range of the matching distance can be [0, 1]. The larger the value, the difference between the actual pose of the vehicle determined based on the laser feature and the current estimated pose P _n of the sampling point C _n The lower the closeness between the two, the smaller the value, which means the higher the closeness between the actual pose of the vehicle determined based on the laser feature and the current estimated pose _Pn of the sampling point _Cn .

Step S104 , for any sampling point C _n , according to the current estimated pose P _n , match at least one visual semantic information with at least one road semantic information to determine the second weight of the current estimated pose P _n .

Wherein, the above-mentioned at least one pavement semantic information is extracted from the image data collected by the camera.

In a specific implementation, as shown in FIG. 14 , the control unit may first perform preprocessing on the image data collected by the camera, such as noise removal, cropping, grayscale processing, and the like. Next, the control unit can use the pre-trained deep neural network to perform pixel-level semantic segmentation on the preprocessed image to extract at least one pavement semantic information from the image. The pavement semantic information can be pixel-level information, and each pavement The semantic information may include: a pixel block containing at least one road marking, the number of pixels in the pixel block, the type and probability of the road marking to which each pixel belongs, and the like. The pixel block should contain at least all pixels of the road marking. In some embodiments, the pixel block may be a regular shape such as a rectangle, a circle, etc., or other shapes, preferably a regular shape to facilitate data processing. In addition, when it is guaranteed that the pixel block includes all the pixels of the road surface marking, the pixel block preferably contains as few pixels as possible that are not the road surface marking.

The deep neural network used in the embodiments of the present application may be, for example, a convolutional neural network (CNN), a long short-term memory (LSTM), a recurrent neural network (RNN), or Other neural networks can also be a combination of various neural networks. The deep neural network uses the training corpus as input in the training phase, the training corpus can be the road image collected in advance, and the road signs in the road image are marked at the pixel level; the output of the deep neural network in the training phase is the annotation result of the training corpus , such as the type of pavement markings being marked, etc. The input of the neural network in the use stage is the image collected by the camera, and the output is the pixel-level information of the road signs contained in the image. The specific method of using the deep neural network for information extraction is not the focus of the discussion in this embodiment of the present application, and due to space limitations, it will not be repeated here.

In addition, it needs to be added that the pavement semantic information actually belongs to the visual semantic information, the difference is that it is extracted from the image collected by the camera, rather than stored in the positioning map.

Based on the above-mentioned extracted pavement semantic information and visual semantic information, step S104 can be implemented by the following steps S201-S204 as shown in FIG. 15 :

Step S201, at least one valid road semantic information is determined from the at least one road semantic information, and a set of valid road semantic information is established, wherein the number of pixels of each valid road semantic information is within a preset range.

In one embodiment, as shown in FIG. 16 , step S201 can be specifically implemented by the following steps S301-S303:

Step S301, respectively calculating the pixel area of each road surface semantic information.

In a specific implementation, the pixel area of the pavement semantic information may be the number of pixels of the pixel block. Taking a rectangular pixel block as an example, assuming that the resolution size of the pixel block is W pixels×H pixels, and W and H are the number of pixels in the horizontal and vertical directions of the pixel block, respectively, then the pixel area of the pixel block S=W×H .

For example, as shown in FIG. 17 , in step S105 , the control unit acquires a plurality of pavement semantic information from the image collected by the camera, for example, including pavement semantic information L0 , pavement semantic information L1 and pavement semantic information L2 . Then, in step S301, the pixel quantities of the pixel blocks of L0, L1, and L2 can be calculated respectively, so as to obtain the pixel area of L0 as S ₀ =W0×H0, and the pixel area of L1 as S ₁ =W1×H1,L2 The pixel area is S ₂ =W2×H2.

Step S302, determining the effective road semantic information according to the pixel area.

In specific implementation, the embodiment of the present application may set a pixel area lower threshold value T ₁ and a pixel area upper threshold value T ₂ for determining the effective road semantic information, and the control unit uses the lower threshold value T ₁ and the upper threshold value T ₂ to be respectively associated with each The pixel area S (for example: S ₀ , S ₁ , S ₂ , etc.) of the road semantic information is compared. When T ₁ <S<T ₂ , the road semantic information is valid semantic information. When S<T ₁ or S> At T ₂ , the pavement semantic information is invalid semantic information. _The lower threshold T1 and the upper threshold _T2 may be preset values or dynamically generated values.

_In _one embodiment, when the lower limit threshold T1 and the upper limit threshold T2 are dynamically generated values, the control unit can count the pixel areas of all road semantic information within a period of time to obtain the distribution range of the pixel areas, and then calculate the pixel area in the pixel area. _A certain range is selected from the distribution range as the range of valid road semantic information, and then the lower threshold T1 and the upper threshold T2 are _determined .

Step S303, establishing an effective pavement semantic information set for the effective pavement semantic information.

Wherein, according to different results of the valid semantic road surface information determined in step S302, the quantity of valid road semantic information included in the valid road semantic information set is also different. For example: when it is determined in step S302 that the pavement semantic information does not contain the effective pavement semantic information set, the effective pavement semantic information set is an empty set; when it is determined in step S302 that a part of pavement semantic information is valid pavement semantic information, the effective pavement semantic information set is pavement semantic information A subset of the semantic information set, when it is determined in step S302 that all the pavement semantic information is valid pavement semantic information, the effective pavement semantic information set is the same as the pavement semantic information set.

For example, any valid pavement semantic information in the effective pavement semantic information set may be in the following form:

a _i =[M, p ₁ to p _M , Ka _i ]

Among them, a _i represents the ith valid pavement semantic information in the effective pavement semantic information set, M represents the number of pixels contained in a _i , and Ka _i represents the semantic type value corresponding to a _i (such as the type value of pavement signs), Different semantic types have different type values, p ₁ -p _M respectively represent the probability that the 1st to M-th pixels in a _i belong to _Kai , and p ₁ -p _M can be obtained from the output results of the deep neural network.

The above steps S301 to S303 are exemplary implementations of step S201.

Step S202, unifying the coordinate system of the effective pavement semantic information and the visual semantic information.

In specific implementation, the control unit may project the at least one valid road semantic information into the coordinate system of the local positioning map according to the current estimated pose P _n . For different sampling points C _n , due to the current estimated pose P _n Therefore, after the at least one valid pavement semantic information is projected according to the current estimated pose P _n of different sampling points C _n , it will correspond to different coordinate distributions in the local positioning map.

In one embodiment, the local positioning map may use a known coordinate system such as the GNSS coordinate system, or may independently have its own coordinate system. For example, the control unit may take the center point of the local positioning map as the origin, and take For the X-axis and Y-axis, establish the coordinate system of the local positioning map.

Further, after determining the coordinate system of the local positioning map, the control unit can project the effective road semantic information from the camera coordinate system to the coordinate system of the local positioning map by means of matrix transformation. Among them, the transformation matrix can be, for example, a matrix with a size of 4×4, and its mathematical meaning represents the spatial transformation process of one translation and one rotation, that is to say, any pixel point in the valid road semantic information can undergo one translation and one rotation. The rotation is projected into the coordinate system of the local positioning map. Projection transformation of spatial points between different coordinate systems is a common method in the field of navigation and positioning, and will not be repeated here.

Step S203, determining a semantic association relationship between at least one valid pavement semantic information and at least one visual semantic information.

The purpose to be achieved in step S203 is: for any valid pavement semantic information a _i (a _i ∈ A) in the effective pavement semantic information set, find a semantically related to a _i from the visual semantic information set B of the local positioning map. Visual semantic information b _j (b _j ∈ B).

In one embodiment, step S203 as shown in FIG. 18 can be implemented by the following steps S401-S404:

Step S401, calculating the semantic weight of the effective pavement semantic information a _i .

In the specific implementation, the following formula can be used:

in,

Represents the semantic weight of a _i , M represents the number of pixels included in a _i , p _m represents the probability that the mth pixel in a _i belongs to _Kai , and Kai _{represents the semantic type value corresponding to a i} _.

Step S402, calculating the semantic weight of the visual semantic information b _j .

In the specific implementation, the following formula can be used:

in,

Represents the semantic weight of b _j , G represents the number of pixels contained in b _j , and p _g represents the gth pixel in b _j belongs to

The probability,

Represents the semantic type value corresponding to b _j .

What needs to be added here is that in the local map, it is known whether a pixel belongs to a road sign or a road traffic sign, so for the gth pixel in b _j , its p _g value has only two kinds of 0 and 0. 1 Two possibilities, if the pixel belongs to a road sign or a road traffic sign, then its p _g =1, otherwise p _g =0.

Step S403, according to the difference between the semantic weight of the effective pavement semantic information a _i and the semantic weight of the visual semantic information b _j , determine the semantic relevance of the effective pavement semantic information a _i and the visual semantic information b _j .

In the specific implementation, for any effective pavement semantic information a _i and any visual semantic information b _j , the semantic relevance Δw is the absolute value of the difference between the effective pavement semantic information a _i and any visual semantic information b _j , that is, the following formula is used:

Step S404, when the semantic correlation degree is less than the preset first threshold, it is determined that the effective road semantic information a _i and the visual semantic information b _j have semantic correlation.

In specific implementation, for any valid pavement semantic information a _i and any visual semantic information b _j , if their semantic relevance Δw is less than the preset first threshold σ, then the effective pavement semantic information a _i and any visual semantic information b _j have Semantic correlation; if its semantic correlation degree Δw is greater than or equal to the preset first threshold σ, the effective road semantic information a _i and any visual semantic information b _j do not have semantic correlation.

In this embodiment of the present application, by performing the operations from step S401 to step S404 multiple times, at least one pair of valid pavement semantic information and visual semantic information with semantic association can finally be obtained. For convenience of description, valid pavement semantic information and visual semantic information with semantic association Information may be referred to as a semantically related group.

The above steps S401 to S404 are exemplary implementations of step S203.

In step S204, semantic matching is performed on each pair of semantically associated valid pavement semantic information and visual semantic information, and a second weight is determined according to the semantic matching result.

In the specific implementation, as shown in FIG. 19 , step S204 can be implemented by the following steps S501-S503:

Step S501, respectively calculating the matching distance between each pair of semantically associated effective pavement semantic information and visual semantic information.

Among them, for each pair of valid pavement semantic information and visual semantic information with semantic association, the matching distance may be the Euclidean distance (ie Euclidean distance) of the effective pavement semantic information and visual semantic information, or cosine distance, etc. The embodiment is not specifically limited.

The mathematical meaning of the Euclidean distance and the cosine distance will be described below with reference to FIG. 20 . FIG. 20 shows the positions of the valid road semantic information a _i and the visual semantic information b _j in the coordinate system of the local localization map. Then the Euclidean distance is the straight-line distance between the effective pavement semantic information a _i and the visual semantic information b _j in the coordinate system of the local positioning map. The cosine distance is the cosine value of the angle α _ij between the effective road semantic information a _i and the visual semantic information b _j and the origin of the coordinate system.

Step S502, the weighted summation of each matching distance is performed to obtain a total matching distance.

To obtain the total matching distance, step S502 can be implemented through the following steps S601-S602 as shown in FIG. 21 .

Step S601, calculating the weight of each semantic association group.

Specifically, the following formula can be used:

Among them, K represents the number of semantic association groups, w′ _k represents the weight of the k-th semantic association group, and Δw _k represents the semantic association degree of the k-th semantic association group.

Step S602: Determine the total matching distance according to the weights and matching distances of all semantic association groups.

Among them, the weight of each semantic association group and the matching distance can be multiplied separately, and then all the multiplication results are summed to obtain the total matching distance E, that is, the following formula is used:

Among them, K represents the number of semantic association groups, w′ _k represents the weight of the k-th semantic association group, and E _k represents the matching distance of the k-th semantic association group.

Step S503, determine the second weight of the current estimated pose P _n according to the total matching distance

The total matching distance represents the closeness between the actual pose of the vehicle determined based on the visual features and the current estimated pose _Pn of the sampling point _Cn , and the higher the closeness, the second weight.

The higher, the lower the degree of proximity, the second weight

lower.

In one embodiment, when the matching distance is the Euclidean distance, the smaller the value of the total matching distance is, the smaller the value of the total matching distance indicates the difference between the actual pose of the vehicle determined based on the visual feature and the current estimated pose P _n of the sampling point C _n The higher the degree of proximity, the corresponding second weight

The larger the value of the total matching distance, the lower the proximity between the actual vehicle pose determined based on the visual feature and the current estimated pose P _n of the sampling point C _n , so the corresponding second weight

smaller.

In one embodiment, when the matching distance is a cosine distance, the smaller the value of the total matching distance, the higher the degree of closeness between the actual pose of the sampling point C _n determined based on the visual feature and the current predicted pose P _n , so the corresponding second weight

The larger the value of the total matching distance, the lower the proximity between the actual pose of the sampling point C _n determined based on the visual feature and the current predicted pose P _n , so the corresponding second weight

smaller.

Based on the above total matching distance and the second weight

The numerical transformation relationship between , the control unit can use any algorithm to determine the second weight

This embodiment of the present application does not limit this. second weight

The value range of , for example, may be [0, 1], or other ranges, which are not limited in this embodiment of the present application. Illustratively, the second weight

It can be obtained by complementing the total matching distance, taking the reciprocal, and normalizing the numerical range.

The above steps S501 to S504 are exemplary implementations of step S204.

Step S105: Calculate the current estimated pose P ₁ according to the current estimated pose P ₁ -PN of the _N sampling points C ₁ to _CN and the first weight and the second weight _- The weighted average of P _N , and the weighted average is used as the current pose of the vehicle.

In the specific implementation, step S105 can be implemented by the following steps S701-S703 as shown in FIG. 22 :

Step S701, using the first weights of the current estimated poses _P ₁ -PN to weight the current estimated poses _P ₁ -PN to obtain a first weighted average.

In one embodiment, the first weighted average can be obtained by the following formula:

Among them, P _l represents the first weighted average value, N is the number of sampling points; P _n represents the current estimated pose of the nth sampling point;

is the first weight corresponding to the current estimated pose of the nth sampling point,

is the first weight corresponding to the currently estimated pose P ₁ - P _N of the sampling points C ₁ _-CN

Sum.

Step S702, using the second weight of the current estimated pose _P ₁ -PN to weight the current estimated pose _P ₁ -PN to obtain a second weighted average.

In one embodiment, the second weighted average can be obtained by the following formula:

Among them, P _c represents the second weighted average, N is the number of sampling points; P _n represents the current estimated pose of the nth sampling point;

is the second weight corresponding to the current estimated pose of the nth sampling point,

is the second weight corresponding to the current estimated pose P ₁ - P _N of the sampling points C ₁ _-CN

Sum.

Wherein, step S701 and step S702 are only used to describe the method steps, and do not represent the sequence of steps. Generally speaking, the control unit may execute step S701 and step S702 in parallel, or may execute step S701 and step S702 in sequence. In addition, in this embodiment of the present application, the first weighted average value and the second weighted average value are both poses.

Step S703, the first weighted average value and the second weighted average value are weighted and averaged to obtain a weighted average value, and the weighted average value is used as the current pose of the vehicle.

In the specific implementation, the positioning result of the vehicle can be obtained by the following formula:

P = α _l ·P _l +α _c ·P _c (14)

Among them, P is the current pose of the vehicle, that is, the actual pose of the vehicle output by the positioning system at this time; P _l represents the first weighted average value, P _c represents the second weighted average value; α _l represents the first weighted average value. Weight, α _c represents the weight of the second weighted average value, α _l +α _c =1. The values of α _l and α _c may be determined according to actual requirements, and the larger the value of α _l or α _c , the higher the weight of the first weighted average or the second weighted average. In specific practice, if the technician wants to use the laser feature to dominate the positioning result, he can increase the value of α _l , for example, the value of α _l is 0.7, 0.8, etc.; if the technician wants to use the visual feature to dominate the positioning result, he can increase the value of α l The value of large α _c , for example, the value of α _c is 0.7, 0.8, etc.; if the technician wants the laser feature and the visual feature to play an equal role in the positioning result, the value of α _l and α _c can be both 0.5.

It can be understood that, after the execution of steps S101 to S105 is completed, the positioning system has completed a complete positioning process. During the driving process of the vehicle, since the pose of the vehicle changes constantly, the positioning process is also continuously performed.

It is understandable that, due to the influence of odometer errors or other factors, with the increase of the number of positioning times, the distribution of sampling points may diverge. Wherein, the divergence condition of the sampling points can be set by those skilled in the art, for example: when the second weight of the currently estimated pose corresponding to a preset ratio of sampling points is lower than the preset threshold, the sampling points are considered divergent; or, When the first weight of the currently estimated pose corresponding to a preset ratio of sampling points is lower than the preset threshold, the sampling points are considered to be divergent. When the sampling points diverge, the control unit may reselect the same number of sampling points as before. The embodiment of the present application also does not limit the method of reselection of sampling points. For example, when the first weight and/or the second weight are compared A certain proportion (for example: 90%) of sampling points are selected around the high one or more currently estimated poses, and the remaining proportion (for example: 10%) of sampling points are selected around the poses output by the GNSS/INS combination module. In addition, the control unit may also reselect the sampling points periodically, for example, every 100 positioning frames is regarded as a reselection period, and the sampling points are reselected.

The positioning method provided by the embodiment of the present application can respectively determine the first weight of the currently estimated pose based on the matching of laser features, and determine the current estimated pose based on the matching of visual semantic information and road semantic information (that is, the matching of visual features). The second weight, and then weighted and averaged the current estimated pose according to the first weight and the second weight to obtain the positioning result of the vehicle, thus realizing the combination of the laser positioning technology based on laser feature matching and the visual feature matching based laser positioning technology. The visual positioning technology is integrated to improve the positioning efficiency. Moreover, the positioning method provided by the embodiments of the present application encodes and stores laser features and visual semantic information in the same positioning map, which realizes the fusion of map data and reduces the data overhead and calculation overhead generated in the positioning process.

The above-mentioned embodiments provided in this application introduce various solutions of the positioning method. It can be understood that, in order to realize the above-mentioned functions, the positioning system may include corresponding hardware structures and/or software modules for performing each function. Those skilled in the art should easily realize that the present application can be implemented in hardware or a combination of hardware and computer software with the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

In one embodiment, the positioning system may implement corresponding functions through the hardware structure shown in FIG. 8 , the positioning system may be installed in the vehicle, and the installation manner of some components may be as shown in FIG. 4 . Among them, the GNSS/INS combination module 130 is used to determine the initial pose of the vehicle; the memory 190 is used to store the positioning map, the positioning map includes a plurality of positioning pictures spliced with each other, the positioning picture includes a color channel, and the color channel stores the first coding of laser features and coding of visual semantic information; lidar 110 for collecting point cloud data, point cloud data containing second laser features; camera 120 for collecting image data, image data containing at least one pavement semantic information; control The unit 140 is used to generate N sampling points C ₁ to C _N around the initial pose, where N is a positive integer; the control unit 140 is also used to extract the first laser feature and the At least one piece of visual semantic information; the control unit 140 is further configured to, for any sampling point C _n , n is a positive integer, n≤N, according to its corresponding current estimated pose P _n , extract the first laser feature of the self-positioning map Matching with the second laser feature to determine the first weight of the current estimated pose P _n , the second laser feature is extracted from the point cloud data collected by the lidar; the control unit 140 is also used for any sampling point C _n , according to the current estimated pose P _n , at least one visual semantic information is matched with at least one pavement semantic information to determine the second weight of the current estimated pose P _n , and the at least one pavement semantic information is collected from the camera extracted from the image data; the control unit 140 is further configured to calculate the current estimated pose according to the current estimated poses P ₁ -PN of the _N sampling points C ₁ to _CN and their first weights and second weights The weighted average of P ₁ - P _N , and the weighted average is taken as the current pose of the vehicle.

In one embodiment, the positioning picture includes a first color channel used to store an encoding of visual semantic information.

In one embodiment, the positioning picture further includes a second color channel for storing the encoding of the first laser characteristic.

In one embodiment, the coding of visual semantic information includes at least one of flag bit, type coding and brightness coding; the flag bit is used to represent the type of the road marking, the type coding is used to represent the content of the road marking, and the brightness information coding is used for Used to represent the brightness information of the picture.

In one embodiment, when the control unit 140 is configured to extract at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle: the control unit 140 is specifically configured to extract the local positioning from the positioning map according to the current predicted pose Map, the local positioning map includes M positioning pictures, the M positioning pictures include the first picture where the current predicted pose is located, and M-1 second pictures near the first picture, where M is a positive integer greater than 1; the control unit 140, which is further used for extracting at least one visual semantic information from the local localization map.

In one embodiment, when the control unit 140 is configured to match at least one visual semantic information with at least one road semantic information according to the current estimated pose P _n for any sampling point C _n to determine the current estimated pose P _n When the second weight is: the control unit 140 is specifically configured to determine at least one valid pavement semantic information from the at least one pavement semantic information, and the number of pixels of each valid pavement semantic information is within a preset range; the control unit 140 is also configured to use projecting at least one valid pavement semantic information into the coordinate system of the local positioning map according to the current estimated pose P _n ; the control unit 140 is further configured to determine the semantic association relationship between the at least one valid pavement semantic information and the at least one visual semantic information; The control unit 140 is further configured to perform semantic matching on each pair of semantically related valid pavement semantic information and visual semantic information, and determine the second weight according to the semantic matching result.

In one embodiment, when the control unit 140 is used to determine the semantic relationship between at least one valid pavement semantic information and at least one visual semantic information: the control unit 140 is specifically adapted to calculate the semantic weight of any valid pavement semantic information a _i and the semantic weight of any visual semantic information b _j ; the control unit 140 is further configured to determine the effective pavement semantic information a according to the difference between the semantic weight of the effective pavement semantic information a _i and the semantic weight of the visual semantic information b _j The semantic correlation degree between _i and the visual semantic information b _j ; the control unit 140 is further configured to determine that the effective road semantic information a _i and the visual semantic information b _j have semantic correlation when the semantic correlation degree is less than the preset first threshold.

In one embodiment, when the control unit 140 is configured to perform semantic matching on each pair of semantically associated valid pavement semantic information and visual semantic information, and determine the second weight according to the semantic matching result: the control unit 140 is specifically configured to respectively Calculate the matching distance of each pair of semantically associated valid pavement semantic information and visual semantic information; the control unit 140 is also used for the weighted summation of the respective matching distances obtained by calculation to obtain the total matching distance; the control unit 140 is also used for according to The total matching distance determines the second weight.

In one embodiment, when the control unit 140 is configured to calculate the current estimated pose P according to the current estimated poses P ₁ -PN of the _N sampling points C ₁ _-CN and their first weights and second weights When the weighted average value of _1- _PN is used as the current pose of the vehicle: the control unit 140 is specifically configured to use the first weight of the current estimated pose _P ₁ -PN to determine the current estimated pose P ₁ -P _N is weighted and averaged to obtain a first weighted average value; the control unit 140 is further configured to use the second weight of the current estimated pose _P ₁ _-PN to weight the current estimated pose P ₁ -PN , to obtain a second weighted average value; the control unit 140 is further configured to perform a weighted average of the first weighted average value and the second weighted average value to obtain a weighted average value, and use the weighted average value as the current pose of the vehicle.

In one embodiment, the pavement semantic information includes: a pixel block containing at least one pavement marker, the number of pixels in the pixel block, and the type of pavement marker to which each pixel belongs.

In one embodiment, the positioning system further includes an odometer 180; the control unit 140 is further configured to determine the relative pose of the vehicle between the current time t and the first historical time t-1 according to the data of the odometer 180; the control unit 140 is further configured to add the predicted pose corresponding to the sampling point C _n at the first historical time t-1 and the relative pose to obtain the current predicted pose.

In one embodiment, the control unit 140 is further configured to regenerate N sampling points C ₁ -C _N when there is a preset ratio of the first weight or the second weight is lower than the second threshold.

In another embodiment, the positioning system may implement corresponding functions through the software modules shown in FIG. 23 . As shown in FIG. 23 , the positioning system may include a sampling point generation module 810 , an extraction module 820 , a first matching module 830 , a second matching module 840 , and a solving module 850 . The functions of the above modules are described in detail below:

The sampling point generating module 810 is used for determining the initial pose of the vehicle, and generating N sampling points C ₁ -C _N around the initial pose, where N is a positive integer;

The extraction module 820 is configured to extract the first laser feature and at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle.

The first matching module 830 is configured to, for any sampling point C _n , where n is a positive integer, n≤N, according to its corresponding current estimated pose P _n , extract the first laser feature and the second laser feature of the self-positioning map Matching is performed to determine the first weight of the current estimated pose _Pn , and the second laser feature is extracted from the point cloud data collected by the lidar.

The second matching module 840 is configured to, for any sampling point C _n , match at least one visual semantic information with at least one pavement semantic information according to the current estimated pose P _n to determine the second weight of the current estimated pose P _n value, at least one pavement semantic information is extracted from the image data collected by the camera.

The solving module 850 is configured to calculate the weight of the current estimated pose P ₁ -PN according to the current estimated pose _P ₁ -PN of the _N sampling points C ₁ _-CN and its first weight and second weight The average value is calculated by weighting as the current pose of the vehicle.

The positioning map includes a plurality of positioning pictures that are spliced with each other, the positioning pictures include a color channel, and the encoding of the first laser feature and the encoding of the visual semantic information are stored in the color channel.

The positioning system provided by the embodiment of the present application can determine the first weight of the currently estimated pose based on the matching of laser features, and determine the current estimated pose based on the matching of visual semantic information and road semantic information (ie, matching of visual features). The second weight, and then calculate the weighted average of the current estimated pose according to the first weight and the second weight, and use the weighted average as the current pose of the vehicle, thereby realizing the laser positioning based on laser feature matching. The technology is integrated with the visual positioning technology based on visual feature matching, which improves the positioning efficiency. In addition, the technical solutions provided by the embodiments of the present application encode and store laser features and visual semantic information in the same positioning map, which realizes the fusion of map data and reduces the data overhead and computing overhead generated in the positioning process.

The embodiments of the present application further provide a vehicle, which may include the positioning system provided by the foregoing embodiments, and the user executes the positioning methods provided by the foregoing embodiments.

Embodiments of the present application further provide a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, when the computer-readable storage medium runs on a computer, the computer executes the methods of the above aspects.

Embodiments of the present application also provide a computer program product containing instructions, which, when run on a computer, cause the computer to execute the methods of the above aspects.

The above specific embodiments further describe in detail the purposes, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementations of the embodiments of the present application, and are not intended to limit the implementation of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

A positioning method, comprising:

Determine the initial pose of the vehicle, and generate N sampling points C 1 to C N around the initial pose, where N is a positive integer;

Extract the first laser feature and at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle; for any sampling point C n , n is a positive integer, n≤N, according to its corresponding current estimated pose P n , the first laser feature extracted from the localization map is matched with the second laser feature to determine the first weight of the current estimated pose P n , and the second laser feature is the point cloud collected from the lidar extracted from the data; and,

For any sampling point C n , according to the current estimated pose P n , the at least one visual semantic information and at least one road semantic information are matched to determine the second weight of the current estimated pose P n , The at least one pavement semantic information is extracted from image data collected by a camera;

According to the current estimated poses P 1 -PN of the N sampling points C 1 -CN and the first weight and the second weight, the current estimated pose P 1 -PN is calculated The weighted average of , and the weighted average is taken as the current pose of the vehicle;

The positioning map includes a plurality of positioning pictures spliced with each other, the positioning pictures include a color channel, and the encoding of the first laser feature and the encoding of the visual semantic information are stored in the color channel.
The method according to claim 1, wherein the positioning picture comprises a first color channel, and the first color channel is used to store the encoding of the visual semantic information.
3. The method of claim 2, wherein the positioning picture further comprises a second color channel for storing the code of the first laser feature.
The method according to any one of claims 1-3, wherein the coding of the visual semantic information comprises at least one of a flag bit, type coding and luminance coding; the flag bit is used to represent the Type, the type code is used to represent the content of the road marking, and the brightness information code is used to represent the brightness information of the picture.
The method according to any one of claims 1-4, wherein the at least one visual semantic information is extracted by the following steps:

A local positioning map is extracted from the positioning map according to the current predicted pose, the local positioning map includes M positioning pictures, the M positioning pictures include the first picture where the current predicted pose is located, and the M-1 second pictures near the first picture, where M is a positive integer greater than 1;

The at least one visual semantic information is extracted from the local localization map.
The method according to claim 5, wherein, for any sampling point C n , according to the current estimated pose P n , the at least one visual semantic information and at least one road semantic information are matched to obtain Determining the second weight of the current estimated pose P n includes:

determining at least one valid pavement semantic information from the at least one pavement semantic information, and the number of pixels of each valid pavement semantic information is within a preset range;

Projecting the at least one valid road semantic information into the coordinate system of the local positioning map according to the current estimated pose P n ;

determining the semantic association relationship between the at least one valid pavement semantic information and the at least one visual semantic information;

Semantic matching is performed on each pair of semantically associated valid pavement semantic information and visual semantic information, and the second weight is determined according to the semantic matching result.
The method according to claim 6, wherein the determining the semantic association relationship between the at least one valid pavement semantic information and the at least one visual semantic information comprises:

Calculate the semantic weight of any valid pavement semantic information a i and the semantic weight of any visual semantic information b j ;

According to the difference between the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j , determine the semantic relevance of the effective pavement semantic information a i and the visual semantic information b j ;

When the semantic correlation degree is less than a preset first threshold, it is determined that the effective road semantic information a i and the visual semantic information b j have semantic correlation.
The method according to claim 6 or 7, wherein the semantic matching is performed on each pair of semantically associated valid pavement semantic information and the visual semantic information, and the second weight is determined according to a semantic matching result. values, including:

respectively calculating the matching distance between the effective pavement semantic information and the visual semantic information for each pair of semantically associated semantic information;

A total matching distance is obtained by the weighted summation of each of the calculated matching distances;

The second weight is determined according to the total matching distance.
The method according to any one of claims 1-8, wherein the currently estimated poses P 1 -PN and the first weights thereof according to the N sampling points C 1 -CN and the second weight, calculating the weighted average of the current estimated pose P 1 - P N , and updating the current predicted pose with the weighted average value, including:

Using the first weight of the currently estimated pose P 1 -PN to weight the current estimated pose P 1 -PN to obtain a first weighted average;

Using the second weight of the current estimated pose P 1 -PN to weight the current estimated pose P 1 -PN to obtain a second weighted average;

The first weighted average value and the second weighted average value are weighted and averaged to obtain the weighted average value, and the weighted average value is used as the current pose of the vehicle.
The method according to any one of claims 1-9, wherein the pavement semantic information comprises: a pixel block containing at least one pavement mark, the number of pixels in the pixel block, and the pavement mark to which each pixel belongs. type.
The method according to any one of claims 1-10, wherein the current predicted pose is determined by the following steps:

Determine the relative pose of the vehicle between the current time t and the first historical time t-1 according to the odometer data;

The current predicted pose is obtained by adding the predicted pose corresponding to the sampling point C n at the first historical time t-1 and the relative pose.
The method according to any one of claims 1-11, further comprising:

When there is a preset proportion of the first weight or the second weight is lower than the second threshold, the N sampling points C 1 -C N are regenerated.
A positioning system, comprising:

Vehicle-mounted GNSS/INS combination module, control unit, memory, lidar and camera;

The GNSS/INS combination module is used to determine the initial pose of the vehicle;

The memory is used to store a positioning map, the positioning map includes a plurality of positioning pictures spliced with each other, and the positioning picture includes a color channel, and the color channel stores the encoding of the first laser feature and the encoding of the visual semantic information;

The lidar is used to collect point cloud data, and the point cloud data includes second laser features;

the camera is used to collect image data, the image data includes at least one pavement semantic information;

the control unit, configured to generate N sampling points C 1 to C N around the initial pose, where N is a positive integer;

The control unit is further configured to extract the first laser feature and at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle;

The control unit is further configured to, for any sampling point C n , where n is a positive integer, n≤N, according to its corresponding current estimated pose P n , extract the first laser feature and the second laser feature of the self-positioning map performing matching to determine the first weight of the currently estimated pose P n , and the second laser feature is extracted from the point cloud data collected by the lidar;

The control unit is further configured to, for any sampling point C n , match the at least one visual semantic information with at least one road semantic information according to the current estimated pose P n to determine the current estimated pose The second weight of P n , the at least one pavement semantic information is extracted from the image data collected by the camera;

The control unit is further configured to calculate the current estimated pose P 1 -PN of the N sampling points C 1 -CN and the first weight and the second weight of the current A weighted average of the poses P 1 - P N is estimated, and the weighted average is used as the current pose of the vehicle.
The positioning system according to claim 13, wherein the positioning picture includes a first color channel, and the first color channel is used to store the encoding of the visual semantic information.
14. The positioning system of claim 13, wherein the positioning picture further comprises a second color channel, wherein the second color channel is used to store the code of the first laser characteristic.
The positioning system according to any one of claims 13-15, wherein the coding of the visual semantic information includes at least one of a marker bit, a type encoding and a brightness encoding; the marker bit is used to represent road signs type, the type code is used to represent the content of the road marking, and the brightness information code is used to represent the brightness information of the picture.
The positioning system according to any one of claims 13-16, wherein when the control unit is used to extract the at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle:

The control unit is specifically configured to extract a local positioning map from the positioning map according to the current predicted pose, where the local positioning map includes M positioning pictures, and the M positioning pictures include the current predicted pose The first picture where it is located, and M-1 second pictures near the first picture, where M is a positive integer greater than 1;

The control unit is further configured to extract the at least one visual semantic information from the local positioning map.
The positioning system according to claim 13, wherein, when the control unit is used for any sampling point C n , according to the current estimated pose P n , the at least one visual semantic information is associated with the at least one road surface When matching semantic information to determine the second weight of the current estimated pose P n :

The control unit is specifically configured to determine at least one valid road semantic information from the at least one road semantic information, and the number of pixels of each valid road semantic information is within a preset range;

The control unit is further configured to project the at least one valid road semantic information into the coordinate system of the local positioning map according to the current estimated pose Pn ;

The control unit is further configured to determine the semantic association relationship between the at least one valid road semantic information and the at least one visual semantic information;

The control unit is further configured to perform semantic matching on each pair of the semantically associated valid road semantic information and the visual semantic information, and determine the second weight according to the semantic matching result.
The positioning system according to claim 18, wherein, when the control unit is used to determine the semantic association relationship between the at least one valid road semantic information and the at least one visual semantic information:

The control unit is specifically used to calculate the semantic weight of any valid pavement semantic information a i and the semantic weight of any visual semantic information b j ;

The control unit is further configured to determine the effective pavement semantic information a i and the visual semantic information according to the difference between the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j Semantic relevance of semantic information b j ;

The control unit is further configured to determine that the effective pavement semantic information a i and the visual semantic information b j have semantic correlation when the semantic correlation degree is less than a preset first threshold.
The positioning system according to claim 18 or 19, wherein when the control unit is used to perform semantic matching on each pair of semantically associated valid road semantic information and the visual semantic information, according to the semantic matching result When determining the second weight:

The control unit is specifically configured to separately calculate the matching distance between the effective pavement semantic information and the visual semantic information for each pair of semantically associated semantic information;

The control unit is also used to obtain the total matching distance by the weighted summation of each of the matching distances obtained by calculation;

The control unit is further configured to determine the second weight according to the total matching distance.
The positioning system according to any one of claims 13-20, characterized in that, when the control unit is used to estimate the poses P 1 -PN and the same according to the current estimated poses of the N sampling points C 1 -CN For the first weight and the second weight, the weighted average of the current estimated pose P 1 - P N is calculated, and the weighted average is used as the current pose of the vehicle:

The control unit is specifically configured to use the first weight of the current estimated pose P 1 -PN to perform a weighted average of the current estimated pose P 1 -PN to obtain a first weighted average;

The control unit is further configured to use the second weight of the current estimated pose P 1 -PN to perform a weighted average of the current estimated pose P 1 -PN to obtain a second weighted average;

The control unit is further configured to perform a weighted average of the first weighted average value and the second weighted average value to obtain the weighted average value, and use the weighted average value as the current pose of the vehicle.
The positioning system according to any one of claims 13-21, wherein the pavement semantic information includes: a pixel block containing at least one pavement mark, the number of pixels in the pixel block, and the pavement to which each pixel belongs. Type of flag.
The positioning system according to any one of claims 13-21, characterized in that, further comprising: an odometer;

The control unit is further configured to determine the relative pose of the vehicle between the current time t and the first historical time t-1 according to the odometer data;

The control unit is further configured to add the predicted pose corresponding to the sampling point C n at the first historical time t-1 and the relative pose to obtain the current predicted pose.
The positioning system according to any one of claims 13-23, wherein,

The control unit is further configured to regenerate the N sampling points C 1 -C N when there is a preset ratio of the first weight or the second weight is lower than a second threshold.
A vehicle, characterized by comprising the positioning system according to any one of claims 13-24.
A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer is made to perform the execution of any one of claims 1-12. Methods.
A computer program product, characterized in that, when the computer program product runs on a computer, the computer is caused to execute the method according to any one of claims 1-12.