WO2024028394A1

WO2024028394A1 - Method for localizing a mobile unit in hd maps, and vehicle

Info

Publication number: WO2024028394A1
Application number: PCT/EP2023/071426
Authority: WO
Inventors: Niklas Stannartz; Robert GONSCHOREK; Mario THEERS; Marc Sons; Adalberto HERNANDEZ LLARENA; Markus Kuhn; Oliver Maria KIND; Torsten Bertram; Philip DORPMÜLLER; Martin Krüger; Stefan Thomas SCHÜTTE
Original assignee: Zf Friedrichshafen Ag
Priority date: 2022-08-03
Filing date: 2023-08-02
Publication date: 2024-02-08
Also published as: DE102022208063A1

Abstract

Localization in HD maps is a challenge with regard to the processing of data and comparison between the actual position and localization of an HD map. The aim of the invention is to improve the comparison. For this purpose, an in particular computer-implemented method is provided which is designed for semantic and ambiguity-conscious localization of at least one mobile unit in at least one HD map, the HD map comprising digital copies of landmarks.

Description

ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 Method for localizing a movable unit in HD maps and vehicle The invention relates to a method, in particular a computer-implemented one, a control device, a vehicle and a computer program product. High-resolution maps, so-called (so-called) HD maps, can be used to enhance the information intake or perception of semi-autonomous or autonomous or highly automated vehicles, as well as the planning of a route, but also the prediction in relation to a situation, such as a To improve everyday driving situations or street scenes. Furthermore, such maps can serve to support the decision algorithms and improve their operation and application. In order to be able to use such maps in the applications mentioned, the vehicle in question, also referred to as an ego vehicle, must be able to be localized in the HD maps in a predictable and repeatable manner, with such localization requiring a high level of precision within the HD card required. The general requirements that must at least be met in order to establish localization algorithms are characterized by high precision and accuracy. Such precision and accuracy can be specified by lateral deviations of less than 0.2 m, by longitudinal deviations of less than 0.5 m and by elevation angle deviations, in so-called pitching movements, of less than 0.5 °. A high level of availability is also required, such as the quotient of “number of algorithm cycles available to the system” divided by the “total number of algorithm cycles”, which must be approximately 1. The closer this quotient is to 1, the better. This can be represented by Formula 1 as (1) ^{^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^} ^^^ ^^^^.

There must also be the highest possible level of completeness or reliability, as shown by Formula 2: (2) given localization error ≤ localization error limit. ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 In addition, high requirements are placed on a short runtime and low storage capacity requirements in order to allow an operation to run in real time if possible. Localization algorithms represent an additional computational overhead for lower-level driver assistance systems and/or driving systems of a highly automated vehicle. A higher computational effort requires a time delay, which is directly reflected in localization errors. Localization that is based on global navigation satellite signals and systems (Global Navigational Satellite Signals/Systems) or dead reckoning does not meet the precision and accuracy requirements mentioned in detail, especially not in lateral direction. For this reason, landmarks are recorded by vehicle sensors that perceive the environment and compared, i.e. "associated", with digital copies of the landmarks stored in the HD maps. Known approaches often choose the most likely data association (English: "Data Association" = DA), e.g. based on a maximum likelihood (German: Maximum Likelihood; English: "Maximum Likelihood" = ML) or they average over all possible ones (and plausible) data association hypotheses, e.g. in the form of association of probabilistic data (English: "Probabilistic Data Association" = PDA). In the first case, however, it can happen that the wrong position is chosen if at least two positions are plausible, whereas in the second case a middle position is chosen that is purely artificial. The German application DE 102019207087 A1 describes a method in which a hypothesis is derived from an initial position estimate, with landmarks being detected using environmental sensors in order to then compare them with a landmark that is stored in an HD map. The aim and described advantage is to directly assign at least one detected landmark to the digital copy of the landmark based on a few characteristic properties, such as map positions. The US is making a corresponding proposal in 2020 ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 0364883 A1. However, neither proposal is able to solve the problems described above. The problem in the prior art is that the known algorithms and systems can cause localization errors, are resource-intensive and do not meet the requirements for use in highly automated vehicles. It is therefore the object of the invention to reduce the amount of localization errors, save resources and improve performance. The object is achieved in particular by a method according to claim 1, a control device according to claim 22, a vehicle according to claim 23 or a computer program product according to claim 24. Advantageous embodiments and/or further developments are the subject of the subclaims, the description and/or the accompanying figures . In particular, the independent claims of one claim category can also be developed analogously to the dependent claims of another claim category. According to one aspect, the task is solved in particular by a method that is designed for the semantic and ambiguity-aware localization of at least one movable unit in at least one HD map. The HD map includes digital copies of landmarks. The method has in particular the following steps: - presenting and/or holding at least one HD map which includes digital copies of landmarks, - detecting several landmarks for association with the digital copies of landmarks in the at least one HD map, - including several initial hypotheses regarding a localization of a moving unit as initial localization hypotheses, - predictions of several localization hypotheses simultaneously, based on the initial localization hypotheses, their association with the detected landmarks ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 and taking into account a modeling of a semantic classification of the detected landmarks for assignment into semantic classes, and - weighting of the several localization hypotheses. The method further comprises in particular at least one of the following steps: - recognizing ambiguous situations with regard to the localization of the at least one movable unit when several localization hypotheses are simultaneously plausible, based on the weighting, the data association and the semantic classification, and/ or - clarifying ambiguous localization hypotheses (over time) by identifying the hypothesis with the highest weight. This allows semantic and ambiguity-aware localization of at least one moving unit in at least one HD map to improve localization. The method can be designed as a computer-implemented method or can be a computer-implemented method. Ambiguity is often a problem for localization algorithms, where the ambiguities can be reflected in data associations, leading to false associations between the measured or recorded landmarks in the real world and the digital copies of the landmarks stored in the HD maps, resulting in a downstream false Localization, a so-called localization error. The combined use of geographical and semantic localization can clarify some of the ambiguous situations/localizations, but this approach is not sufficient to clarify every situation or localization. The solution proposed here has the advantage that both the association of the landmarks in the real world with the associations in the HD map can be included in the evaluation and prediction, and the semantic classification of the landmarks can be taken into account. A semantic class misclassification on the part of the detector can also be modeled, for example through a conditional classification probability (CCP). This means that the probability of a misclassification on the part of the detector can also be taken into account, ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 which increases the prediction probability and thus also the security of the system, as localization errors can be reduced. Landmarks here and elsewhere in the description are side line boundaries, colored or white road markings, road signs, road features, points of interest (POIs), traffic lights, road intersection features, such as prominent ones (for a camera). LIDAR or another sensor that can be easily detected) features of streets or of the road itself. HD maps can display high-resolution digital maps that depict the world in a simplified (abstracted) manner, in particular the digital copies of the landmarks in these HD cards may be included. The general requirements that must at least be met in order to establish localization algorithms are characterized by high precision and accuracy. Such precision and accuracy can be specified by lateral deviations of less than 0.2 m, by longitudinal deviations of less than 0.5 m and by elevation angle deviations, in so-called pitching movements, of less than 0.5 °. The HD maps must be able to provide corresponding information. Since the world is depicted in an abstract way to optimize computing power and storage space resources, this can also apply to the digital copies of the landmarks. This means that the landmarks can in particular be abstracted in such a way that they do not satisfy any aesthetic needs, since the HD maps may (possibly) not be displayed, but are simply kept as a digital space. In this case, all structural features that may represent an internal structure that are not relevant for determining the distance of an ego vehicle to the landmark can have been removed. This can apply, for example, to plants, in particular to trees and/or shrubs, whose structure, particularly in the area of the crown, is simplified, for example into a ball or a cone or a cube or a cylinder. “Presentation” here can be understood to mean that the HD maps could have been made available before the start of the process. Under “hold” can ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 In contrast, the HD card remains independent of time, for example in a memory, whereby the retention can also occur after the start (initiation) of the process. In particular, the HD map is kept available as long as it is still needed, for example to enable localization hypotheses in the HD map. The term “in at least one HD map” refers to the fact that the localization hypotheses can be depicted in the HD map. The localization hypotheses are therefore not processed in a reference system of a sensor, in particular not in the reference system of an optical system, such as a camera. The detection of multiple landmarks can be made possible in particular by non-camera-based and non-optical systems. This may also include lidar, radar and/or comparable sensor systems. It may be possible to identify map-based position points as candidates for an initial localization hypotheses, which are based on measured global navigation satellite data, as well as on distance information, for example intra-roadway offset information, which determines the offset of a mobile unit, for example a vehicle, relative to the center of the roadway and therefore also relative to the road markings or away from the road markings. The inclusion of several initial hypotheses can be the initial estimation of hypotheses as a starting point for calculating further, subsequent calculation cycles of the localization hypotheses. The initial localization hypotheses can be based on an initial estimate or on a previously carried out calculation, for example because the method has already been carried out before and a localization data set generated in the process is (re) fed back into the method as initial localization hypotheses. The association between the detected landmarks and the digital copies of the landmarks can correspond in particular to a mathematical digital link or mapping, which makes it possible to use the detected landmarks ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 with digital copies of these landmarks in such a way that the landmarks and their digital copies in the HD map can be placed in a distance relationship with the mobile unit, whereby the mobile unit also has a can have a digital copy. In other words, the initial localization hypotheses can be associated with the digital copies of the landmarks in the HD map as a data association, with semantic class information relating to the landmarks being included in the prediction in addition to the data association, with at least one misclassification of a detected landmark is modeled into a semantic class in order to determine this as a semantic probability in addition to the data association and the weight calculation to weight the multiple localization hypotheses. ^ Predicting multiple localization hypotheses can be done simultaneously and continuously. This means that the predictions can be calculated at the same time. In addition, the predictions based on the multiple localization hypotheses can be made in a continuous phase space or continuous state space. The calculation does not have to be calculated in a concrete phase or state space. This is also a deterministic, not necessarily a stochastic calculation. The weighting of the multiple localization hypotheses can be calculated from the underlying structure of the algorithm, whereby the weightings can correspond to a probability of occurrence of a localization hypothesis, or can correspond to the probability that the movable unit is at the position in the real world (there is located) where the localization hypothesis predicted it accordingly. Resolving ambiguous localization hypotheses (over time) can be done by identifying the hypothesis with the highest weight. In addition, the clarification can be done taking data association and semantic classification into account. The hypothesis that has the highest weighting and therefore the highest probability of occurrence can be preferred ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03, or the highest probability that the hypothesis turns out to be the actual localization state. In addition, the one in which a misclassification was modeled by conditional classification probabilities (CCP) of the semantic classification of certain landmarks can also be rejected. The semantic classification can be included as an additional semantic likelihood within the data associations and the hypothesis probability calculation. The modeling of the CCP can be achieved, for example, through a confusion matrix of the detector. Here, the term “over time” can be understood to mean the time in which the method is operated or the number of repetition cycles in which the localization hypotheses are updated. According to a further aspect, in particular at least one of the steps of simultaneously estimating or predicting the multiple localization hypotheses and/or recognizing ambiguous situations with respect to the localization of the at least one mobile unit and/or clarifying ambiguous localization hypotheses will be subject to a recursive update. This allows the prediction to be further improved. As a result, the localization hypotheses can be continually improved, in particular the prediction accuracy can be further improved. The simultaneous estimation of a hypothesis, in particular for each lane, also allows efficient processing by the, in particular computer-implemented, method. A recursive update is to be understood in particular as an algorithm that uses the calculated parameters, weights, mixture components and/or localization hypotheses in a previous calculation cycle, or that uses the parameters, weights, mixture components and/or localization hypotheses calculated in the current cycle feeds back the coming cycle. This means that the localization hypotheses are continuously improved. Parameters, weights, mixture components and/or localization hypotheses can be used ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 presented or (initially) included are referred to as prior components, whereas parameters, weights, mixture components and/or localization components that are "passed through" or calculated through the cycle are referred to as posterior components can be designated. According to a further aspect, the localization can be provided as a 2D orientation of the movable unit. Additionally or alternatively, the localization hypotheses can be provided as 2D orientation hypotheses of the movable unit and/or the localization distribution can be represented as a Gaussian Mixture Model (GMM), whereby each mixture component h is connected to one of the localization hypotheses is. This allows a resource-saving, efficient model to be provided. In particular, the calculations and/or images are carried out directly in the HD map, with these taking place in a 2D view (English: bird-eye view = BEV). This means that instead of a 3D model with at least 6 degrees of freedom (DoF), you can work with only 3 degrees of freedom. This significantly reduces the necessary computing power and storage capacity requirements. This means that the states are calculated as a 2D orientation according to formula 3 (x: vector with coordinates x and y, as well as angular orientation ^^^ (3) ^^= [^^ ^^^^ ^T According to the Gaussian mixture model, the localization hypotheses are each expressed as a Gaussian standard or normal distribution is shown, which has a center and a distribution arranged around it (mean value covari-

no. In other words, the model is

Mixture formed according to formula 4: (4) %&^^ ' ^(* + _,- ) ^# .(! ^# , $ ^# )

In one example, a number of the mixture components may each correspond to a number of the localization hypotheses. In particular, the localization hypothesis can correspond to a localization in a lane. In other words ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 the number of lanes can correspond to the maximum of the mixture components, i.e. the parameter h. In other words, the localization hypotheses result in a superposition of all the possible Gaussian distributions that can be assigned to them. These can be weighted in such a way that the weights add up to one. This shows that the mobile unit is localized somewhere, i.e. exists and does not fall out of localization with a certain probability (missing localization if < 1), or can exist in several places (wrong localization if > 1). Because each localization hypothesis can be connected to a mixture component h, it is possible that a smaller number of (initial) localization hypotheses can be calculated than is the case with particle filter algorithms (PF) (as a stochastic algorithm ) is the case, which requires a higher number of components (h >> number of realistic localizations, whereby the realistic localizations correspond in particular to the number of lanes). The same applies to histogram filters (HF), in which the state space is discretized, but in which the necessary, small-scale initial hypothesis formation leads to a very large number of components h, which significantly exceeds the number of realistic localizations. Such an algorithm is therefore very resource-intensive. According to a further aspect, the inclusion can be an initiation, comprising presenting initial localization hypotheses, wherein at least part of the information on which at least part of the initial localization hypotheses is based is sensor information. This means that a direct relationship can be chosen between the initial localization hypotheses and the actual localization as the starting situation for carrying out the procedure. This means that further resources can be saved, since less computing time may be necessary to run an update process that optimizes the localization hypotheses in such a way that a localization hypothesis can be selected. ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 According to a further aspect, at least part of the information on which at least part of the initial localization hypotheses is based can be sensor information, which can include at least one of the following sensor information: - Global navigation satellite signal (Global Navigational Satellite Signal = GNSS), - movement data of at least one mobile unit or movement data of the ego vehicle, - landmark information, comprising a localization of landmarks, the landmarks being assignable to semantic classes and/or representations of the landmarks on at least one HD map exist, - tram marking information as landmark information, wherein the tram marking information is recorded by means of tram marking measurements or by means of a camera or by means of an on-board camera or by means of an on-board camera of the ego vehicle, and / or wherein the tram marking information is a lateral distance and / or a relative orientation with respect to the tram markings, and/or wherein the tram marking information contains class information, and/or wherein the tram marking information comprises semantic class information, and/or wherein the tram marking information comprises semantic class information, of which at least one of semantic classes "solid line" and/or "broken line" is/are included, and/or - intra-lane offset information, which at least shows the distance of the ego vehicle to the tram markings of the tram (= lane) on which the ego vehicle is is located according to at least one localization hypothesis, - positions aligned with at least one HD map, which is based on measured global navigation satellite data and intra-roadway offset information. This means that a large number of sensor data can be fed into the method in order to bring the initial localization hypotheses as close as possible to the real localization in order to save further resources in the calculation. GNSS can be used to support this, a vehicle movement parameter such as speed or yaw rate can be used ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 or road boundaries are recorded using an on-board camera, whereby the measurement data for road boundary data can include a lateral distance and a relative orientation relative to the road boundary or relative to the road boundaries. The orientation aids can also be assigned to semantic classes, for example (e.g.) “dashed line” or “solid line”. The yaw rate (also known as yaw velocity) refers to the angular speed of the rotation of a vehicle around the vertical axis. According to a further aspect, the simultaneous estimation or prediction of multiple localization hypotheses can include a prediction of multiple future localization hypotheses. This means that a corresponding cycle does not have to be run through for each localization hypothesis, but rather the calculations can be carried out at the same time, which further reduces the computing time. In addition, the individual localization hypotheses can be updated in a mathematically sound manner without having to start a new initialization, which would correspond to aborting a running cycle. In particular, both the weights as the probability of localization according to a localization hypothesis and the precise geometric orientation in the form of the mixture components can be estimated simultaneously, i.e. at the same time and in particular by the same algorithm and in particular within the same update cycle become. The probability that the localization corresponds to one of the localization hypotheses can correspond in particular to the localization in a lane, whereby the vehicle can be assigned to staying/driving in a lane. In a further aspect, predicting future localization hypotheses may include multi-hypothesis motion prediction, where each localization hypothesis is predicted independently. This also allows, in particular, the prediction of future localizations and, in particular, also allows the initial localization hypotheses to be updated not only on the basis of the initially entered initial localization hypotheses, but in particular also on the basis of the dynamic development caused by the movement of the movable unit. ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 According to a further aspect, the independent multi-hypothesis movement prediction can be based on a constant angular velocity model (English: "Constant-Turn-Rate-Velocity Model" = CTRV model), which uses measured velocities of and includes rotation rates and/or angular velocities w. This further improves the dynamics of the prediction. The parameter w can also be referred to as the yaw rate (also yaw speed), as described in detail. In particular, each hypothesis (with mean !"−1|"−1 ^#"−1 and covariance $"−1|"−1 ^#"−1 ) is calculated independently. According to a further aspect, the localization hypotheses can be weighted by weights during the prediction and/or the weights can remain constant during the prediction. The weights (wh(k-1)) of the mixture components (. the prediction can remain constant, with the respective pre-

Formula 5 can be calculated. The sum is given over the mixture components h from 1 to k-1, where k is the running parameter: ' ^{^}

According to a further aspect, the recognition of ambiguous situations can include at least one update step, the update being based on a cost matrix which is designed to associate data with incorrect predictions and/or with incorrect measurements and/or with measurement errors and to associate data. ths with classification probabilities. A cost matrix can be used, which is structured in such a way that the individual components calculate the costs of a false detection, as well as a sum comprising a geometric logarithm of the probability (Geometric log-likelihood), the costs for a Poisson distributed scatter related to measurement error (Poisson Clutter) and a semantic one ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 Classification as a logarithm of a probability (English: “semantic classification log-probability”). The number of lines corresponds in particular to the number of landmarks, for example the number of lanes. According to a further aspect, clarifying ambiguous localization hypotheses (over time) may include at least one update step, wherein the calculation of a best assignment matrix includes a Murty algorithm. In particular, a Murty algorithm works in such a way that at least one best solution to a problem is found, based on an initialized priority loop of problem/solution pairs. In particular, the pair that is at the top of a priority loop is the pair with the solution with the lowest cost. In particular, the top problem-solution pair <P, S> is taken from the priority loop and the solution S is included in the list of solutions that are to be output. Additional solutions can then be identified, based on the identified parameters of the first identified solution with the lowest cost. In particular, problem-solution pairs are identified whose solution has the lowest possible costs, which corresponds to a best solution. According to another aspect, clarifying ambiguous localization hypotheses (over time) can solve a linear assignment problem (LAP). This LAP can be designed to determine relationships between measurements and landmarks, so-called measurement-to-landmark associations. In one aspect, the linear assignment problem can be solved using a Hungarian algorithm. The Hungarian method is also known as the Kuhn-Munkres algorithm. The landmarks from the measurements and the digital copies of the landmarks to be associated can be of the same number and a unique association can be established between them, ie each source (s, measured or detected landmark) is assigned at most one target (t, digital Copy of the ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 Landmark) and each destination in particular is assigned at most one source. Each (permissible) assignment of a “source” to a “target” can have a price or a profit c(s,t). The algorithm can therefore minimize the total price (= sum of the individual prices) of the assignment or maximize the total profit (= sum of the individual profits) of the assignment. In particular, a maximum number of pairs must always be formed. According to one aspect, a posterior mixture component may be calculated and/or a posterior mixture component may be presented and/or maintained as a prior mixture component for a subsequent update step. In one aspect, a Kalman update can be carried out for each measurement-to-landmark association that is based on the previous prior mixture component. In addition, in another aspect, non-normalized weights or non-normalized log weights may be calculated based on corresponding measurement log probabilities and prior log weights. This means that the calculated posteriors can also be used as priors for the next update step, which allows further optimization and a corresponding association. In particular, those mixture components that are presented or (initially) included are referred to as priors, whereas mixture components that are "passed through" or calculated through the cycle are referred to in particular as posteriors. According to a further aspect, the weights can be normalized and/or the mixture components can be connected based on a distance and/or the mixture components can be sorted in descending order according to their weight and/or the number of hypotheses can be set to a maximum number of mixture components can be limited. This allows resources to be saved when calculating the mixture components, which means there are lower demands on computing power and storage space. This also reduces the additional computational effort. The connection of the mixture components also allows this ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 Achieving a localization that approaches reality or corresponds to it, since the moving unit is also localized and is not present in several locations. According to a further aspect, the geometric orientation of the localization hypotheses can be output as mixture components and the probability of the presence of the localization hypotheses is output in particular by their weights, with the mixture components and the weights being estimated simultaneously. This can speed up the implementation of the procedure. The simultaneous estimation can take place as a continuous estimation. This means that the predictions can be calculated at the same time. Furthermore, the predictions can be made based on the multiple localization hypotheses in a continuous phase space or continuous state space. The calculation does not have to be calculated in a specific phase or state space. This is also a deterministic, not necessarily a stochastic, calculation. In particular, determinism ensures that no “particle depletion” occurs, which “deprives” the system of any hypotheses. According to a further aspect, a recursive Bayesian estimation framework (“Bayesian Estimation Framework” = BEFW) can be applied, whereby inherent prior information is evaluated. This means that the method can be continued even if a mislocalization has been carried out, since further information is continuously supplied since prior information is not discarded. Bayesian filters are, in particular, recursive probabilistic methods for estimating probability distributions of hidden (unobserved) states of a system given observations and measurements. The measurements here are in particular the measurements of the landmarks that are to be associated with the copies of the landmarks. ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 According to a further aspect, a hypothesis can be calculated and/or output per lane, and/or the method can be designed to be deterministic, and/or within the update step, each hypothesis can be updated in a mathematically sound manner and/ or correctable, and/or a Kalman filter equation can be used as the basis for the mathematically based update. This allows for an improvement in resource requirements and reduces them. In particular, a Kalman filter is designed as a special case of a Bayesian filter for normally distributed states. The recursive update capability allows every hypothesis to be updated and thus corrected within an update step, as has already been described in detail. The mathematical foundation corresponds in particular to the ability of the method to stabilize itself again by means of a calculation and also to enable corrections that could force a restart if implemented differently. This reduces additional resources and costs for operating the process. According to a further aspect, the update of the hypothetical weights may be based on a non-Gaussian probability model and/or the update of the hypothetical weights may be based on a particle filter model. This means that the process has to be adapted. According to a further independent aspect, a control device or vehicle control device can be presented, designed and/or set up in such a way as to carry out a method according to the invention. The control device can also be specified in particular by the features of the method. The control device is also connected in particular to exteroreceptive sensors, i.e. to sensors that detect the surroundings of a movable unit, in order to be able to make the corresponding information (landmark positions) available to a method according to the invention. In addition, the control device can be designed to control a highly automated movable unit. The control device can include a storage medium (flash storage medium, magnetic storage medium, hard drive, USB stick, etc.) and a CPU. ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 A control device or a system of several control devices can be used in a motor vehicle. For example, one's own vehicle and/or its driver assistance system can include the control unit or the system of several control units. The control device or the system can be set up and intended for use in a motor vehicle. The control device or the system can have an electronic control. The control device or the system can be or have an electronic control unit (ECU). Several control devices can be provided. The multiple control devices can be connected via a bus system, for example a “Controller Area Network” (CAN), and/or exchange data with one another. The electronic control and/or the control device or system can have a microcomputer and/or processor. The control device or system can include one or more sensors and/or be connected to them. The control device or system can include the computer program product described above and/or below. The control device or system can have a memory. The computer program product may be stored in the memory. The control device or system can be designed to carry out the method described above and/or below. According to a further independent aspect, a vehicle or semi-autonomous vehicle or autonomous vehicle can be provided which includes a control device according to the invention. The vehicle can be a motor vehicle, for example an ego vehicle. The vehicle can be set up and/or intended to carry out the method described above and/or below. The vehicle can include the computer program product, control device or system of several control devices described above and/or below. In particular, a semi-autonomous vehicle can be understood as a level 1 to level 4 vehicle, with a level 5 vehicle being able to be provided as an autonomous vehicle. All vehicles in these categories are, in particular, highly automated vehicles. The levels are defined as follows in particular according to German federal law at the time of registration (or priority date): ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 Level 1: Assisted driving, whereby drivers are constantly in control of their vehicle. Level 2: Partially automated driving, where drivers are constantly in control of their vehicle. Level 3: Highly automated driving, whereby drivers are allowed to temporarily turn away from the driving task and traffic. Level 4: Fully automated driving, where drivers can hand over control of the vehicle over longer distances and in different traffic situations. Level 5: Autonomous driving, where there are only passengers without driving duties. The vehicle can be specified in particular by the features and functions of the method according to the invention and/or the control device according to the invention. According to a further independent aspect, a computer program product can be designed to be executed on a control device according to the invention in order to carry out a method according to the invention. The computer program product can in particular be designed in such a way that it can be stored (= saved) on the storage medium of the control unit. In particular, the computer program product can be designed in such a way that it can be executed on the CPU of the control unit in order to carry out the method according to the invention. The computer program product can cause a device, such as, for example, an electronic control and/or control and/or computing unit/device, a control system, a driver assistance system, a processor or a computer, to carry out the method described above and/or below. For this purpose, the computer program product can have corresponding data sets and/or program code means and/or the computer program and/or a storage medium for storing the data sets or the program. The computer program product can include program code means in order to carry out the method described above and/or below when the computer program product is executed on a processor. ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 The method can be implemented at least partially as a computer program on a computer, microcomputer, in an electronic control and/or computing unit, in a control system, on a storage medium or on a machine-readable - stored on a portable carrier and/or implemented there. In terms of software, the computer program can be distributed across one or more storage media, control and/or computing units, such as Electronic Control Units (ECUs) or computers, etc., in particular in one's own vehicle (ego vehicle). The storage medium can be a semiconductor memory, hard disk memory or an optical memory. Further training and advantageous embodiments result from the subclaims. The invention is described in more detail by the figures. Shown therein: a multi-hypothesis movement prediction, Fig. 5A is a schematic representation of a processing step according to a method according to the invention, and Fig. 5B is a graphic representation of a processing step according to a method according to the invention. The same reference numbers are used for features that are the same and have the same effect. Fig. 1 shows an everyday traffic situation 100 in which several moving units, shown here as vehicles 1, 2, are traveling on a multi-lane road that has several lanes. The road itself is divided into two ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 solid lines 4 are delimited to the outside, whereas a single lane is delimited from another lane by dashed lines 5. These lane markings serve as landmarks which are made available to a method according to the invention by means of an HD map which contains digital copies of these landmarks. There is an ambiguous driving situation for vehicles 1 in the middle lane, as no clear lane assignment can be made even by using the landmarks. The vehicles 2, on the other hand, are more likely to have a clear lane assignment, since their landmark combination (dashed line on the right, solid line on the left vs. dashed line on the left and solid line on the right) can be better clarified by the semantic class information. The vehicles can also be offset in the direction of travel shown here. The situation considered here is that all vehicles, as ego vehicles, want to determine their localization using a procedure. For this purpose, a large number of localization hypotheses 26a, 26b, 26c, 26d with corresponding orientation hypotheses are shown schematically. According to FIG. 2, a method 30 is represented in detail by the diagram shown. In a multi-lane initialization step A, a series of localization hypotheses 16a, 16b, 26c, 26d are introduced into the method, each of the initial localization hypotheses representing a Gaussian mixed component, which is summed according to formula 6:

The multi-lane initialization A is shown in further detail in FIG. 3 and will be described in more detail in this regard. Based on this, a first prediction B is made, which leads to the mixture components of formula 7 ^

In a step S31, which is to be assigned to a recursion C, the parameters h'k = 0 and hk-1 = 1 are selected as the starting point of the recursion. The recursion is an update process that involves the respective ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 Localization hypotheses updated in order to arrive at a statement for the localization of vehicle 1. h is the parameter through which the lane assignment is given as a localization hypothesis. In a step S32 it is checked whether this parameter h(k-1) is less than or equal to the penultimate value 4&" 5 ^^^ of the maximum number of lanes. A distinction is therefore made between cases. If this condition is affirmative, then a cost matrix is generated in step S33, which has the components of formula 8, which takes into account any incorrect detection of the landmarks: (8) ^ ^6.0 = log(1 − 7D) The parameter PD represents the detection probability. This part The cost matrix is diagonalized, whereas the off-diagonal elements approach infinity. The measurements themselves are then represented by the following formula 9 and a classification is carried out, also according to the semantic classification and according to the correct assignment probability. (9) ^ ^6.8 =log(.&9 ⁸ ; 9: ⁶ ,;^^^+ log(7D<^=^>?^^+log(7(DetectedClass|LandmarkClass)) Where 9 represents ⁸ ^@AB^C1DB^EBFFGHIJ^9 : ⁶ = g(x, i, L) the i-th expected measurement and ;^'^K^$^ K ^L ^the innovation covariance, with the measurement Jacobi matrices^K^'^@I&MJ^AJ^N^^=^ @MJ^dar. The function g is a measurement model, V is the volume of the sensor FoV and >? that for measurement errors (English: “Poisson Clutter Rate”). The first summand in Formula 9 represents a geometry likelihood, the second summand represents the costs of the Poisson Clutter and the third summand represents the probability that the semantic class was correctly assigned according to the actual class, which corresponds to a semantic classification according to a log -probability corresponds. This cost matrix combines the costs of data association with those of the probability of measurement error ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 False measurement costs, where there is a genuine incorrect measurement, and the probability of incorrect categorization in the semantic category. The cost matrix is then fed to a Murty algorithm, whereby a linear allocation problem LAP is solved using the Hungarian method. This determines the best assignment matrix and determines the measurement-to-landmark associations, according to Formula 10 (10) O ^{P^} = argminO Σ6Σ8Q6,8^R6,8^ ^ where C represents the cost matrix with components i and j and A becomes the linearization matrix in which the value of the definition space at which a minimum is given is selected. This means that the hypothesis with the lowest costs is preferentially selected or calculated. In step S35, the parameter is first set to 1 before a further case distinction is made in step S36 as to whether this parameter m is less than or equal to ^ES&T1^^^, the number of elements of the best association matrix. If this is not the case, i.e. the answer is NO or FALSE, then the parameter h(k-1) is increased by 1 in step S40 and returned to step S32. However, if the answer is YES or TRUE, then the factor h'k is increased by 1 in step S37 and a posterior component is calculated in step S38, i.e. a localization probability is updated. The parameter m is then increased by 1 in step S39 and transferred back to step S36. A further update cycle is therefore established which calculates the posterior components. This is done using a Kalman update, for each measurement-to-landmark association based on the h(k-1) prior mixture component, according to formulas 11 to 13 ^ &^^^^ U"^#"V^'^$"W"5^^{#"5^}&X"^#"V ^ ^{L^} &;"^#"V ^ ^{^5^^} &^Y^^ !"W"^#"V'^!"W"5^^{#"5^}TO"^#"V^&9"^#"V1^9:"^#[" ^^ &^\^^ $"|"^{^"^} = $"|"−1 ^{^"−1} −U"^{^"^}X"^{^"^}$"|"−1^{^"−1} ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 In addition, non-normalized (logarithmic) weights ^"^#" ^are calculated, which add to the measurement log-likelihood^^ ^AJC ^GH@^@B]^^_A`_1a`I1KB) AbSD^aT1^ ^{S&T1^^} ^correspond according to formula 14

^ &^c^^ de"^{#"V^}'^a`I^&fe"^#"V^^'^d"5^^{#"5^} Zd ^6J8 ^^

in step S36 is FALSE and is led back to step S32 via step S40. If the answer there continues to be TRUE, YES or TRUE, then the recursive update process continues and the corresponding matrices are filled and the posterior components are calculated. However, if the answer is FALSE, FALSE or NO in step S32, then the process is transferred to processing step S41. There the weights are normalized and mixture components that are in the same lane or are otherwise close together are brought together. Furthermore, the mixture components are sorted in descending order according to their weights and a maximum number of mixture components Hmax is limited. The normalization is carried out by the quotient of the individual weight and the sum of the weights, which is summed over the parameter h'k. The merging is based on formulas 16 and 17, which respectively concern the mean and the covariances. ^{(16) !^"|"#"^'^} Σ ₆ ^{^)TA^!^"|"A} ^{$^"|"#"^'^ TA^&!^"|"#"^1^!^ "|"A^&^!^"|"#"^1^!^"|"A^L}

represents. ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 The parameter k is then increased by 1 in step S42, the result of the recursion is fed back to prediction B as a new prior. In parallel, the hypothesis that is most likely is output. 3 shows a candidate position 7 matched with the map from a series of positions 6 at which a vehicle 1 can be present. The lane markings 4, 5 are as described in detail. A time delay-compensated GNSS measurement is available as measuring point 8, whereby the coherence of the GNSS measurement and thus its uncertainty is shown as circle 9. This also makes it clear why GNSS alone is not suitable for using localization in the situation described here. It cannot resolve the ambiguities alone or together with the semantic classification of the lane markings. In addition, an intra-lane offset with a distance to the left ?0,left or to the right lane marking ?0,right is clarified. The one lane that carries the vehicle has been “extended” for illustration purposes only. As shown in Fig. 3B, the multi-lane initialization is a localization with respect to candidates that is compared with the map, which corresponds to a normal distribution function 18, according to the formula 18 (18) ^6 = log (f6) = log (. (h6 ; hGNSS, $GNSS)) + log (7(LBMeasClass|LBMapClass)). This results in an initial distribution function 18 according to the formula 19 (19) %2|2(^2) ' ^(* + ₃ ³ ,- ) #2^.(!#2,$#2)^!#2^'^ij#2^ ^{L^} ^ ^ ^{GNSS ^L} where H0 represents the number of initial candidates pi. There is an initial localization 13 with an estimated maximum probability of 17, whereas two further candidates 14, 15 have lower occurrence probabilities. A fourth candidate 16 is very unlikely to be the correct localization as it has a clear

4 illustrates a multi-hypothesis movement prediction, as already described with reference to formula 5. A car has to ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 At the time of the initial estimate a measured speed 11 and a yaw rate 12, which forces the vehicle onto trajectory 10. In one embodiment, these parameters of speed and yaw rate are introduced into the described method steps and the mathematical modeling and processed there. In other words and intuitively understandable, the car changes location and also changes its orientation on the 2D plane. It also becomes clear here that the calculations and the associations take place from a bird's eye view, which optimizes the process for resources. 5A illustrates the combination of mixture components 26a and 26b described by formulas 16 and 17 with their respective graphically indicated covariances 25a and 25b. These overlap in a region 27. Step S41 will therefore result in both being merged into a single mixture component 29 with a new covariance 28. 5B shows a graphical representation of a result of the method, with a prior 20 representing an initial estimated probability distribution of the localization, as described above. Based on this, a probabilistic data association 31 is created. The procedure ultimately results in a posterior mixing component, which is seen as the most likely localization. As an example, it can be assumed that the vehicle is in the third lane from the left (second from the right). There the probability 22 for a localization is highest, while the second highest probability 21 and the third highest probability 23 fall significantly behind. This means that the risk of incorrect localization is significantly reduced compared to the initial situation. The method is not limited to the embodiments shown here, but is also able to cover the various implementations that emerge from the description, figures and claims here to a person skilled in the art. “May” refers in particular to optional features of the invention. Accordingly, there are also further developments and/or exemplary embodiments of the invention that ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 additionally or alternatively have the respective feature or features. If necessary, isolated features can also be selected from the feature combinations disclosed here and used in combination with other features to delimit the subject matter of the claim, while breaking down any structural and/or functional connection that may exist between the features. The order and/or number of all steps of the method can be varied.

ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 reference sign 1, 2 vehicle / movable unit 4 dashed line 5 Covered line 6 Localization candidates 7 with card of compensated localization candidate PI 8 from GNS's localization candidate 9 Kovariance of the localization hypothesis 11 Vehicle speed vector 12 greed rate Initial localization candidate with the highest probability 14, 15 Initial localization candidates with a low probability 16 Initial localization candidate with probability against 0 17 probability of the initial localization candidates with the highest probability 18 Initial localization hypothesis distribution function 20 Prior 21 Localization hypothesis with second highest probability with the highest probability 23 localization hypothesis with third Highest probability 24 posterior 25a, 25b Covariances 26a, 26b 26c, 26d (initial) localization hypotheses 27 Overlap of covariance areas 28 new covariance area 29 new localization mixture component 30 Process scheme 31 Probabilistic data association 100 Everyday street situation ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 A Initialization of a multi-lane hypothesis B Prediction C Recursion D Output S31, S35, S37, S39, S40, S42 Parameter setting steps for the recursion loops S32, S36 decision points

Claims

ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 Patent claims 1. Method designed for semantic and ambiguity-aware localization of at least one movable unit (2) in at least one HD map, the HD map containing digital copies of landmarks (4, 5) comprises, comprising the steps: - presenting and/or holding at least one HD map comprising digital copies of landmarks (4, 5), - detecting several landmarks (4, 5) for association with the digital copies of landmarks (4 , 5) in the at least one HD map, - including (A) several initial hypotheses regarding a localization of a mobile unit (2) as initial localization hypotheses (26a, 26b, 26c, 26d), - predictions (B) of several localization hypotheses (26a , 26b, 26c, 26d) simultaneously, based on the initial localization hypotheses (26a, 26b, 26c, 26d), their association with the detected landmarks (4, 5) and taking into account modeling of a semantic classification of the detected landmarks (4 , 5) for assignment into semantic classes, - weighting of the multiple localization hypotheses (26a, 26b, 26c, 26d), the method further comprising at least one of the following steps: - recognizing (C) ambiguous situations with regard to the localization of the minimum at least one moving unit (2), if several localization hypotheses (26a, 26b, 26c, 26d) are simultaneously plausible, based on the weighting, data association and semantic classification, and/or - clarifying (D) ambiguous localization hypotheses (26a , 26b, 26c, 26d) (over time) by identifying the hypothesis with the highest weight. 2. The method according to claim 1, characterized in that at least one of the steps of predicting (B) several localization hypotheses (26a, 26b, 26c, 26d) simultaneously and / or recognizing (C) ambiguous situations in relation to the localization of the at least one movable unit (2) and/or clarifying (D) ambiguous localization hypotheses (26a, 26b, 26c, 26d) are subject to a recursive update. ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 3. Method according to claim 1 or 2, characterized in that the localization is provided as a 2D orientation of the movable unit (2) and / or wherein the localization hypotheses (26a, 26b, 26c, 26d) are provided as 2D orientation hypotheses of the movable unit and / or wherein the localization distribution (18) is represented as a Gaussian Mixture Model (GMM), each mixture component h with one of the localization hypotheses (26a, 26b, 26c, 26d) connected is. 4. Method according to one of the preceding claims, characterized in that the inclusion (A) is an initiation, comprising presenting initial localization hypotheses (26a, 26b, 26c, 26d), wherein at least part of the information corresponds to at least part of the initial localization hypotheses (26a, 26b, 26c, 26d) are based on sensor information. 5. Method according to one of the preceding claims, characterized in that at least part of the information on which at least part of the initial localization hypotheses (26a, 26b, 26c, 26d) is based is sensor information which includes at least one of the following sensor information: - Global navigation satellite signal (Global Navigational Satellite Signal = GNSS), - movement data of at least one mobile unit (2) or movement data of the ego vehicle (1), - landmark information, comprising a localization of landmarks, wherein the landmarks can be assigned to semantic classes and/or wherein representations - stations of the landmarks exist on at least one HD map, - tram marking information as landmark information, the tram marking information being recorded by means of tram marking measurements or by means of a camera or by means of an on-board camera or by means of an on-board camera of the ego vehicle (1), and/or where the tram marking information includes a lateral distance and/or a relative orientation with respect to the tram markings (4, 5), and/or wherein the tram marking information contains class information, and/or wherein the tram marking information comprises semantic class information, and/or wherein the tram marking information is semantic ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 have class information, of which at least one of the semantic classes “solid line” (4) and/or “broken line” (5) are included, and/or - intra-roadway offset information (co, left, co, right), which include at least the distance of the ego vehicle (1) to the tram markings (4, 5) of the tram on which the ego vehicle (1) is localized according to at least one localization hypothesis, - Positions aligned with at least one HD map, which is based on measured global navigation satellite data and intra-lane offset information (co,left, co,right). 6. Method according to one of the preceding claims, characterized in that the simultaneous prediction (B) of several localization hypotheses (26a, 26b, 26c, 26d) includes a prediction of several future localization hypotheses. 7. Method according to one of the preceding claims, characterized in that the prediction of future localization hypotheses comprises a multi-hypothesis movement prediction, each localization hypothesis (26a, 26b, 26c, 26d) being predicted independently. 8. Method according to one of claims 6 or 7, characterized in that the independent multi-hypothesis movement prediction is based on a constant angular velocity model (Constant Turn Rate Velocity Model = CTVR model), which includes measured velocities v and rotation rates and/or angular velocity. speeds w included. 9. Method according to one of the preceding claims, characterized in that the localization hypotheses (26a, 26b, 26c, 26d) are weighted by weights during the prediction and/or the weights remain constant during the prediction. 10. Method according to one of the preceding claims, characterized in that the recognition (C) of ambiguous situations requires at least one update step ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03, the update being based on a cost matrix which is designed to associate data with incorrect predictions and/or with incorrect measurements and/or with measurement errors and to associate data with classification probabilities. 11. The method according to any one of the preceding claims, characterized in that clarifying (D) ambiguous localization hypotheses (26a, 26b, 26c, 26d) (over time) comprises at least one update step and wherein the calculation of a best assignment matrix includes a Murty algorithm includes. 12. Method according to one of the preceding claims, characterized in that clarifying (D) ambiguous localization hypotheses (26a, 26b, 26c, 26d) (over time) solves a linear assignment problem (LAP), designed to determine relationships between measurements and landmarks, so-called measurement-to-landmark associations. 13. The method according to claim 12, characterized in that the linear assignment problem is solved by a Hungarian method. 14. The method according to any one of the preceding claims, characterized in that a posterior mixture component is calculated and/or wherein a posterior mixture component is presented and/or retained as a prior mixture component for a subsequent update step. 15. The method according to any one of the preceding claims, characterized in that a Kalman update is performed for each measurement-to-landmark association that is based on the previous prior mixture component. 16. The method according to claim 15, characterized in that non-normalized weights or non-normalized logarithmic weights are calculated based on corresponding measurement logarithm probabilities and prior logarithm weights. ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 17. Method according to one of the preceding claims, characterized in that the weights are normalized and / or the mixture components are connected based on a distance and / or wherein mixture components according to their Weighting can be sorted in descending order and/or the number of hypotheses is limited to a maximum number of mixture components. 18. Method according to one of the preceding claims, characterized in that the geometric orientation of the localization hypotheses are output as mixture components and the probability of the presence of the localization hypotheses is output by their weights, the mixture components and the weights being estimated simultaneously. 19. Method according to one of the preceding claims, characterized in that a recursive Bayesian estimation framework (Bayesian Estimation Framework (BEFW) is applied, with inherent prior information being evaluated. 20. Method according to one of the preceding claims, characterized in that one hypothesis per tram is calculated and/or output and/or wherein the method is designed to be deterministic and/or wherein within the update step each hypothesis can be updated and/or corrected in a mathematically sound manner and/or wherein a Kayman filter equation is used as the basis for the mathematically sound update. 21. Method according to one of the preceding claims, characterized in that the update of the hypothetical weights is based on a non-Gaussian probability model and/or the update of the hypothetical weights is based on a particle filter model. 22. Control device or vehicle control device, designed and/ or set up in such a way as to carry out a method according to one of the preceding claims. ZF Friedrichshafen AG 212639 Friedrichshafen 2022-08-03 23. Vehicle or semi-autonomous vehicle or autonomous vehicle, comprising a control device according to claim 22. 24. Computer program product designed to be executed on a control device according to claim 22 in order to implement a method according to to carry out one of claims 1 to 21.