WO2022062355A1

WO2022062355A1 - Fusion positioning method and apparatus

Info

Publication number: WO2022062355A1
Application number: PCT/CN2021/084792
Authority: WO
Inventors: 丁磊; 戴必林
Original assignee: 华人运通(上海)自动驾驶科技有限公司
Priority date: 2020-09-23
Filing date: 2021-03-31
Publication date: 2022-03-31
Also published as: CN112150550B; CN112150550A

Abstract

A fusion positioning method and apparatus. The fusion positioning method comprises: collecting a picture related to a surrounding environment, and acquiring repositioning posture information on the basis of the picture related to the surrounding environment (S110); fusing at least one piece of predicted posture information in a predicted posture information queue on the basis of the repositioning posture information, so as to obtain fused posture information, wherein the predicted posture information queue comprises a plurality of pieces of predicted posture information (S120); updating the predicted posture information queue on the basis of the fused posture information (S130); and obtaining target positioning posture information on the basis of the updated predicted posture information queue (S140). By means of the above-mentioned method for fusing various types of positioning, positioning that is high in precision and high in robustness can be realized.

Description

A fusion positioning method and device

This application claims the priority of the Chinese patent application with the application number 202011007686.9 and the invention titled "A Fusion Positioning Method and Device" filed with the China Patent Office on September 23, 2020, the entire contents of which are incorporated into this application by reference .

technical field

The present application relates to the field of positioning, and in particular, to a fusion positioning method based on semantics and corner point information.

Background technique

With the rapid development of autonomous driving technology, the positioning function has almost become an essential function of autonomous driving. Currently, autonomous driving usually uses the Global Positioning System (GPS) to locate the autonomous driving. However, in indoor areas and areas with relatively dense buildings, GPS accuracy is very poor or even unable to work. Therefore, how to continuously provide stable and high-precision positioning information for vehicles or other movable and/or equipment has become a problem that needs to be solved. .

SUMMARY OF THE INVENTION

The embodiments of the present application provide a fusion positioning method and device to solve the problems existing in the related art, and the technical solutions are as follows:

In a first aspect, an embodiment of the present application provides a fusion positioning method, including:

Collect pictures related to the surrounding environment, and obtain relocation pose information based on the pictures related to the surrounding environment;

Based on the repositioning pose information, at least one predicted pose information in the predicted pose information queue is fused to obtain the fused pose information; wherein, the predicted pose information queue contains multiple predicted pose information;

Based on the fused pose information, update the predicted pose information queue;

Based on the updated predicted pose information queue, the target positioning pose information is obtained.

In a second aspect, an embodiment of the present application provides a fusion positioning device, including:

The relocation module is used to collect pictures related to the surrounding environment, and obtain relocation pose information based on the pictures related to the surrounding environment;

a fusion module, configured to fuse at least one predicted pose information in the predicted pose information queue based on the repositioning pose information to obtain the fused pose information; wherein the predicted pose information queue contains multiple predictions pose information;

an update module for updating the predicted pose information queue based on the fused pose information;

The pose determination module is used for obtaining target positioning pose information based on the updated predicted pose information queue.

In a third aspect, an embodiment of the present application provides an electronic device, the electronic device includes: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor , so that at least one processor can execute the above method for fusion positioning.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the method in any one of the implementation manners of the above aspects is executed.

The advantages or beneficial effects of the above technical solutions include at least: the present invention obtains the repositioning pose information through different positioning methods based on pictures related to the surrounding environment, which can ensure that the repositioning pose information can be obtained in different environments, ensuring that the repositioning pose information can be obtained in different environments. High robustness is achieved; the present invention also uses the repositioning pose information to update the predicted pose information queue, and obtains the target positioning pose information from the updated predicted pose information queue, and by fusing the environment-based repositioning pose information And the predicted positioning pose information, and finally obtain the target positioning pose information, which can realize the high-precision positioning of the vehicle or other movable machinery and equipment.

The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will become apparent by reference to the drawings and the following detailed description.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments described in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings.

1 is a schematic diagram of a fusion positioning method according to an embodiment of the present application;

2 is a schematic flowchart of a fusion positioning method according to another embodiment of the present application;

3 is a schematic diagram of a fusion positioning method according to another embodiment of the present application;

4 is a schematic diagram of corner extraction according to another embodiment of the present application;

5 is a schematic diagram of a fusion positioning method according to another embodiment of the present application;

6 is a schematic diagram illustrating the association between a repositioned pose and a predicted pose queue according to another embodiment of the present application;

7 is a structural block diagram of a fusion positioning apparatus according to an embodiment of the present application;

8 is a structural block diagram of a relocation module in a fusion location device according to another embodiment of the present application;

9 is a block diagram of an electronic device used to implement the fusion positioning method according to the embodiment of the present application;

10 is a structural block diagram of a fusion positioning apparatus according to another embodiment of the present application;

FIG. 11 is a structural block diagram of a fusion module in a fusion positioning apparatus according to another embodiment of the present application.

detailed description

In the following, only certain exemplary embodiments are briefly described. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present application. Accordingly, the drawings and description are to be regarded as illustrative in nature and not restrictive.

The following disclosure provides many different embodiments or examples for implementing different structures of the present application. To simplify the disclosure of the present application, the components and arrangements of specific examples are described below. Of course, they are only examples and are not intended to limit the application. Furthermore, this application may repeat reference numerals and/or reference letters in different instances for the purpose of simplicity and clarity, and does not in itself indicate a relationship between the various embodiments and/or arrangements discussed. In addition, this application provides examples of various specific processes and materials, but one of ordinary skill in the art will recognize the application of other processes and/or the use of other materials.

FIG. 1 shows a flowchart of a fusion positioning method according to an embodiment of the present application. As shown in Figure 1, the fusion positioning method may include:

Step S110: Collect pictures related to the surrounding environment, and obtain repositioning pose information based on the pictures related to the surrounding environment;

Step S120: fuse at least one predicted pose information in the predicted pose information queue based on the repositioned pose information to obtain the fused pose information; wherein the predicted pose information queue includes multiple predicted poses information;

Step S130: based on the fused pose information, update the predicted pose information queue;

Step S140: Based on the updated predicted pose information queue, obtain target positioning pose information.

Optionally, the fusion positioning method can be used for vehicles, and can also be used for mobile devices or equipment that need to be positioned at any time, such as robots.

In one embodiment, the picture related to the surrounding environment may be a front-view picture used for corner localization, or a look-around picture used for semantic localization. After that, the corresponding information in the picture is extracted, and based on different pictures, different positioning methods are adopted to obtain the repositioning pose information. The repositioning pose information may specifically include the repositioning position coordinates, the repositioning direction angle and the repositioning time, wherein , and the relocation time is the acquisition time corresponding to the above picture related to the surrounding environment.

After obtaining the repositioning pose information, as shown in Figure 2, "associate" it with at least one predicted pose information in the predicted pose information queue to obtain the fused pose information, the selected predicted pose information. It may also be referred to as a "temporary object"; in a specific embodiment, the predicted pose information queue contains multiple predicted pose information calculated at different times using a motion model, and the motion model is a The pose (including position coordinates and direction angle) at time t ₁ , the calculation parameters and the motion time t', the model of the pose (including position coordinates and direction angle) at time t ₁ +t' is calculated, and each predicted pose information can include predicted position coordinates, predicted direction angle and corresponding predicted positioning time. According to the repositioning time in the repositioning pose information, at least one predicted pose information that is close to the repositioning time is found, and the above repositioning pose is fused with it to obtain the fused pose information. Then, based on the fused pose information, update the predicted pose information queue. Specifically, based on the predicted positioning time of the fused pose information, the predicted pose information is recalculated by using the motion model and the fused pose information. The predicted pose information in the queue whose time is after the predicted positioning time, and the predicted pose information queue is updated.

Based on the updated predicted pose information queue, the target positioning pose information is obtained. For example, based on the updated predicted pose information queue, the newly added predicted pose information in the predicted pose information is found as the target positioning position. Posture. In this embodiment, the repositioning pose information can be obtained based on different positioning methods, and then fused with the predicted pose information to update the predicted pose information queue, and determine the final target positioning pose based on the updated predicted pose information queue , the accuracy of the fused positioning pose information is higher, which ensures the high precision of the fusion positioning method; different positioning methods can match different moving driving environments, ensuring that the repositioning pose information can be obtained no matter what the environment is. The stability and high robustness of the fusion positioning method are guaranteed.

FIG. 3 is a flowchart of an implementation of obtaining repositioning pose information in a fusion positioning method according to an embodiment of the present application. As shown in FIG. 3 , in some embodiments, the process of acquiring the repositioning pose information in the above step S110 includes:

Step 210: in the case of performing initial positioning, collect a front-view picture;

Step 220 : Based on the corners in the front-view picture, by matching with the corner map, obtain corner-based pose information, and use the corner-based pose information as the repositioning pose information.

Optionally, the initial positioning may be the first positioning during driving or moving, or the second positioning after a long period of time.

In one embodiment, when it is determined as initial positioning, a front-view image is collected, and corner points in the front-view image are extracted. Optionally, a Fast method can be used to extract corner points. The Fast method mainly focuses on The 16 pixel corner points on the circular window near the pixel point, as shown in Figure 4, p is the center pixel point, and the point pixel marked by the white box is the 16 pixel corner points that need to be extracted, and the Brief descriptor is used. Describes the extracted 16 pixel corners. Then, the extracted corners are matched with the corner map to obtain multiple candidate keyframes. For example, based on the Brief descriptor, the Bow dictionary is used to search for the four matching keyframes with the highest scores in the corner map. This step can also be called Perform brute force matching; after obtaining multiple candidate keyframes, select the optimal keyframe from them, and then determine whether the matching is successful based on the matching corners in the optimal keyframe, for example, select the corner with the highest score and a score higher than the threshold. Point the map frame, and use the Hamming distance to match the corner points in the map with the current corner points. When the minimum Hamming distance of the matching is less than the threshold, the matching is considered successful. Finally, based on the successfully matched corner map, the repositioning pose information is obtained. Specifically, because the corners included in the corner map have 3D positions, the 2D corner and corner map extracted from the front-view image can be obtained. The matching of the 3d corner points in the middle, and then solved by the pnp method, and finally the repositioning pose information of the current front-view image in the corner map coordinate system is obtained.

Optionally, due to errors in the above matching process, the repositioning pose information calculated by pnp may also have large errors, so a fundamental matrix can be established based on the above 2d corner points and compared with the angle vector calculated by pnp. , if the angle error is greater than the threshold, it is considered that the pose calculated by pnp is wrong, and the positioning fails; if the angle error is less than the threshold, the repositioning pose information is returned. As shown in FIG. 2 , the above-mentioned process of finally obtaining corner-based positioning based on the front-view picture can also be generally referred to as “image processing”. The above-mentioned positioning method based on corner points has the advantages of high precision and accurate positioning, so the positioning method based on corner points is selected in the initial positioning.

As shown in FIG. 3 , in some embodiments, the process of acquiring the repositioning pose information in the above step S110 further includes:

Step 230: in the case of performing non-initial positioning, collect a look-around picture;

Step 240: Extract the semantic features in the look-around picture, obtain semantic-based pose information by matching with the semantic map, and use the semantic-based pose information as the relocation pose information.

In a specific implementation, when initial positioning has been performed, non-initial positioning is performed, a 360° look-around picture is collected, and then semantic features are extracted from the 360° look-around image, and the semantic features may include lane lines, road edges , at least one of parking space points; if the extracted semantic feature is a parking space point, the position of the parking space point is described in the current Cartesian coordinate system; if the extracted semantic feature is a lane line or a road edge, then use Distances and angles in polar coordinates are described. For example, a straight line is represented by xcosθ+ysinθ-r=0, where θ is the angle of the vertical vector, r is the distance from the straight line to the origin, and the representation in polar coordinates is ρcos(θ-α)=r, where the coordinates of the point are (ρ, θ). The same filter equation can be used after converting the parking spot and the lane line (or road edge), where the gain calculation formula is as follows:

k=P' _n+1 H(HP' _n+1 H ^T +R) ^-1

Among them, P' _n+1 is the covariance of the vehicle state quantity n+1 time, R is the observation covariance, and H is the corresponding observation Jacobian matrix. The ratio matrices are as follows:

In the above parking spot observation Jacobian matrix,

Refers to the x-coordinate at time n in the map coordinate system (world coordinate system), and similarly

Refers to the y coordinate at time n in the map coordinate system (world coordinate system).

After comparing the description information obtained based on the above formula with the semantic map, the pose information based on semantic localization is obtained, as shown in FIG. 2 , the above process can also be summarized as “image processing”. Because the corner-based positioning method requires relatively rich scene textures, otherwise the positioning will fail, and the corner-based positioning method occupies a lot of resources in the process of corner map matching, and the calculation is time-consuming, while the semantic-based positioning method The method can make up for the above shortcomings to a certain extent. Therefore, when performing non-initial positioning, the semantic-based positioning method is preferred, which can save computing resources and obtain repositioning pose information faster.

In some embodiments, after the above step S240, the method further includes:

Step 250: Determine whether the look-around picture is valid;

Step 260: in the case of judging that the surround view picture is invalid, collect the front view picture;

Step 270: Based on the corners in the front-view picture, by matching with the corner map, obtain corner-based pose information, and use the corner-based pose information as the repositioning pose information.

In a specific embodiment, to determine whether the look-around picture is valid, as long as the semantic-based pose information cannot be obtained, the look-around picture is judged to be invalid, which may specifically include: whether semantic features can be extracted based on the look-around image, and if it cannot be extracted If the semantic features are extracted, the look-around picture is judged to be invalid; whether the extracted semantic features can be matched with the semantic map, if they cannot be matched, the look-around picture is judged to be invalid. After judging that the look-around picture is invalid, the front-view picture is collected, and based on the corners in the front-view picture, the corner-based pose information is obtained by matching with the corner map, and the corner-based pose information is used as the Relocate pose information. The specific steps of obtaining the pose information based on the corner points based on the corner points in the front-view picture are the same as the above step 220, and are not repeated here. The basis of semantic localization recognizes at least one of semantic features such as lane lines, road edges, and parking spots. If these features happen to be absent in the environment, or the recognized features cannot be matched with the semantic map, semantic-based localization cannot be performed. , at this time, it is necessary to replace the positioning based on corner points to ensure that continuous positioning can be provided during driving or moving, and relatively stable repositioning pose information can be obtained.

In some embodiments, as shown in FIG. 5, before the above step S110, it further includes:

Step 510: obtain the current plane position;

Step 520: According to the plane position, obtain a corner map and a semantic map within a limited radius with the plane position as the center.

In a specific implementation manner, to obtain the current plane position, the approximate plane position can be obtained through a positioning instrument such as GPS, or the approximate position can be obtained by manually clicking on the electronic map, or, after In the case of initial positioning, the plane position can be obtained based on the repositioned pose information of the initial positioning. According to the plane position, a corner map and a semantic map within a limited radius are obtained with the plane position as the center, and the limited radius can be set manually. Because whether it is corner-based or semantic-based positioning, the corresponding corner map or semantic map needs to be loaded. If all map data is loaded at the beginning, it will take up storage space and waste loading time. Therefore, a map within a certain range can be downloaded based on the approximate location, which not only saves time, but also reduces resource occupation.

In some embodiments, the fusion positioning method may further include:

Calculated based on the motion model to obtain multiple predicted pose information at different times;

The predicted pose information queue is generated based on the predicted pose information at the multiple different times.

For example, the motion model can be used to calculate the predicted pose information at different times. Specifically, the motion model is a motion model that can use the pose (including position coordinates and direction angle), calculation parameters and motion time t' at time _t1 . , the model of the vehicle pose (including position coordinates and direction angle) at time t ₁ +t' is calculated, as shown in Figure 2, the motion model can also be a vehicle motion model, and the calculation parameters can include wheel pulse messages and gears. Position message, based on wheel pulse message, gear position message, pose information at the last moment, and motion time from the last moment, the real-time predicted pose information can be obtained. Optionally, the predicted pose information may be calculated every fixed time, and a predicted information queue may be generated based on the predicted pose information at multiple different times. Optionally, if too much predicted pose information is included in the queue, some temporally earlier predicted pose information may be discarded based on its predicted positioning time. The positioning method based on the motion model is more commonly used, and the calculation is relatively simple, but the positioning accuracy is not high. Taking it as the basis and fusing it with the repositioning pose information, a more accurate positioning pose can be obtained; the prediction based on the motion model is used. The pose information is stored in the form of a queue, which can ensure that during fusion, at least one predicted pose information that is closest to the time corresponding to the repositioning pose information is selected for fusion, so as to obtain fused positioning pose information with higher accuracy.

In some embodiments, the fusion positioning method further includes:

According to the relocation time of the relocation pose information, obtain at least one predicted pose information before the relocation time from the predicted pose information queue as the first predicted pose information, and/or at least one piece after the relocation time One piece of predicted pose information is used as the second predicted pose information; specifically, the repositioning pose information includes the repositioning time, that is, the acquisition time of the repositioning pose information. Based on this, it is determined in the predicted pose information queue at The predicted pose information before this relocation time, and determine the predicted pose information after this relocation time, for example, the relocation time in the relocation pose information is AM 8:00, in the predicted pose information queue , find all the predicted pose information before AM 8:00, and find all the predicted pose information after AM 8:00 in the predicted pose information queue. Further, among the predicted pose information before the acquisition time of the repositioning pose information, select one as the first predicted pose information, optionally, select the one with the closest time as the first predicted pose information, such as There are multiple predicted pose information before AM 8:00, the time is AM 7:40, AM 7:50, AM 7:58, that is, the predicted pose information of AM 7:58 is used as the first predicted pose information ; Similarly, among the predicted pose information after the acquisition time of the repositioning pose information, select one as the second predicted pose information, optionally, also select the one with the closest time as the second predicted pose information , for example, there are multiple predicted pose information after AM 8:00, the time is AM 8:40, AM 8:50, AM 8:58, that is, the predicted pose information of AM 8:40 is used as the second predicted position posture information.

In a specific embodiment, based on the association and fusion of the repositioned pose information with the first predicted pose information and/or the second predicted pose information, the fused pose information can be divided into three types condition. In the first case, as shown in Fig. 2 and Fig. 6(a), the relocation time is between the predicted location times of multiple predicted pose information, so the first predicted pose information t ₁ before the relocation time can be determined. and the second predicted pose information t ₂ after the relocation time, the pose increments of t ₁ and t ₂ are δp(δxδyδθ), and the time difference between the relocation pose information t _i and t ₂ is δti, t ₁ and t _The _time _difference _between _the _{_} Among them, the update equation of covariance is:

Where f _xk can be calculated according to the function CaculateFxk(dxR, PoseTheta, dTheta), and pk can be obtained according to the function CaculatePk(vehicle_RR, vehicle_RL, PoseTheta, dTheta). Finally, EKF fusion is performed based on the recursive value of the above pose, the repositioned pose information t _i and the updated covariance, and the fused pose information is obtained.

The second case is that the time for relocating the pose information t _i is too early, before the predicted positioning time corresponding to all the predicted pose information in the predicted pose information queue, as shown in Figure 2 and Figure 6(b), at this time , obtain the predicted pose information t ₂ after the repositioning pose information, directly discard the repositioning pose information, and use t ₂ as the fused pose information. Optionally, t ₂ is also the earliest piece of predicted pose information in the predicted pose information queue.

The last case is that the time of relocating the pose information t _i is too late, after the predicted positioning time corresponding to all the predicted pose information in the predicted pose information queue, that is, only the first time before the relocation time can be determined. A predicted pose information t ₁ , as shown in Figure 2 and Figure 6(c), at this time, if the difference between the relocation time and the predicted positioning time of the first predicted pose information t ₁ is greater than a given threshold, then directly Discard the repositioning pose information t _i , and use the first predicted pose information t ₁ as the fused pose information; if the time difference between the repositioning time and the first predicted pose information t ₁ is less than or equal to a given threshold, then predict from In the pose information queue, a predicted pose information t' whose time is before the first predicted pose information is selected, and then combined with the first predicted pose information t ₁ and t ', the linear interpolation method is used to obtain the time t ₁ The recursive value of the pose is obtained. After the recursive value is obtained, the covariance is updated. The specific steps of calculating the recursive value and updating the covariance refer to the first case above. Finally, based on the recursive value of the above pose and repositioning the pose The information t _i and the updated covariance are EKF fused to obtain the fused pose information.

In some embodiments, updating the predicted pose information queue further includes: in the predicted pose information queue, the fused pose information corresponds to other predicted pose information after the time. Specifically, the updating step It involves using the motion model containing the wheel pulse message and the gear position message to obtain the updated predicted pose information queue. For example, after obtaining the fused pose information, find the predicted pose information in the queue of predicted pose information, all the predicted pose information whose predicted positioning time occurs after the corresponding time of the fused pose information, for example, when the predicted positioning time is AM 8: After the fused pose information of 40, insert it into the predicted pose queue, and then use the motion model to recalculate the predicted pose information at AM 8:50 and AM 8:58 in the predicted pose information queue. Optionally, the map point cloud can also be released according to the fused pose information.

In some embodiments, the above step S140 further includes:

Based on the updated predicted pose information queue, the target positioning pose information is obtained. Specifically, as shown in FIG. 2 , based on the updated predicted pose information queue, a newly added predicted pose information in the predicted pose information is found and released as the target positioning pose. Optionally, before publishing, the latest predicted pose information may also be stored together with the corresponding process noise, Jacobian matrix, and the like.

FIG. 7 shows a structural block diagram of a fusion positioning apparatus 700 according to an embodiment of the present application. As shown in Figure 7, the apparatus may include:

A relocation module 710, configured to collect pictures related to the surrounding environment, and obtain relocation pose information based on the pictures related to the surrounding environment;

A fusion module 720, configured to fuse at least one predicted pose information in the predicted pose information queue based on the repositioning pose information to obtain fused pose information; wherein the predicted pose information queue contains multiple Predicted pose information;

an update module 730, configured to update the predicted pose information queue based on the fused pose information;

The pose determination module 740 is configured to obtain target positioning pose information based on the updated predicted pose information queue.

In one embodiment, as shown in FIG. 8, the relocation module 710 includes:

The first front-view picture acquisition unit 711 is configured to acquire a front-view picture under the condition of initial positioning;

The first corner locating unit 712 is configured to obtain corner-based pose information by matching with the corner map based on the corners in the front-view picture, and use the corner-based pose information as the relocation. pose information.

In one embodiment, as shown in FIG. 8 , the relocation module 710 further includes:

A look-around picture acquisition unit 713, configured to collect a look-around picture in the case of performing non-initial positioning;

The semantic positioning unit 714 is configured to extract the semantic features in the look-around picture, obtain semantic-based pose information by matching with the semantic map, and use the semantic-based pose information as the relocation pose information.

Judging unit 715, for judging whether the look-around picture is valid;

The second front-view picture acquisition unit 716 is configured to acquire a front-view picture when it is judged that the surround-view picture is invalid;

The second corner locating unit 717 is configured to obtain corner-based pose information by matching with the corner map based on the corners in the front-view picture, and use the corner-based pose information as the relocation pose information.

In an embodiment, as shown in FIG. 10 , the fusion positioning apparatus 700 further includes:

a position obtaining module 750, used for obtaining the current plane position;

The map loading module 760 is configured to obtain, according to the plane position, a corner point map and a semantic map within a limited radius with the plane position as the center of the circle.

In one embodiment, the fusion positioning apparatus 700 further includes:

In one embodiment, as shown in FIG. 11 , the fusion module 720 further includes:

The predicted pose information selection unit 721 is configured to obtain at least one predicted pose information before the relocation time from the predicted pose information queue as the first predicted pose information according to the relocation time of the relocated pose information, And/or at least one predicted pose information after the relocation time is used as the second predicted pose information;

The predicted pose information fusion unit 722 is configured to fuse the first predicted pose information and/or the second predicted pose information based on the repositioned pose information to obtain fused pose information.

In one embodiment, the update module 730 further includes:

The updating queue unit 731 is configured to update other predicted pose information after the corresponding time in the predicted pose information queue, the fused pose information, to obtain an updated predicted pose information queue.

In one embodiment, the pose determination module 740 includes:

The obtaining unit 741 is configured to predict the pose information based on the target to obtain the target positioning pose information.

For the functions of each module in each device in this embodiment of the present application, reference may be made to the corresponding description in the foregoing method, and details are not described herein again.

It should be noted that although the fusion positioning method and device are described above, those skilled in the art can understand that the present application should not be limited thereto. In fact, the user can flexibly set the fusion positioning method and device according to personal preferences and/or actual application scenarios. For example, the repositioning method may not be limited to corner-based positioning or semantic-based positioning, and the predicted pose information is not limited to For the motion model, the rest of the positioning models can be adopted, as long as high-precision and robust positioning can be finally obtained.

In this way, through fusion positioning, the fusion positioning method and device according to the above embodiments of the present application can combine multiple positioning methods to obtain a more accurate and stable positioning result.

FIG. 9 shows a structural block diagram of an electronic device according to an embodiment of the present application. As shown in FIG. 9 , the electronic device includes: a memory 910 and a processor 920 , and instructions that can be executed on the processor 920 are stored in the memory 910 . When the processor 920 executes the instruction, the fusion positioning method in the foregoing embodiment is implemented. The number of the memory 910 and the processor 920 may be one or more. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the application described and/or claimed herein.

The electronic device may further include a communication interface 930 for communicating with external devices and performing interactive data transmission. The various devices are interconnected using different buses and can be mounted on a common motherboard or otherwise as desired. The processor 920 may process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In other embodiments, multiple processors and/or multiple buses may be used with multiple memories and multiple memories, if desired. Likewise, multiple electronic devices may be connected, each providing some of the necessary operations (eg, as a server array, a group of blade servers, or a multiprocessor system). The bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used in FIG. 9, but it does not mean that there is only one bus or one type of bus.

Optionally, in specific implementation, if the memory 910, the processor 920 and the communication interface 930 are integrated on one chip, the memory 910, the processor 920 and the communication interface 930 can communicate with each other through an internal interface.

It should be understood that the above-mentioned processor may be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processing, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), Field Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or any conventional processor or the like. It should be noted that the processor may be a processor supporting an advanced RISC machine (ARM) architecture.

Embodiments of the present application provide a computer-readable storage medium (such as the above-mentioned memory 910 ), which stores computer instructions, and when the program is executed by a processor, implements the methods provided in the embodiments of the present application.

Optionally, the memory 910 may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required by at least one function; data etc. Additionally, memory 910 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 910 may optionally include memory located remotely from processor 920, and these remote memories may be connected to the fused location electronic device via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present application. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present application, "plurality" means two or more, unless otherwise expressly and specifically defined.

Any description of a process or method in a flowchart or otherwise described herein may be understood to represent a representation of executable instructions comprising one or more (two or more) steps for implementing a specified logical function or process. A module, fragment or section of code. Also, the scope of the preferred embodiments of the present application includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved.

The logic and/or steps represented in flowcharts or otherwise described herein, for example, may be considered an ordered listing of executable instructions for implementing the logical functions, may be embodied in any computer-readable medium, For use with, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions from an instruction execution system, apparatus, or apparatus) or equipment.

It should be understood that various parts of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the method in the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or its combination.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the above-mentioned integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various changes or replacements within the technical scope disclosed in the present application. , which should be covered within the scope of protection of this application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

A fusion positioning method, characterized in that the method comprises:

Collect pictures related to the surrounding environment, and obtain repositioning pose information based on the pictures related to the surrounding environment;

At least one predicted pose information in the predicted pose information queue is fused based on the repositioned pose information to obtain the fused pose information; wherein the predicted pose information queue includes multiple predicted pose information ;

updating the predicted pose information queue based on the fused pose information;

Based on the updated predicted pose information queue, target positioning pose information is obtained.
The fusion positioning method according to claim 1, wherein collecting pictures related to the surrounding environment, and obtaining relocation pose information based on the pictures related to the surrounding environment, comprising:

In the case of initial positioning, the front-view picture is collected;

Based on the corners in the front-view picture, the corner-based pose information is obtained by matching with the corner map, and the corner-based pose information is used as the repositioning pose information.
The fusion positioning method according to claim 1, wherein collecting pictures related to the surrounding environment, and obtaining relocation pose information based on the pictures related to the surrounding environment, comprising:

In the case of non-initial positioning, collect look-around pictures;

Semantic features in the look-around picture are extracted, and semantic-based pose information is obtained by matching with a semantic map, and the semantic-based pose information is used as the relocation pose information.
The fusion positioning method according to claim 3, the method further comprises:

determine whether the look-around picture is valid;

In the case of judging that the look-around picture is invalid, collect the front-view picture;

Based on the corners in the front-view picture, the corner-based pose information is obtained by matching with the corner map, and the corner-based pose information is used as the repositioning pose information.
The fusion positioning method according to any one of claims 2-4, wherein the method further comprises:

Get the current plane position;

According to the plane position, a corner point map and a semantic map within a limited radius are obtained with the plane position as the center.
The fusion positioning method according to claim 1, wherein the method further comprises:

Calculated based on the motion model to obtain multiple predicted pose information at different times;

The predicted pose information queue is generated based on the predicted pose information at the multiple different times.
fusion positioning method according to claim 1, it is characterised in that, based on the repositioning position information At least one predicted position information in the predicted position information queue is fused to obtain the fused position information, including:

According to the relocation time of the relocation pose information, at least one piece of predicted pose information before the relocation time is obtained from the predicted pose information queue as the first predicted pose information, and/or the relocation at least one predicted pose information after time is used as the second predicted pose information;

Based on the relocation pose information and the first predicted pose information and/or the second predicted pose information, the fused pose information is obtained.
The method according to claim 7, wherein the updating the predicted pose information queue further comprises:

In the predicted pose information queue, the fused pose information corresponding to other predicted pose information after the time is updated to obtain an updated predicted pose information queue.
The fusion positioning method according to claim 1, wherein, obtaining target positioning pose information based on the updated predicted pose information queue, comprising:

The newly generated predicted pose information in the updated predicted pose information queue is used as the target positioning pose information.
A fusion positioning device, characterized in that the device comprises:

a relocation module, configured to collect pictures related to the surrounding environment, and obtain relocation pose information based on the pictures related to the surrounding environment;

The fusion module is configured to fuse at least one predicted pose information in the predicted pose information queue based on the repositioning pose information to obtain the fused pose information; wherein the predicted pose information queue contains multiple A predicted pose information;

an update module, configured to update the predicted pose information queue based on the fused pose information;

The pose determination module is configured to obtain target positioning pose information based on the updated predicted pose information queue.
The apparatus according to claim 10, wherein the relocation module comprises:

a first front-view picture acquisition unit, configured to acquire a front-view picture under the condition of initial positioning;

The first corner point positioning unit is used to obtain corner point-based pose information by matching with the corner point map based on the corner points in the front-view picture, and using the corner point-based pose information as the Relocate pose information.
The apparatus according to claim 10, wherein the relocation module comprises:

A look-around picture acquisition unit, used to collect a look-around picture in the case of non-initial positioning;

The semantic positioning unit is used to extract the semantic features in the look-around picture, obtain semantic-based pose information by matching with the semantic map, and use the semantic-based pose information as the relocation pose information.
The apparatus according to claim 12, wherein the relocation module further comprises:

a judging unit for judging whether the look-around picture is valid;

a second front-view picture acquisition unit, configured to acquire a front-view picture when it is judged that the surround-view picture is invalid;

The second corner point positioning unit is configured to obtain corner point-based pose information by matching with the corner point map based on the corner points in the front-view picture, and use the corner point-based pose information as the Relocate pose information.
The device according to any one of claims 11-13, wherein the device further comprises:

The position obtaining module is used to obtain the current plane position;

The map loading module is configured to obtain, according to the plane position, a corner point map and a semantic map within a limited radius with the plane position as the center.
The apparatus of claim 10, wherein the apparatus further comprises:

Calculated based on the motion model to obtain multiple predicted pose information at different times;

The predicted pose information queue is generated based on the predicted pose information at the multiple different times.
The device according to claim 10, wherein the fusion module further comprises:

A predicted pose information selection unit, configured to obtain at least one predicted pose information before the relocation time from the predicted pose information queue as the first predicted pose according to the relocation time of the relocated pose information information, and/or at least one predicted pose information after the relocation time as the second predicted pose information;

A predicted pose information fusion unit, configured to fuse the first predicted pose information and/or the second predicted pose information based on the repositioned pose information to obtain the fused pose information.
The apparatus according to claim 16, wherein the update module further comprises:

The updating queue unit is configured to update other predicted pose information after the corresponding time of the fused pose information in the predicted pose information queue to obtain an updated predicted pose information queue.
The device according to claim 10, wherein the pose determination module further comprises:

The obtaining unit is configured to use the newly generated predicted pose information in the updated predicted pose information queue as the target positioning pose information.
An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the execution of any of claims 1-9 Methods.
A computer-readable storage medium having computer instructions stored therein, the computer instructions implementing the method according to any one of claims 1-9 when executed by a processor.