CN112432637B

CN112432637B - Positioning method and device, electronic equipment and storage medium

Info

Publication number: CN112432637B
Application number: CN202011379583.5A
Authority: CN
Inventors: 谢卫健; 钱权浩; 王楠; 章国锋; 鲍虎军
Original assignee: Zhejiang Shangtang Technology Development Co Ltd
Current assignee: Zhejiang Shangtang Technology Development Co Ltd
Priority date: 2020-11-30
Filing date: 2020-11-30
Publication date: 2023-04-07
Anticipated expiration: 2040-11-30
Also published as: CN112432637A; WO2022110776A1; JP7487321B2; KR20230086782A; JP2023514724A

Abstract

The disclosure relates to a positioning method and apparatus, an electronic device and a storage medium, wherein the method is applied to a first electronic device and comprises the following steps: sending a kth first positioning request to second electronic equipment, wherein the kth first positioning request comprises a kth frame environment image of the environment where the first electronic equipment is located; under the condition that a kth first positioning result sent by second electronic equipment is received, correcting the kth second positioning result according to the kth first positioning result, and determining a kth third positioning result of the first electronic equipment, wherein the kth second positioning result is obtained by the first electronic equipment through local positioning on a kth frame environment image; and displaying the display interface of the first electronic equipment according to the kth third positioning result. The embodiment of the disclosure can improve the positioning precision and stability.

Description

Positioning method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a positioning method and apparatus, an electronic device, and a storage medium.

Background

When people move indoors and outdoors (such as inside a large shopping mall, on a city road, etc.), the people often need to determine their own position by positioning, go to a destination by navigation, and the like. In the related art, under the condition that terminal positioning is realized in a visual positioning mode, a set of synchronous positioning And Mapping (SLAM for short) system can be locally operated on a terminal, and the position And attitude information of a local SLAM can be directly corrected through a positioning result of a cloud. The method can determine the absolute position of the terminal, but cannot avoid relative errors generated in the system operation process, and jitter, drifting and the like can be generated in some scenes, so that the positioning effect is poor.

Disclosure of Invention

The present disclosure provides a positioning technical solution.

According to an aspect of the present disclosure, there is provided a positioning method applied to a first electronic device, the method including:

sending a kth first positioning request to second electronic equipment, wherein the kth first positioning request comprises a kth frame environment image of an environment where the first electronic equipment is located, and k is an integer greater than 1; under the condition that a kth first positioning result sent by the second electronic device is received, correcting the kth second positioning result according to the kth first positioning result, and determining a kth third positioning result of the first electronic device, wherein the kth second positioning result is obtained by the first electronic device performing local positioning on the kth frame environment image; and displaying a display interface of the first electronic device according to the kth third positioning result.

In a possible implementation manner, the determining a kth positioning result of the first electronic device by correcting the kth second positioning result according to the kth first positioning result includes determining the kth first positioning result by using the kth first positioning result, where the determining the kth first positioning result includes: determining the current pose deviation of the kth positioning according to the kth first pose information and the kth second pose information; determining the total pose deviation of the k times of positioning according to the k second pose information and the k third pose information of the k times of positioning; and correcting the pose according to the current pose deviation of the kth positioning and the total pose deviation of the kth positioning, and determining the kth third pose information.

In one possible implementation manner, determining an overall pose deviation of the k times of positioning according to the k second pose information of the k times of positioning and the k third positioning results includes: determining first position and posture change information between the ith second position and posture information, wherein i and j are integers, and i is more than or equal to 1 and is more than or equal to j and less than or equal to k; determining second posture change information between the ith third posture information and the jth third posture information; determining the position and orientation deviation from the ith frame to the jth frame according to the first position and orientation change information and the second position and orientation change information; and determining the overall pose deviation of the k times of positioning according to the pose deviations.

In a possible implementation manner, the kth first positioning result further includes feature point matching information of a kth frame, where the feature point matching information of the kth frame includes matching information between a two-dimensional feature point in an environmental image of the kth frame and a three-dimensional feature point in a point cloud map of a second electronic device, and the kth second positioning result is corrected according to the kth first positioning result to determine a kth third positioning result of the first electronic device, further including: and correcting the kth third pose information according to the feature point matching information of the kth frame, and determining a corrected kth third positioning result.

In a possible implementation manner, the determining a kth third positioning result by correcting the kth second positioning result according to the kth first positioning result includes: and correcting the kth second positioning result according to the feature point matching information of the kth frame, and determining a corrected kth third positioning result.

In one possible implementation, the method further includes: sending an a first positioning request to second electronic equipment, wherein the a first positioning request comprises an a frame environment image of the environment where the first electronic equipment is located, a is an integer and a is more than or equal to 1; under the condition of receiving the a-th first positioning result sent by the second electronic device, initializing a synchronous positioning and mapping SLAM system in the first electronic device according to the a-th first positioning result, and determining an initial positioning result of the first electronic device, wherein the a-th first positioning result comprises a-th first positioning posture information and/or a-th frame feature point matching information.

In a possible implementation manner, the kth first positioning result sent by the second electronic device further includes a first area map, the first area map includes a point cloud sub-map of a geographic area corresponding to the first electronic device,

after determining the kth third positioning result, the method further comprises: performing projection matching on the kth third positioning result and a point cloud sub-map in the first area map, and determining a fourth positioning result of the first electronic device; and displaying a display interface of the first electronic equipment according to the fourth positioning result.

In one possible implementation manner, before sending the kth first positioning request to the second electronic device, the method further includes: judging whether the first electronic equipment is located in a second regional map according to a kth second positioning result of the first electronic equipment and the local second regional map of the first electronic equipment, wherein the second regional map is a regional map sent by the second electronic equipment;

wherein the sending the kth first positioning request to the second electronic device includes: and sending a kth first positioning request to a second electronic device under the condition that the first electronic device is in the second regional map.

In one possible implementation, the method further includes: under the condition that the first electronic equipment is not located in the second regional map, sending a second positioning request to the second electronic equipment, wherein the second positioning request comprises a kth frame environment image of the environment where the first electronic equipment is located; under the condition that a third area map sent by the second electronic device is received, performing projection matching on a current second positioning result and a point cloud sub-map in the third area map, and determining a fifth positioning result of the first electronic device, wherein the third area map comprises the point cloud sub-map of a geographic area corresponding to the first electronic device; and displaying a display interface of the first electronic equipment according to the fifth positioning result.

In one possible implementation, before sending the first positioning request to the second electronic device, the method further includes: acquiring a network state of the first electronic equipment, wherein the network state comprises network signal strength and/or information transmission speed; wherein the sending the kth first positioning request to the second electronic device includes:

and sending a kth first positioning request to second electronic equipment under the condition that the network state does not meet preset regional map acquisition conditions, wherein the regional map acquisition conditions comprise: the network signal strength is greater than or equal to a signal strength threshold; and/or the information transmission speed is greater than or equal to a speed threshold.

According to an aspect of the present disclosure, there is provided a positioning method applied to a second electronic device, the method including:

under the condition that a kth first positioning request from first electronic equipment is received, carrying out visual positioning on the first electronic equipment according to a preset point cloud map and a kth frame environment image in the kth first positioning request to obtain a kth first positioning result of the first electronic equipment; and sending the kth first positioning result to the first electronic equipment so that the first electronic equipment corrects the positioning result according to the kth first positioning result, wherein the kth first positioning result comprises kth first pose information and/or characteristic point matching information of a kth frame, and the characteristic point matching information of the kth frame comprises matching information between two-dimensional characteristic points in the kth frame environment image and three-dimensional characteristic points in the point cloud map.

In one possible implementation, the method further includes: under the condition that a second positioning request from first electronic equipment is received, carrying out visual positioning on the first electronic equipment according to the point cloud map and an environment image in the second positioning request to obtain a sixth positioning result of the first electronic equipment; according to the sixth positioning result, a point cloud sub-map of a geographic area corresponding to the first electronic device is determined from the point cloud map; and sending a regional map to the first electronic equipment so that the first electronic equipment can be positioned and displayed according to the regional map, wherein the regional map comprises a point cloud sub-map of a geographic region corresponding to the first electronic equipment.

According to an aspect of the present disclosure, there is provided a positioning apparatus including: applied to a first electronic device, the apparatus comprises:

a first request sending module, configured to send a kth first positioning request to a second electronic device, where the kth first positioning request includes a kth frame environment image of an environment where the first electronic device is located, and k is an integer greater than 1;

a result correction module, configured to correct a kth second positioning result according to a kth first positioning result sent by the second electronic device when the kth first positioning result is received, and determine a kth third positioning result of the first electronic device, where the kth second positioning result is obtained by the first electronic device performing local positioning on the kth frame of environment image;

and the first display module is used for displaying the display interface of the first electronic device according to the kth third positioning result.

In a possible implementation manner, the kth first positioning result includes kth first position information, the kth second positioning result includes kth second position information, the kth third positioning result includes kth third position information, and the result correction module includes:

the current deviation determining submodule is used for determining the current pose deviation of the kth positioning according to the kth first pose information and the kth second pose information; the overall deviation determining submodule is used for determining the overall pose deviation of k times of positioning according to k second pose information and k third pose information of k times of positioning; and the first correction submodule is used for correcting the pose according to the current pose deviation of the kth positioning and the total pose deviation of the kth positioning, and determining the kth third pose information.

In one possible implementation, the overall deviation determining submodule is configured to: determining first position change information between the ith second position information and the jth second position information, wherein i and j are integers, and i is more than or equal to 1 and less than or equal to j and less than or equal to k; determining second posture change information between the ith third posture information and the jth third posture information; determining pose deviation from the ith frame to the jth frame according to the first pose change information and the second pose change information; and determining the overall pose deviation of the k times of positioning according to the pose deviations.

In a possible implementation manner, the kth first positioning result further includes feature point matching information of the kth frame, where the feature point matching information of the kth frame includes matching information between a two-dimensional feature point in an environmental image of the kth frame and a three-dimensional feature point in a point cloud map of the second electronic device, and the result modification module further includes: and the second correction submodule is used for correcting the kth third pose information according to the feature point matching information of the kth frame and determining a corrected kth third positioning result.

In one possible implementation manner, the kth first positioning result includes feature point matching information of a kth frame, the feature point matching information of the kth frame includes matching information between a two-dimensional feature point in an environment image of the kth frame and a three-dimensional feature point in a point cloud map of the second electronic device, and the result modification module includes: and the third correction submodule is used for correcting the kth second positioning result according to the feature point matching information of the kth frame and determining a corrected kth third positioning result.

In one possible implementation, the apparatus further includes: the second request sending module is used for sending an a-th first positioning request to second electronic equipment, wherein the a-th first positioning request comprises an a-th frame environment image of the environment where the first electronic equipment is located, a is an integer and is not less than 1; the initialization module is used for initializing a synchronous positioning and mapping SLAM system in the first electronic equipment according to the a-th first positioning result and determining the initial positioning result of the first electronic equipment under the condition of receiving the a-th first positioning result sent by the second electronic equipment, wherein the a-th first positioning result comprises a-th first positioning posture information and/or feature point matching information of a-th frame.

after the result modification module, the apparatus further comprises: the first matching positioning module is used for performing projection matching on the kth third positioning result and a point cloud sub-map in the first area map to determine a fourth positioning result of the first electronic device; and the second display module is used for displaying the display interface of the first electronic equipment according to the fourth positioning result.

In a possible implementation manner, before the first request sending module, the apparatus further includes:

the area judgment module is used for judging whether the first electronic equipment is located in a second area map according to a kth second positioning result of the first electronic equipment and the local second area map of the first electronic equipment, wherein the second area map is an area map sent by the second electronic equipment; wherein the first request sending module comprises: the first sending submodule is used for sending a kth first positioning request to the second electronic equipment under the condition that the first electronic equipment is located in the second regional map.

In one possible implementation, the apparatus further includes: a third request sending module, configured to send a second positioning request to the second electronic device when the first electronic device is not located in the second regional map, where the second positioning request includes a kth frame environment image of an environment where the first electronic device is located; the second matching positioning module is used for performing projection matching on a current second positioning result and a point cloud sub-map in a third area map under the condition that the third area map sent by the second electronic device is received, and determining a fifth positioning result of the first electronic device, wherein the third area map comprises the point cloud sub-map of a geographic area corresponding to the first electronic device; and the third display module is used for displaying the display interface of the first electronic equipment according to the fifth positioning result.

In a possible implementation manner, before the first request sending module, the apparatus further includes: the network state acquisition module is used for acquiring a network state of the first electronic equipment, wherein the network state comprises network signal intensity and/or information transmission speed; wherein the first request sending module comprises: the second sending submodule is configured to send a kth first positioning request to the second electronic device when the network state does not meet a preset regional map obtaining condition, where the regional map obtaining condition includes: the network signal strength is greater than or equal to a signal strength threshold; and/or the information transmission speed is greater than or equal to a speed threshold.

According to an aspect of the present disclosure, there is provided a positioning apparatus applied to a second electronic device, the apparatus including:

the first positioning module is used for carrying out visual positioning on first electronic equipment according to a preset point cloud map and a kth frame environment image in a kth first positioning request under the condition that the kth first positioning request from the first electronic equipment is received, so as to obtain a kth first positioning result of the first electronic equipment;

a result sending module, configured to send the kth first positioning result to the first electronic device, so that the first electronic device performs positioning result correction according to the kth first positioning result,

the kth first positioning result comprises kth first pose information and/or characteristic point matching information of a kth frame, and the characteristic point matching information of the kth frame comprises matching information between two-dimensional characteristic points in the kth frame environment image and three-dimensional characteristic points in the point cloud map.

In one possible implementation, the apparatus further includes: the second positioning module is used for carrying out visual positioning on the first electronic equipment according to the point cloud map and the environment image in the second positioning request under the condition of receiving the second positioning request from the first electronic equipment to obtain a sixth positioning result of the first electronic equipment; the sub-map determining module is used for determining a point cloud sub-map of a geographic area corresponding to the first electronic device from the point cloud map according to the sixth positioning result; the regional map sending module is used for sending a regional map to the first electronic equipment so that the first electronic equipment can be positioned and displayed according to the regional map, and the regional map comprises a point cloud sub-map of a geographic region corresponding to the first electronic equipment.

According to an aspect of the present disclosure, there is provided an electronic device including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the memory-stored instructions to perform the above-described method.

According to an aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the above-described method.

In an embodiment of the present disclosure, a positioning request including an environment image can be transmitted to a second electronic device; correcting a local positioning result according to a cloud positioning result returned by the second electronic equipment; and the positioning result after correction is displayed, so that the positioning precision and stability are improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

Fig. 1 shows a flow chart of a positioning method according to an embodiment of the present disclosure.

Fig. 2 shows a flow chart of a positioning method according to an embodiment of the present disclosure.

Fig. 3 shows an interaction diagram of a positioning method according to an embodiment of the present disclosure.

Fig. 4 shows a schematic diagram of a process of a positioning method according to an embodiment of the present disclosure.

Fig. 5 shows a block diagram of a positioning device according to an embodiment of the present disclosure.

FIG. 6 shows a block diagram of a positioning device according to an embodiment of the present disclosure.

FIG. 7 shows a block diagram of an electronic device in accordance with an embodiment of the disclosure.

FIG. 8 shows a block diagram of an electronic device in accordance with an embodiment of the disclosure.

Detailed Description

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of a, B, and C, and may mean including any one or more elements selected from the group consisting of a, B, and C.

Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the present disclosure.

The positioning method according to the embodiment of the disclosure can be applied to indoor and outdoor scenes such as large malls, transportation hubs, hospitals and large exhibition halls, and the positioning accuracy and stability are improved. The positioning method can be implemented by a first electronic device and a second electronic device. The first electronic device may, for example, comprise a terminal device and the second electronic device may, for example, comprise a cloud server.

Fig. 1 shows a flow chart of a positioning method according to an embodiment of the present disclosure, as shown in fig. 1, the positioning method includes:

in step S11, a kth first positioning request is sent to a second electronic device, where the kth first positioning request includes a kth frame environment image of an environment where the first electronic device is located, and k is an integer greater than 1;

in step S12, when a kth first positioning result sent by the second electronic device is received, modifying a kth second positioning result according to the kth first positioning result, and determining a kth third positioning result of the first electronic device, where the kth second positioning result is obtained by the first electronic device performing local positioning on the kth frame of environment image;

in step S13, a display interface of the first electronic device is displayed according to the kth third positioning result.

In one possible implementation, the positioning method may be applied to a first electronic device, which may be a terminal device, such as a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA) device, a handheld device, a computing device, an in-vehicle device, a wearable device, or the like, and the method may be implemented by a processor calling a computer readable instruction stored in a memory.

For example, relative errors may be generated during operation of the SLAM system local to the first electronic device, such as cumulative errors generated by operation of the SLAM itself, jitter, wander in weak texture scenes, and so on. In this case, the local SLAM system can be constrained through the positioning information of the cloud, so that the accumulated error at the back end of the SLAM system is suppressed.

In a possible implementation manner, when a user holding or wearing the first electronic device needs to determine the position of the user, an environmental image of an environment where the user is located may be acquired by an acquisition component (e.g., a camera) of the first electronic device, for example, an image of a scene faced by the first electronic device is captured. The environmental image may be one or more images, or may be a short video including a plurality of frames of images, which is not limited in this disclosure.

In one possible implementation manner, for a certain positioning (set to be the kth positioning, where k is an integer greater than 1), in step S11, the first electronic device may send a kth first positioning request to the second electronic device so as to determine its location. The kth first positioning request comprises a kth frame of environment image acquired by the first electronic equipment. The second electronic device may be, for example, a cloud server, and stores a point cloud map of an entire geographic area (e.g., a mall internal area, a city area, etc.) where the first electronic device is located.

In a possible implementation manner, after receiving the kth first positioning request, the second electronic device may extract feature information of the kth frame environment image in the kth first positioning request. Feature extraction can be performed on the kth frame of environmental image through a pre-trained neural network, for example, so as to obtain feature information of the kth frame of environmental image. The present disclosure does not limit the specific manner of feature extraction.

In one possible implementation, after obtaining the feature information of the kth frame of environment image, the second electronic device may match the feature information with the point cloud map, and determine a matching visual positioning result (which may be referred to as a kth first positioning result). The present disclosure does not limit the specific manner in which the feature information is matched with the point cloud map.

In a possible implementation manner, the kth first positioning result includes pose information (may be abbreviated as position) and/or feature point matching information (may be abbreviated as match) of the first electronic device. The pose information comprises position information and attitude information, and the position information can comprise position coordinates of the first electronic equipment; the attitude information may include an orientation, a pitch angle, etc. of the first electronic device.

In one possible implementation, the feature point matching information may include matching information between two-dimensional feature points in the k-th frame environment image and three-dimensional feature points in the point cloud map. For example, N +1 2D feature points (x) of the k-th frame environment image ₀ ，x ₁ ，x ₂ ，…，x _N ) And N +1 3D feature points (X) of point cloud map ₀ ，X ₁ ，X ₂ ，…，X _N ) Respectively matching, the feature point matching information includes 2D feature point (x) ₀ ，x ₁ ，x ₂ ，…，x _N ) Coordinates of (2), 3D feature points (X) ₀ ，X ₁ ，X ₂ ，…，X _N ) And matching relation between them, N is an integer greater than 0.

In one possible implementation, the second electronic device may send the k first positioning results to the first electronic device so as to constrain the positioning results of the first electronic device.

In a possible implementation manner, in step S12, when the first electronic device receives the kth first positioning result sent by the second electronic device, the kth second positioning result may be corrected according to the kth first positioning result, and the kth third positioning result of the first electronic device is determined. That is, the local kth second positioning result of the first electronic device is constrained by pose information and/or feature point matching information in the kth first positioning result, so that the kth second positioning result is corrected, and a corrected positioning result (which may be referred to as a kth third positioning result) is obtained.

The kth second positioning result is obtained by the first electronic device performing local positioning on the kth frame of environment image, that is, performing visual positioning according to the kth frame of environment image and the local map to obtain a local positioning result, that is, the kth second positioning result, where the positioning result may include pose information. The present disclosure is not limited to a particular manner of positioning.

In a possible implementation manner, a Pose Graph (position Graph) can be established, for example, through Pose information in the kth first positioning result, pose information in the local kth second positioning result, and Pose information of the preceding k-1 second positioning results and third positioning results, so as to constrain the local positioning results; the local positioning result can be subjected to reprojection error constraint through feature point matching information in the kth first positioning result; and respectively constraining the local positioning result through the pose information and the feature point matching information in the kth first positioning result. The present disclosure is not limited in the specific manner of constraints.

In a possible implementation manner, in step S13, according to the kth third positioning result, a display interface of the screen of the first electronic device may be displayed. That is, the output result of the VIO (visual-inertial odometer) module of the local SLAM system may be constrained by the kth third positioning result. And according to the current application scene of the first electronic equipment, rendering corresponding content to be displayed through an output result of the VIO module, and displaying in a display interface.

For example, in an application scene of visual navigation, a navigation path can be determined and displayed in a display interface according to a first positioning result and a destination position, so that a navigation function is realized; under the application scene of virtual object display, the position and the posture of the virtual object to be displayed can be determined according to the first positioning result, and the virtual object can be displayed in the display interface, so that accurate and stable virtual object display is realized. The present disclosure is not limited to the specific content presented in the display interface.

According to an embodiment of the present disclosure, a positioning request including an environment image can be transmitted to a second electronic device; correcting a local positioning result according to a cloud positioning result returned by the second electronic equipment; and the positioning result after correction is displayed, so that the positioning precision and stability are improved.

In a possible implementation manner, the kth first positioning result includes kth first pose information, the kth second positioning result includes kth second pose information, and the kth third positioning result includes kth third pose information, and step S12 may include:

determining the current pose deviation of the kth positioning according to the kth first pose information and the kth second pose information;

determining the total pose deviation of the k times of positioning according to the k times of second pose information and the k times of third pose information of the k times of positioning;

and correcting the pose according to the current pose deviation of the kth positioning and the overall pose deviation of the kth positioning, and determining the kth third pose information.

For example, under the condition that the cloud positioning result includes pose information, a pose graph can be established according to the pose information of the current positioning and the pose information of the historical positioning, a local Map (Map) is restrained, and correction of the local pose is achieved. Among them, the Pose Graph (position Graph) is generally used to correct the accumulated error when Loop (Loop) occurs, and thus may also be applied to correct the accumulated error when positioning is successful.

In one possible implementation, the first position information is determined according to the kth position

And the kth second posture information T _k The current pose deviation of the kth fix, i.e., the deviation between the cloud pose and the local pose of the current fix, may be determined and expressed as £ er>

Determining the total pose deviation of the k times of positioning, namely the total deviation between the poses of the multiple times of positioning according to the k second pose information and the k third pose information of the k times of positioning; furthermore, a pose graph can be established according to the current pose deviation and the overall pose deviation, and pose correction is realized through constraint.

In a possible implementation manner, the step of determining the overall pose deviation of k times of positioning according to k second pose information and k third pose information of k times of positioning may include:

determining first position change information between the ith second position information and the jth second position information, wherein i and j are integers, and i is more than or equal to 1 and less than or equal to j and less than or equal to k;

determining second posture change information between the ith third posture information and the jth third posture information;

determining pose deviation from the ith frame to the jth frame according to the first pose change information and the second pose change information;

and determining the overall pose deviation of the k times of positioning according to the pose deviations.

For example, for any two times of k positioning (i, j are integer and 1 ≦ i)<j is less than or equal to k), the ith second position information T can be determined _i And j-th second attitude information T _j First posture change information T therebetween _ij And is used for representing the pose change of the local positioning result from the ith frame to the jth frame, wherein T _ij ＝T _j *T _i ^-1 (ii) a The ith third posture information T can also be determined _i ' with jth third posture information T _j ' second position variation information T ' therebetween ' _ij Pose change for representing the third positioning result of the ith to jth frames, wherein T' _ij ＝T′ _j *T _i ′ ^-1 (ii) a According to the first position posture change information and the second position posture change information, the position posture deviation from the ith frame to the jth frame can be determined and expressed as

Summing the multiple pose deviations to obtain a total pose deviation of k positioning times, and representing the total pose deviation as->

In one possible implementation, the pose graph E can be represented by the following formula ₁ ：

Optimizing and solving the formula (1) to obtain globally optimized T' _k That is, the kth third pose information is obtained after pose correction.

By the mode, the current local pose can be corrected according to the current cloud pose and the previous historical pose, so that the accumulated error of the SLAM system is reduced, and the positioning accuracy and stability are improved.

In a possible implementation manner, the kth first positioning result includes feature point matching information of the kth frame, the feature point matching information of the kth frame includes matching information between a two-dimensional feature point in an environment image of the kth frame and a three-dimensional feature point in a point cloud map of the second electronic device,

the correcting the kth second positioning result according to the kth first positioning result, and determining the kth third positioning result of the first electronic device, includes:

and correcting the kth second positioning result according to the feature point matching information of the kth frame, and determining a corrected kth third positioning result.

For example, when the cloud positioning result includes feature point matching information, the reprojection error constraint may be performed in Local constraint adjustment (LBA) according to the feature point matching information, and the current Local pose, that is, the kth second pose information T, may be corrected _k 。

As described above, it is assumed that the feature point matching information includes the 2D feature point (x) of the kth frame environment image ₀ ，x ₁ ，x ₂ ，…，x _N ) Coordinates of (2), 3D feature points (X) of point cloud map ₀ ，X ₁ ，X ₂ ，…，X _N ) And their matching relationship, the reprojection error E can be expressed by the following formula ₂ ：

In the formula (2), pi represents the projection from the camera coordinate system to the image coordinate system, N is an integer and is greater than or equal to 0 and less than or equal to N. And (3) carrying out optimization solution on the formula (2) to obtain optimized pose information, namely obtaining the kth third pose information after pose correction.

By the method, the reprojection error constraint can be carried out according to the feature point matching information, and the local pose is corrected, so that the positioning precision and stability are improved.

In a possible implementation manner, the kth first positioning result further includes feature point matching information of the kth frame, the feature point matching information of the kth frame includes matching information between a two-dimensional feature point in the environmental image of the kth frame and a three-dimensional feature point in the point cloud map of the second electronic device,

the correcting the kth second positioning result according to the kth first positioning result to determine a kth third positioning result of the first electronic device further includes:

and correcting the kth third pose information according to the feature point matching information of the kth frame, and determining a corrected kth third positioning result.

For example, under the condition that the cloud positioning result includes pose information and feature point matching information, a pose graph can be established according to the pose information of the current positioning and the pose information of the historical positioning, and a local map (map) is restrained; and then, carrying out reprojection error constraint according to the feature point matching information, and further correcting the pose.

In a possible implementation manner, after the kth third pose information is obtained through optimization by formula (1), the reprojection error constraint may be performed on the kth third pose information:

in the formula (3), pi represents the projection from the camera coordinate system to the image coordinate system, N is an integer and is greater than or equal to 0 and less than or equal to N. And (4) carrying out optimization solution on the formula (3) to obtain the optimized pose information, namely obtaining the kth third positioning result after pose correction again.

By the method, the positioning result can be respectively constrained according to the pose information and the feature point matching information in the kth first positioning result, and the local pose is corrected, so that the positioning precision and stability are further improved.

In one possible implementation, the method further includes:

sending an a first positioning request to second electronic equipment, wherein the a first positioning request comprises an a frame environment image of the environment where the first electronic equipment is located, a is an integer and a is more than or equal to 1;

initializing a synchronous positioning and mapping SLAM system in the first electronic equipment according to the a-th first positioning result under the condition of receiving the a-th first positioning result sent by the second electronic equipment, determining the initial positioning result of the first electronic equipment,

the a-th first positioning result comprises a-th first pose information and/or a-th frame feature point matching information.

For example, in general, when the SLAM system is initialized, the first electronic device needs to move for multiple times to acquire a multi-frame environment image, so as to ensure that sufficient observation is generated on a point in the environment image, thereby enabling a positioning result to converge, determining an initial positioning result of the first electronic device, and implementing initialization of the SLAM system. In this case, the first electronic device may utilize the first positioning result returned by the cloud to implement initialization.

In a possible implementation manner, the first electronic device may send an a-th first positioning request to the second electronic device so as to determine a position of the first electronic device, where the a-th first positioning request includes an a-th frame environment image acquired by the first electronic device, and a is an integer greater than or equal to 1. After receiving the a-th first positioning request, the second electronic device may extract feature information of the a-th frame environmental image in the a-th first positioning request, match the feature information with a point cloud map, determine a matched visual positioning result (which may be referred to as the a-th first positioning result), and send the result to the first electronic device. The first positioning result includes pose information (position) and/or feature point matching information (match) of the first electronic device.

In a possible implementation manner, when the first electronic device receives the a-th first positioning result, the first electronic device may initialize the synchronous positioning and mapping SLAM system in the first electronic device according to the a-th first positioning result, and determine the initial positioning result of the first electronic device.

When a =1, the first positioning result can be directly used as an initial positioning result of the first electronic device; when a is greater than 1, the local positioning result of the first electronic device may be corrected according to the a-th first positioning result, so as to determine the initial positioning result of the first electronic device. The present disclosure is not limited to the specific manner of modification.

In this way, the speed of initialization of the SLAM system of the first electronic device can be increased, and initialization can be completed even in a stationary state.

The positioning method according to the embodiment of the disclosure can reduce the accumulated error of the SLAM system through the processing mode; and a tight coupling strategy between the high-precision map and the local SLAM system can be realized by requesting the high-precision map at the cloud, so that the mapping requirement of the local SLAM is reduced, and the positioning precision is further improved.

In a possible implementation manner, the kth first positioning result sent by the second electronic device further includes a first area map, where the first area map includes a point cloud sub-map of a geographic area corresponding to the first electronic device.

In a possible implementation manner, after determining the kth third positioning result in step S12, the method further includes:

performing projection matching on the kth third positioning result and a point cloud sub-map in the first area map, and determining a fourth positioning result of the first electronic device;

and displaying a display interface of the first electronic equipment according to the fourth positioning result.

That is to say, the kth first positioning request sent by the first electronic device may also request a sub-map of the cloud. The second electronic equipment can perform visual positioning and determine a kth first positioning result when receiving a kth first positioning request sent by the first electronic equipment; according to the kth first positioning result, the second electronic equipment can determine a point cloud sub-map of the geographic area corresponding to the first electronic equipment from the point cloud map. The geographic area may be an area within a certain range near the location of the first electronic device, for example, a circular area within a preset distance (e.g., 10 meters) from the location of the first electronic device, and the specific range of the geographic area is not limited by the present disclosure.

In a possible implementation manner, the second electronic device may package the point cloud sub-map of the geographic area into a map format of SLAM, generate the first area map, and send the first area map to the first electronic device.

Because the high-precision map of the cloud end is constructed by fully optimizing the real scene, the positioning precision and the stability can be improved by matching the sub-map of the cloud end locally.

In a possible implementation manner, the kth first positioning result received by the first electronic device includes a first area map. After determining the kth third positioning result in step S12, the first electronic device may perform projection matching on the kth third positioning result (Pose) and the point cloud sub-map in the first area map, that is, project the three-dimensional point cloud sub-map into a two-dimensional image, and then perform matching on the two-dimensional image and the kth third positioning result, so as to determine a positioning result (which may be referred to as a fourth positioning result) of the first electronic device. The fourth positioning result includes pose information of the first electronic device.

Because the image in the cloud map is inconsistent with the local environment image, when image matching is performed in the SLAM system by using, for example, an ORB (Oriented Fast and Rotated description, a Fast feature point extraction and description algorithm) descriptor, a projection matching mode is adopted, so that the search radius can be reduced, and the probability of mismatching can be reduced.

In a possible implementation manner, a display interface of the first electronic device may be displayed according to the fourth positioning result. That is, the output result of the VIO module of the local SLAM system may be constrained by the fourth positioning result. And according to the current application scene of the first electronic equipment, rendering corresponding content to be displayed through an output result of the VIO module, and displaying in a display interface.

By the method, accumulated errors can be eliminated to a smaller range through the Pose position of the cloud and the feature point matching information Match, and then projection matching is carried out on the positioning result position and the point cloud sub-map, so that the matching success rate and the accuracy can be improved, and the positioning accuracy is further improved.

The above processing method can improve the positioning accuracy, but the high-accuracy sub-map of the request cloud may consume more network resources. Therefore, whether the sub-map is requested or not can be selected according to the conditions that whether the current frame sufficiently observes the loaded high-precision sub-map (namely, whether the current frame is in the area of the loaded high-precision sub-map or not) or whether the network state meets the requirement or not.

In a possible implementation manner, before the sending the kth first positioning request to the second electronic device in step S11, the method further includes:

judging whether the first electronic equipment is located in a second regional map according to a kth second positioning result of the first electronic equipment and the local second regional map of the first electronic equipment, wherein the second regional map is a regional map sent by the second electronic equipment;

wherein the sending the kth first positioning request to the second electronic device includes:

and sending a kth first positioning request to a second electronic device under the condition that the first electronic device is in the second regional map.

For example, when the first electronic device performs positioning for the first time, the high-precision sub-map of the cloud can be directly requested, so that accurate positioning is realized. During multiple positioning of the first electronic device (for example, during navigation to a destination through an AR), the high-precision sub-map of the cloud may have been received and stored locally. In this case, it may be determined whether the first electronic device is within a range of a locally existing sub-map.

In a possible implementation manner, according to a kth second positioning result local to the first electronic device and an area map (which may be referred to as a second area map) sent by the second electronic device, it may be determined whether the position coordinate of the first electronic device is within a coordinate range of a geographic area corresponding to the second area map. If the first electronic equipment is within the range of the second regional map, the loaded high-precision sub-map can be considered to be fully observed, and the point cloud sub-map in the local second regional map can be directly used for accurate positioning.

In a possible implementation manner, when the first electronic device is located in the second area map, the kth first positioning request may be sent in step S11 to request to acquire the Pose information Pose and the feature point matching information Match of the cloud, so as to correct the Pose and reduce the accumulated error.

By the method, the times of requesting the cloud high-precision sub-map can be reduced, the transmitted data volume is reduced, and the positioning efficiency and the positioning speed are improved.

In one possible implementation, the method may further include:

under the condition that the first electronic equipment is not located in the second regional map, sending a second positioning request to the second electronic equipment, wherein the second positioning request comprises a kth frame environment image of the environment where the first electronic equipment is located;

under the condition that a third area map sent by the second electronic device is received, performing projection matching on a current second positioning result and a point cloud sub-map in the third area map, and determining a fifth positioning result of the first electronic device, wherein the third area map comprises the point cloud sub-map of a geographic area corresponding to the first electronic device;

and displaying a display interface of the first electronic equipment according to the fifth positioning result.

As described above, if the first electronic device is located within the range of the second area map, that is, the location of the first electronic device is within the area of the second area map information, the current second positioning result may be projection-matched with the point cloud sub-map in the second area map, so as to determine the positioning result of the first electronic device (which may be referred to as a fifth positioning result). The fifth positioning result includes pose information of the first electronic device.

In a possible implementation manner, a display interface of the first electronic device may be displayed according to a fifth positioning result. That is, the output result of the VIO module of the local SLAM system may be constrained by the fifth positioning result. And according to the current application scene of the first electronic equipment, rendering corresponding content to be displayed through an output result of the VIO module, and displaying in a display interface.

In this way, the accuracy and stability of positioning can be improved through the loaded high-accuracy sub-map, the search radius is reduced through projection matching, and the probability of mismatching is reduced.

In a possible implementation manner, before sending the first positioning request to the second electronic device in step S11, the method further includes:

acquiring a network state of the first electronic equipment, wherein the network state comprises network signal strength and/or information transmission speed;

sending a kth first positioning request to a second electronic device under the condition that the network state does not meet the preset regional map acquisition condition,

wherein the regional map acquisition condition includes: the network signal strength is greater than or equal to a signal strength threshold; and/or the information transmission speed is greater than or equal to a speed threshold.

For example, the network status of the first electronic device may be obtained, including network signal strength and/or information transmission speed, etc. If the network signal intensity is greater than or equal to the signal intensity threshold value and/or the information transmission speed is greater than or equal to the speed threshold value, the network state is considered to be good, the delay of the high-precision sub-map of the request cloud end is small, and the preset regional map acquisition condition is met; on the contrary, if the network signal strength is smaller than the signal strength threshold value and/or the information transmission speed is smaller than the speed threshold value, the network state is considered to be poor, the delay of requesting the high-precision sub-map of the cloud end is large, and the preset regional map acquisition condition is not met.

In a possible implementation manner, if the network state of the first electronic device does not satisfy the preset regional map obtaining condition, a kth first positioning request may be sent in step S11 to request to obtain the Pose information dose and the feature point matching information Match of the cloud, so as to correct the Pose and reduce the accumulated error.

By the method, whether the high-precision sub-map of the cloud is requested or not can be determined according to the network state, the transmitted data volume is reduced, and the positioning efficiency and the positioning speed are improved.

In one possible implementation, step S13 may include:

determining a navigation path of the first electronic device according to the kth third positioning result and the geographic position of the destination;

and displaying the augmented reality AR navigation path in the display interface according to the navigation path.

For example, in an application scenario of AR navigation, a navigation path of the first electronic device may be determined according to pose information in the kth third positioning result and a geographic location of a destination set by the user. Similarly, when the fourth positioning result or the fifth positioning result is obtained, the navigation path of the first electronic device may be determined according to the pose information in the fourth or fifth positioning result and the geographic location of the destination set by the user. The present disclosure does not limit the specific manner of determining the navigation path.

In one possible implementation, according to the navigation path, the AR navigation path may be presented in a live-action image or a panoramic image of a display interface of the first electronic device so as to instruct the user to travel along the AR navigation path. The AR navigation path includes, for example, AR arrows along the navigation path. The present disclosure does not limit the specific form of the AR navigation path.

By the method, AR navigation in the live-action image can be realized, the intuitiveness of the navigation route is improved, and the precision and the stability of AR navigation path display are improved.

In one possible implementation, step S13 may include:

determining second object pose information of the AR object in the display interface according to the kth third positioning result and the first object pose information of the AR object;

and displaying the AR object in the display interface according to the second object pose information.

For example, in an application scene showing the AR object, the pose information of the AR object in the display interface may be determined according to the pose information in the kth third positioning result and the pose information of the first object of the AR object to be shown. The present disclosure is not limited to the specific manner of determination.

Similarly, under the condition that the fourth positioning result or the fifth positioning result is obtained, the second object pose information of the AR object in the display interface can be determined according to the pose information in the fourth positioning result or the fifth positioning result and the first object pose information of the AR object to be displayed.

In one possible implementation, the AR object may be presented in a live view image of the display interface according to the second object pose information. The AR objects to be presented may include, among other things, AR markers, virtual objects, such as virtual landscapes, virtual animals, and the like. The present disclosure does not limit the specific categories of AR objects.

By the method, the AR object display in the live-action image can be realized, and the precision and the stability of the AR object display are improved.

Fig. 2 shows a flow chart of a positioning method according to an embodiment of the present disclosure. The method is applied to a second electronic device, and as shown in fig. 2, the positioning method includes:

in step S21, when a kth first positioning request from a first electronic device is received, visually positioning the first electronic device according to a preset point cloud map and a kth frame environment image in the kth first positioning request, so as to obtain a kth first positioning result of the first electronic device;

in step S22, the kth first positioning result is sent to the first electronic device, so that the first electronic device performs positioning result correction according to the kth first positioning result,

For example, for a certain positioning (set as kth positioning, k being an integer greater than 1), in step S11, the first electronic device may send a kth first positioning request to the second electronic device so as to determine its own position. The kth first positioning request comprises a kth frame environment image acquired by the first electronic equipment.

In a possible implementation manner, after receiving the kth first positioning request, the second electronic device may extract feature information of the kth frame of environment image in the kth first positioning request. Feature extraction can be performed on the kth frame of environmental image through a pre-trained neural network, for example, so as to obtain feature information of the kth frame of environmental image. The present disclosure does not limit the specific manner of feature extraction.

In a possible implementation manner, the kth first positioning result includes pose information (may be abbreviated as "position") and/or feature point matching information (may be abbreviated as "match") of the first electronic device. The pose information comprises position information and attitude information, and the position information can comprise position coordinates of the first electronic equipment; the attitude information may include an orientation, a pitch angle, etc. of the first electronic device.

In one possible implementation, the feature point matching information may include matching information between two-dimensional feature points in the k-th frame environment image and three-dimensional feature points in the point cloud map. For example, N +1 2D feature points (x) of the k-th frame environment image ₀ ，x ₁ ，x ₂ ，…，x _N ) And N +1 3D feature points (X) of point cloud map ₀ ，X ₁ ，X ₂ ，…，X _N ) Respectively matching, the feature point matching information includes 2D feature point (x) ₀ ，x ₁ ，x ₂ ，…，x _N ) Coordinates of (2), 3D feature points (X) ₀ ，X ₁ ，X ₂ ，…，X _N ) And matching relationship between them, N is an integer greater than 0.

In a possible implementation manner, the second electronic device may send the k first positioning results to the first electronic device, so that the first electronic device performs positioning result correction according to the k first positioning result.

According to the embodiment of the disclosure, visual positioning can be performed according to the environment image in the positioning request, a cloud positioning result is obtained and sent to the first electronic device, so that the first electronic device can correct the positioning result, and the positioning accuracy and stability are improved.

In one possible implementation, the method further includes:

under the condition that a second positioning request from first electronic equipment is received, carrying out visual positioning on the first electronic equipment according to the point cloud map and an environment image in the second positioning request to obtain a sixth positioning result of the first electronic equipment;

determining a point cloud sub-map of a geographic area corresponding to the first electronic device from the point cloud map according to the sixth positioning result;

and sending a regional map to the first electronic equipment so that the first electronic equipment can be positioned and displayed according to the regional map, wherein the regional map comprises a point cloud sub-map of a geographic region corresponding to the first electronic equipment.

For example, the first electronic device may send a second positioning request requesting a high-precision sub-map of the cloud. After receiving the second positioning request, the second electronic device may extract feature information of the environment image in the second positioning request, match the feature information with the point cloud map, and determine a matched visual positioning result (which may be referred to as a sixth positioning result), where the sixth positioning result includes pose information and/or feature point matching information of the first electronic device.

In a possible implementation manner, the second electronic device may determine, according to the sixth positioning result, a point cloud sub-map of a geographic area corresponding to the first electronic device from the point cloud map. The geographic area may be an area within a certain range around the location of the first electronic device, such as a circular area within a preset distance (e.g., 5 meters) from the location of the first electronic device, which is not limited by this disclosure.

In a possible implementation manner, the second electronic device may package the point cloud sub-map of the geographic area into a map format of SLAM, generate an area map, and send the area map to the first electronic device, so that the first electronic device performs positioning and display according to the area map.

In this way, the positioning result can be obtained through visual positioning according to the point cloud map and the environment image in the second positioning request; determining a point cloud sub-map corresponding to the geographic area according to the positioning result; and generating and sending a regional map, so that the first electronic equipment can be matched locally through a high-precision sub-map of the cloud, and the positioning precision and the stability are improved.

In a possible implementation manner, the step of determining, according to the sixth positioning result, a point cloud sub-map of a geographic area corresponding to the first electronic device from the point cloud map includes:

determining a predicted position of the first electronic device according to the pose information in the sixth positioning result, the predicted moving speed of the first electronic device and preset response time;

and determining a point cloud sub-map of a geographic area corresponding to the first electronic device from the point cloud map, wherein the geographic area comprises a first geographic area where the sixth positioning result is located and a second geographic area where the predicted position is located.

For example, in some application scenarios (e.g., AR navigation scenarios), the user may be in a state of continuous movement, and when the second electronic device returns the first area map information within a certain range near the location of the first electronic device, the user may have walked out of the range, resulting in the first electronic device failing to match to the corresponding location and posture, resulting in a positioning failure.

In one possible implementation, the second electronic device may predict the speed of movement of the first electronic device and the response time required to return the first area map information. For example, the predicted moving speed of the first electronic device may be determined according to the time when the first electronic device sent the positioning request last time and the positioning result last time, and the time when the first electronic device sent the positioning request this time and the positioning result; the predicted moving speed of the first electronic device can also be determined according to a plurality of moments when the first electronic device sends the positioning request and a plurality of positioning results within a period of time. The present disclosure does not limit the specific manner of determining the predicted movement speed.

In a possible implementation manner, the second electronic device may determine the response time according to a time difference between a time when the first electronic device previously sent the request and a time when the second electronic device previously sent the area map; and determining a plurality of time differences according to a plurality of moments when the first electronic device sends the request and a plurality of moments when the second electronic device sends the regional map within a period of time, and further determining the response time. The specific manner in which the response time is determined is not limited by this disclosure.

In a possible implementation manner, according to the position x in the sixth positioning result, the predicted moving speed v of the first electronic device, and the preset response time t, the predicted position x' = vt + x of the first electronic device may be determined. Where t can be set, for example, to 5-10s, which the present disclosure does not limit.

In one possible implementation manner, the geographic area corresponding to the first electronic device may be set to include a first geographic area within a certain range around the position in the sixth positioning result, and a second geographic area within a certain range around the predicted position. The second electronic device can determine the point cloud sub-map of the geographic area from the point cloud map of the whole area, package the point cloud sub-map into a map format of SLAM, generate the area map and send the area map to the first electronic device.

In a possible implementation manner, the second electronic device may further determine a panoramic sub-map of the geographic area from the panoramic map of the entire area, package the panoramic sub-map into a map format of SLAM, and add the map format to the area map. The present disclosure is not so limited.

By the method, the efficiency and the success rate of the accurate positioning of the first electronic equipment can be improved, and the positioning effect is further improved.

Fig. 3 shows an interaction diagram of a positioning method according to an embodiment of the present disclosure. As shown in fig. 3, a user may hold or wear a first electronic device 31, and when the user needs to determine his/her own position, an environmental image of the environment where the user is located may be collected by a collecting component (not shown) of the first electronic device 31, and a positioning request may be sent to a second electronic device 32 through a wireless network.

The second electronic device 32 stores a point cloud map of a geographic area (e.g., an inner area of a mall, a city area, etc.) where the first electronic device is located. When receiving the visual positioning request, the second electronic device 32 may perform visual positioning according to the environment image and the point cloud map, and return a positioning result, including pose information pos and/or feature point matching information match. The first electronic device 31 corrects the local positioning result according to the returned positioning result.

Fig. 4 shows a schematic diagram of a process of a positioning method according to an embodiment of the present disclosure. As shown in fig. 4, when the first electronic device needs to be located, a location request may be sent to the second electronic device, so as to implement location; when a positioning result returned by the second electronic device is received, the accumulated error can be repaired by using the positioning result (including the pose information (position) and/or the feature point matching information (match)) of the cloud.

In an example, if the repair fails, the positioning request may be sent again to obtain a new positioning result for repair, or may be directly repaired again; if the repair is successful, the accumulated error may be considered to have been eliminated to a lesser extent. In this case, projection matching can be performed by the positioning result and the point cloud sub-map

In an example, if matching fails, a new positioning result or a new high-precision sub-map may be requested from the cloud, matching is performed, or direct re-matching is performed; if the matching is successful, it can be considered that accurate positioning has been obtained, and the local SLAM system of the first electronic device can automatically implement position tracking, and perform corresponding processing according to the actual application scenario, for example, perform AR navigation. By the method, the success rate and the accuracy of matching can be improved, and the positioning precision is further improved.

According to the positioning method disclosed by the embodiment of the disclosure, a positioning request comprising an environment image can be sent to the second electronic equipment; correcting a local positioning result according to a cloud positioning result returned by the second electronic equipment and a previous historical positioning result; and the positioning result after correction is displayed, so that the accumulated error of the SLAM system is reduced, and the positioning precision and stability are improved.

In the correction process, a pose graph can be established through the cloud pose of the current positioning and the historical positioning pose, the local Map is restrained, and the correction of the local pose is realized; the re-projection error constraint can be carried out in the local constraint adjustment LBA through the feature point matching information of the cloud end, and the correction of the local pose is realized; and the local pose can be corrected by combining the two modes. In this way, the accumulated error of the SLAM back end Map can be suppressed, and the VIO to Map offset (Drift) can be modified so that the VIO track matches the Map.

According to the positioning method disclosed by the embodiment of the disclosure, the accumulated error can be eliminated to a smaller range through the position and Pose of the cloud and the feature point matching information Match, and then the projection matching is performed through the positioning result (Pose) and the point cloud sub-map, so that the matching success rate and the accuracy are improved, and the positioning precision is further improved.

According to the positioning method disclosed by the embodiment of the disclosure, whether the sub-map is requested or not can be selected according to the conditions that whether the current frame sufficiently observes the loaded high-precision sub-map or whether the network state meets the requirements or not, so that the sub-map requesting efficiency is improved, and the dependence of a system on the sub-map in the application scene of navigation and the like is reduced. The strategy of eliminating accumulated errors through the Pose Pose of the cloud and the feature point matching information Match is combined with the strategy of requesting a high-precision sub-map of the cloud, so that a robust high-precision map tightly-coupled SLAM system is constructed, the positioning precision and stability are improved, and the robust AR effect of the SLAM system can be kept in various scenes.

The positioning method according to the embodiment of the disclosure can be applied to various application scenes of augmented reality AR, such as scenes of AR cloud, AR navigation and the like; and Location Based Services (LBS) in various application scenarios, to improve the effects of positioning, presentation, and interaction. Under the application scenes of AR navigation, AR display and the like, the relative error generated when the AR effect is displayed in a local range can be reduced, and the phenomena of shaking, drifting and the like of the AR effect under a weak texture scene are reduced.

It is understood that the above-mentioned embodiments of the method of the present disclosure can be combined with each other to form a combined embodiment without departing from the principle logic, which is limited by the space, and the detailed description of the present disclosure is omitted. Those skilled in the art will appreciate that in the above methods of the specific embodiments, the specific order of execution of the steps should be determined by their function and possibly their inherent logic.

In addition, the present disclosure also provides a positioning apparatus, an electronic device, a computer-readable storage medium, and a program, which can be used to implement any one of the positioning methods provided by the present disclosure, and the corresponding technical solutions and descriptions and corresponding descriptions in the method sections are not repeated.

Fig. 5 shows a block diagram of a positioning apparatus according to an embodiment of the present disclosure, as shown in fig. 5, the apparatus is applied to a first electronic device, and includes:

a first request sending module 51, configured to send a kth first positioning request to a second electronic device, where the kth first positioning request includes a kth frame environment image of an environment where the first electronic device is located, and k is an integer greater than 1;

a result correction module 52, configured to, when a kth first positioning result sent by the second electronic device is received, correct a kth second positioning result according to the kth first positioning result, and determine a kth third positioning result of the first electronic device, where the kth second positioning result is obtained by performing local positioning on the kth frame of environment image by the first electronic device;

and the first displaying module 53 is configured to display the display interface of the first electronic device according to the kth third positioning result.

In a possible implementation manner, the kth first positioning result includes kth first position information, the kth second positioning result includes kth second position information, and the kth third positioning result includes kth third position information, and the result correction module includes:

a current deviation determining submodule, configured to determine a current posture deviation of the kth positioning according to the kth first posture information and the kth second posture information; the overall deviation determining submodule is used for determining the overall pose deviation of k times of positioning according to k second pose information and k third pose information of k times of positioning; and the first correction submodule is used for correcting the pose according to the current pose deviation of the kth positioning and the total pose deviation of the kth positioning, and determining the kth third pose information.

In one possible implementation, the overall deviation determining submodule is configured to: determining first position change information between the ith second position information and the jth second position information, wherein i and j are integers, and i is more than or equal to 1 and less than or equal to j and less than or equal to k; determining second posture change information between the ith third posture information and the jth third posture information; determining pose deviation from the ith frame to the jth frame according to the first pose change information and the second pose change information; and determining the total pose deviation of the k times of positioning according to the plurality of pose deviations.

In a possible implementation manner, the kth first positioning result further includes feature point matching information of a kth frame, where the feature point matching information of the kth frame includes matching information between a two-dimensional feature point in an environmental image of the kth frame and a three-dimensional feature point in a point cloud map of the second electronic device, and the result modification module further includes: and the second correction submodule is used for correcting the kth third pose information according to the feature point matching information of the kth frame and determining a corrected kth third positioning result.

In one possible implementation, the apparatus further includes: the second request sending module is used for sending an a-th first positioning request to the second electronic equipment, wherein the a-th first positioning request comprises an a-th frame environment image of the environment where the first electronic equipment is located, a is an integer and a is more than or equal to 1; the initialization module is configured to initialize a synchronous positioning and mapping SLAM system in the first electronic device according to an a-th first positioning result sent by the second electronic device when the a-th first positioning result is received, and determine an initial positioning result of the first electronic device, where the a-th first positioning result includes a-th first pose information and/or feature point matching information of an a-th frame.

Fig. 6 shows a block diagram of a positioning apparatus according to an embodiment of the present disclosure, as shown in fig. 6, the apparatus is applied to a second electronic device, and includes:

the first positioning module 61 is configured to, when a kth first positioning request from a first electronic device is received, perform visual positioning on the first electronic device according to a preset point cloud map and a kth frame environment image in the kth first positioning request, and obtain a kth first positioning result of the first electronic device;

a result sending module 62, configured to send the kth first positioning result to the first electronic device, so that the first electronic device performs positioning result correction according to the kth first positioning result,

In one possible implementation, the apparatus further includes: the second positioning module is used for carrying out visual positioning on the first electronic equipment according to the point cloud map and the environment image in the second positioning request under the condition that the second positioning request from the first electronic equipment is received, so as to obtain a sixth positioning result of the first electronic equipment; the sub-map determining module is used for determining a point cloud sub-map of a geographic area corresponding to the first electronic device from the point cloud map according to the sixth positioning result; the regional map sending module is used for sending a regional map to the first electronic equipment so that the first electronic equipment can be positioned and displayed according to the regional map, and the regional map comprises a point cloud sub-map of a geographic region corresponding to the first electronic equipment.

In some embodiments, functions of or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementation thereof may refer to the description of the above method embodiments, and for brevity, will not be described again here.

Embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the above-mentioned method. The computer readable storage medium may be a non-volatile computer readable storage medium.

An embodiment of the present disclosure further provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the memory-stored instructions to perform the above-described method.

Embodiments of the present disclosure also provide a computer program product, which includes computer readable code, when the computer readable code runs on a device, a processor in the device executes instructions for implementing the positioning method provided in any embodiment.

The embodiments of the present disclosure also provide another computer program product for storing computer readable instructions, which when executed cause a computer to perform the operations of the positioning method provided in any of the above embodiments.

The electronic device may be provided as a terminal, server, or other form of device.

Fig. 7 illustrates a block diagram of an electronic device 800 in accordance with an embodiment of the disclosure. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like terminal.

Referring to fig. 7, electronic device 800 may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, and communications component 816.

The processing component 802 generally controls overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operations at the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

The power supply component 806 provides power to the various components of the electronic device 800. The power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 800 is in an operation mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the electronic device 800. For example, the sensor assembly 814 may detect an open/closed state of the electronic device 800, the relative positioning of components, such as a display and keypad of the electronic device 800, the sensor assembly 814 may also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, orientation or acceleration/deceleration of the electronic device 800, and a change in the temperature of the electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G) or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.

In an exemplary embodiment, a non-transitory computer-readable storage medium, such as the memory 804, is also provided that includes computer program instructions executable by the processor 820 of the electronic device 800 to perform the above-described methods.

Fig. 8 illustrates a block diagram of an electronic device 1900 in accordance with an embodiment of the disclosure. For example, the electronic device 1900 may be provided as a server. Referring to fig. 8, electronic device 1900 includes a processing component 1922 further including one or more processors and memory resources, represented by memory 1932, for storing instructions, e.g., applications, that are executable by processing component 1922. The application programs stored in memory 1932 may include one or more modules that each correspond to a set of instructions. Further, the processing component 1922 is configured to execute instructions to perform the above-described method.

The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input/output (I/O) interface 1958. The electronic device 1900 may operate based on an operating system, such as the Microsoft Server operating system (Windows Server), stored in the memory 1932 ^TM ) Apple Inc. of the present invention ^TM ) Multi-user, multi-process computer operating system (Unix) ^TM ) Free and open native code Unix-like operating System (Linux) ^TM ) Open native code Unix-like operating System (FreeBSD) ^TM ) Or the like.

In an exemplary embodiment, a non-transitory computer readable storage medium, such as the memory 1932, is also provided that includes computer program instructions executable by the processing component 1922 of the electronic device 1900 to perform the above-described methods.

The present disclosure may be systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement various aspects of the present disclosure.

The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be interpreted as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

The computer program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the disclosure are implemented by personalizing an electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), with state information of computer-readable program instructions, which can execute the computer-readable program instructions.

Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The computer program product may be embodied in hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.

The foregoing description of the embodiments of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A positioning method applied to a first electronic device includes:

sending a kth first positioning request to second electronic equipment, wherein the kth first positioning request comprises a kth frame environment image of an environment where the first electronic equipment is located, and k is an integer greater than 1;

under the condition that a kth first positioning result sent by the second electronic device is received, correcting the kth second positioning result according to the kth first positioning result, and determining a kth third positioning result of the first electronic device, wherein the kth second positioning result is obtained by the first electronic device performing local positioning on the kth frame environment image;

displaying a display interface of the first electronic device according to the kth third positioning result;

the kth first positioning result comprises kth first pose information, the kth second positioning result comprises kth second pose information, the kth third positioning result comprises kth third pose information,

determining the total pose deviation of the k times of positioning according to the k second pose information and the k third pose information of the k times of positioning;

and correcting the pose according to the current pose deviation of the kth positioning and the total pose deviation of the kth positioning, and determining the kth third pose information.

2. The method according to claim 1, wherein determining an overall pose deviation of the k-th position according to the k second pose information of the k-th position and the k third position results comprises:

determining the position and orientation deviation from the ith frame to the jth frame according to the first position and orientation change information and the second position and orientation change information;

3. The method of claim 1 or 2, wherein the kth first positioning result further comprises feature point matching information of the kth frame, wherein the feature point matching information of the kth frame comprises matching information between two-dimensional feature points in the environment image of the kth frame and three-dimensional feature points in the point cloud map of the second electronic device,

4. The method of claim 1, wherein the kth first positioning result comprises feature point matching information of a kth frame, wherein the feature point matching information of the kth frame comprises matching information between a two-dimensional feature point in an environment image of the kth frame and a three-dimensional feature point in a point cloud map of the second electronic device,

the determining a kth third positioning result of the first electronic device by correcting the kth second positioning result according to the kth first positioning result includes:

5. The method of claim 1, further comprising:

sending an a-th first positioning request to second electronic equipment, wherein the a-th first positioning request comprises an a-th frame environment image of the environment where the first electronic equipment is located, a is an integer and a is more than or equal to 1;

initializing a synchronous positioning and mapping SLAM system in the first electronic equipment according to the a-th first positioning result under the condition of receiving the a-th first positioning result sent by the second electronic equipment, determining an initial positioning result of the first electronic equipment,

6. The method of any of claims 1-5, wherein the kth first positioning result sent by the second electronic device further comprises a first area map, the first area map comprising a point cloud sub-map of a geographic area corresponding to the first electronic device,

after determining the kth third positioning result, the method further comprises:

7. The method of any of claims 1-5, wherein prior to sending the kth first positioning request to the second electronic device, the method further comprises:

8. The method of claim 7, further comprising:

9. The method of any of claims 1-8, wherein prior to sending the first positioning request to the second electronic device, the method further comprises:

transmitting a kth first positioning request to a second electronic device under the condition that the network state does not satisfy a preset regional map acquisition condition,

10. A positioning method applied to a second electronic device includes:

under the condition that a kth first positioning request from first electronic equipment is received, carrying out visual positioning on the first electronic equipment according to a preset point cloud map and a kth frame environment image in the kth first positioning request to obtain a kth first positioning result of the first electronic equipment;

sending the kth first positioning result to the first electronic device so that the first electronic device performs positioning result correction according to the kth first positioning result,

the kth first positioning result comprises kth first pose information and/or characteristic point matching information of a kth frame, and the characteristic point matching information of the kth frame comprises matching information between two-dimensional characteristic points in the kth frame environment image and three-dimensional characteristic points in the point cloud map;

after the first electronic device receives the kth first positioning result, correcting the kth second positioning result determined by the first electronic device according to the kth first positioning result, and determining a kth third positioning result of the first electronic device;

the first electronic device corrects a kth second positioning result determined by the first electronic device according to the kth first positioning result, and the process of determining a kth third positioning result of the first electronic device includes:

11. The method of claim 10, further comprising:

12. A positioning device applied to a first electronic device, comprising:

the first request sending module is used for sending a kth first positioning request to second electronic equipment, wherein the kth first positioning request comprises a kth frame environment image of an environment where the first electronic equipment is located, and k is an integer greater than 1;

the first display module is used for displaying a display interface of the first electronic device according to the kth third positioning result;

the kth first positioning result includes kth first pose information, the kth second positioning result includes kth second pose information, the kth third positioning result includes kth third pose information, and the result correction module includes:

13. A positioning device applied to a second electronic device, comprising:

the first positioning module is used for carrying out visual positioning on first electronic equipment according to a preset point cloud map and a kth frame environment image in a kth first positioning request under the condition that the kth first positioning request from the first electronic equipment is received, so that a kth first positioning result of the first electronic equipment is obtained;

after receiving the kth first positioning result, the first electronic device corrects the kth second positioning result determined by the first electronic device according to the kth first positioning result, and determines a kth third positioning result of the first electronic device;

14. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to invoke the memory-stored instructions to perform the method of any of claims 1 to 11.

15. A computer readable storage medium having computer program instructions stored thereon, which when executed by a processor implement the method of any one of claims 1 to 11.