CN115952248A

CN115952248A - Pose processing method, device, equipment, medium and product of terminal equipment

Info

Publication number: CN115952248A
Application number: CN202211641553.6A
Authority: CN
Inventors: 张志远; 王煜安
Original assignee: Apollo Zhilian Beijing Technology Co Ltd
Current assignee: Apollo Zhilian Beijing Technology Co Ltd
Priority date: 2022-12-20
Filing date: 2022-12-20
Publication date: 2023-04-11

Abstract

The disclosure provides a pose processing method, a pose processing device, pose processing equipment, pose processing media and pose processing products of terminal equipment, relates to the field of artificial intelligence, and particularly relates to technologies such as automatic driving, intelligent transportation, deep learning and intelligent cabins, and can be applied to software development scenes. The method comprises the following steps: responding to a pose processing request of a terminal device, and obtaining an initial pose of the terminal device and an image to be positioned acquired at the initial pose; inquiring target map data matched with the initial pose from a map database, wherein the map data in the map database are used for recording pose tags and key map information associated with the pose tags; and optimizing the initial pose according to the target map data and the image to be positioned to obtain the target pose of the terminal equipment.

Description

Pose processing method, device, equipment, medium and product of terminal equipment

Technical Field

The disclosure relates to the field of artificial intelligence, in particular to technologies such as automatic driving, intelligent transportation, deep learning and intelligent cabins, which can be applied to software development scenes, and in particular relates to a pose processing method, device, equipment, medium and product of terminal equipment.

Background

The auxiliary driving technology is a technology for providing auxiliary support for an automobile through equipment such as a radar, a camera and a positioning module in the driving process. Vehicle positioning in the auxiliary driving technology is a common application requirement, and can be widely applied to common application scenes such as vehicle tracking, navigation and the like. At present, a common Positioning technology generally positions a vehicle through a Positioning module, such as a GPS (Global Positioning System) Positioning module, a beidou Positioning module, and the like. In the running process of the vehicle, the vehicle may pass through a tunnel or a culvert, the positioning signal of the vehicle is poor, and the vehicle positioning processing executed by using a weak GNSS (Global Navigation Satellite System) signal may cause the vehicle positioning to be inaccurate and the vehicle running danger exists.

Disclosure of Invention

The disclosure provides a pose processing method, a pose processing device, pose processing equipment, pose processing media and a pose processing product of terminal equipment.

According to a first aspect of the present disclosure, there is provided a pose processing method of a terminal device, including:

responding to a pose processing request of a terminal device, and obtaining an initial pose of the terminal device and an image to be positioned acquired at the initial pose;

inquiring target map data matched with the initial pose from a map database, wherein the map data in the map database are used for recording pose tags and key map information associated with the pose tags;

and optimizing the initial pose according to the target map data and the image to be positioned to obtain the target pose of the terminal equipment.

According to a second aspect of the present disclosure, there is provided a pose processing apparatus of a terminal device, including:

a response unit comprising: responding to a pose processing request of a terminal device, and obtaining an initial pose of the terminal device and an image to be positioned acquired at the initial pose;

a query unit comprising: inquiring target map data matched with the initial pose from a map database, wherein the map data in the map database are used for recording pose tags and key map information associated with the pose tags;

an optimization unit comprising: and optimizing the initial pose according to the target map data and the image to be positioned to obtain the target pose of the terminal equipment.

According to a third aspect of the present disclosure, there is provided an electronic device comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the first aspect and various possible described methods of the first aspect.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the first aspect and the various possible described methods of the first aspect.

According to a fifth aspect of the present disclosure, there is provided a computer program product comprising: a computer program, stored in a readable storage medium, from which at least one processor of an electronic device can read the computer program, the execution of which by the at least one processor causes the electronic device to perform the first aspect and the various possible described methods of the first aspect.

According to the pose processing method of the terminal equipment, the initial pose of the terminal equipment and the image to be positioned collected in the initial pose can be obtained by responding to the pose processing request. The initial pose obtained by detection is not high in precision, the target map data matched with the initial pose is inquired from the map database, the target map data can comprise pose tags and key map information associated with the pose tags, the initial pose can be optimized by using the target map data and the image to be positioned, and the more accurate target pose is obtained. By optimizing the initial pose, the problem that the pose detection of the terminal equipment is inaccurate is solved.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

fig. 1 is a schematic diagram of a system architecture of a pose processing method applied to a terminal device according to the present disclosure;

fig. 2 is a flowchart of an embodiment of a pose processing method of a terminal device according to an embodiment of the present disclosure;

fig. 3 is a flowchart of another embodiment of a pose processing method of a terminal device according to an embodiment of the present disclosure;

FIG. 4 is an exemplary diagram of object pose acquisition provided by an embodiment of the present disclosure;

fig. 5 is a diagram of a pose processing example provided according to an embodiment of the present disclosure;

fig. 6 is a schematic structural diagram of a pose processing method of a terminal device according to still another embodiment of the present disclosure;

FIG. 7 is a diagram illustrating an example of map data establishment in a map database according to an embodiment of the present disclosure;

fig. 8 is a schematic structural diagram of a pose processing method according to yet another embodiment of the present disclosure;

fig. 9 is a schematic structural view of an embodiment of a pose processing apparatus provided according to an embodiment of the present disclosure;

fig. 10 is a block diagram of an electronic device for implementing a pose processing method of a terminal device of the embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

The disclosure provides a pose processing method, a pose processing device, a pose processing medium and a pose processing product of terminal equipment, relates to the field of artificial intelligence, particularly relates to technologies such as automatic driving, intelligent transportation, deep learning and intelligent cabins, and can be applied to software development scenes.

It should be noted that the pose processing method, apparatus, device, medium, and product of the terminal device provided by the present disclosure may be applied to various driving assistance scenes, in particular, vehicle navigation and positioning scenes during vehicle driving.

Generally, the pose recognition of the terminal device is generally realized by positioning devices such as a GPS, but in practical application, when a vehicle passes through a tunnel or other road with weak signals, the detection of the positioning signals of the vehicle is inaccurate. Therefore, there is a need for a terminal device that has other solutions to supplement the existing positioning solutions. A more common positioning scheme is to use an image sensing manner to match a sensed image with an existing map, that is, store map segments to form a map database, compare an image acquired in real time, such as a panoramic image, with a map image in the database, and use the acquired position of the successfully compared map image as a positioning result of the vehicle. But positioning by means of image matching has low efficiency and poor positioning speed.

In order to solve the above problem, the pose processing method of the terminal device according to the present disclosure considers that an IMU (Inertial Measurement Unit) is used to measure the vehicle position. However, in practical application, the measurement result precision of the IMU is not high, and in order to solve the problem, a sensing image is adopted to perform positioning optimization on the IMU so as to obtain a more accurate pose.

Accordingly, in the embodiment of the disclosure, the initial pose of the terminal device and the to-be-positioned image acquired by the initial pose can be obtained in response to the pose processing request of the terminal device. And searching target map data matched with the initial pose from a map database to realize the matching of the map data and the image to be positioned. The target map data can be used for recording the pose tags and key map information associated with the pose tags, so that the initial pose can be optimized according to the target map data and the image to be positioned, and a more accurate target pose can be obtained. The initial pose can be optimized through the target map data, so that the initial pose and the image to be positioned acquired in real time are optimized, and the accuracy of the target pose is improved.

The technical solution of the present disclosure will be described in detail with reference to the accompanying drawings.

As shown in fig. 1, a system architecture diagram of a pose processing method applied to a terminal device provided by the present disclosure may include a terminal device 1, a map database 2, and an electronic device 3. At least one piece of map data may be stored in the map database 2. The map data may be used to record pose tags and key map information associated with the pose tags.

Among them, the terminal device 1 may be configured with a positioning sensor and an image sensor. The positioning sensor can acquire the initial pose of the terminal device 1 in real time. The image sensor can acquire an image to be positioned of the terminal equipment in real time.

The terminal device 1 may initiate a pose processing request to the electronic device 3, and upload the initial pose and the image to be positioned.

The electronic device 3 may obtain the initial pose of the terminal device and the image to be positioned acquired at the initial pose in response to the pose processing request. Thereafter, target map data that matches the initial pose may be queried from the map database 2. And optimizing the initial pose according to the target map data and the image to be positioned to obtain the target pose of the terminal equipment.

In addition, in practical application, the electronic device 3 can also perform pose application such as map navigation and path planning on the terminal device 1 according to the target pose, so as to realize high-precision real-time positioning processing on the terminal device.

As shown in fig. 2, which is a flowchart of an embodiment of a pose processing method for a terminal device according to an embodiment of the present disclosure, an execution subject of the pose processing method for a terminal device according to this embodiment is a pose processing apparatus. The method may comprise the steps of:

step 201: and responding to the pose processing request of the terminal equipment, and obtaining the initial pose of the terminal equipment and the image to be positioned collected in the initial pose.

Optionally, the terminal device may serve as an electronic device to configure the pose processing method of the present disclosure, so as to implement actual acquisition of the terminal device. Certainly, the terminal device can be a terminal separated from the electronic device, and the pose processing method of the terminal device can be realized through the electronic device. In practical application, the terminal device can be a device in an automatic driving vehicle, an auxiliary driving vehicle and a cockpit, and processing such as pose acquisition of various vehicles or cockpits and navigation based on the pose is realized.

The pose processing request can be obtained by detection of the terminal equipment. The terminal device may generate the pose processing request when detecting that the positioning signal strength of the positioning device such as a GPS is smaller than the signal strength threshold. The pose processing request may also be triggered by the user.

A pose sensor, such as an IMU, can be configured in the terminal equipment, and the initial pose of the terminal equipment is detected through the pose sensor. The terminal equipment can be also provided with an image sensor, such as a camera, and an image to be positioned of the terminal equipment in an initial pose can be acquired through the image sensor. The acquisition time of the initial pose and the image to be positioned can be the same.

Alternatively, the initial pose and the image to be positioned may be packaged in a pose processing request, and the initial pose and the image to be positioned of the terminal device may be obtained by resolving the pose processing request. The pose processing request can be a request for performing pose optimization on the initial pose of the terminal equipment through the image to be positioned.

Step 202: and inquiring target map data matched with the initial pose from a map database, wherein the map data in the map database is used for recording the pose tag and key map information associated with the pose tag.

At least one piece of map data may be included in the map database. Each piece of map data may be used to record pose tags and key map information for pose tag keys.

In practical application, the map image can be acquired through the pose tag, but the storage cost of the map image is too high, so that key map information of the map image can be extracted and stored, and the storage cost of the map database is reduced. Meanwhile, the matching of the pose can be quickly finished through the association of the label and the key map information, and the quick query of the target map data is realized.

Alternatively, the pose can be used for describing the position and the posture of the terminal device under a specified coordinate system, specifically, the pose can comprise coordinate points and orientation, and the orientation can be represented by a three-dimensional vector. The coordinate points may be represented using three-dimensional coordinate points or two-dimensional coordinate points. The key map information of the present disclosure may be three-dimensional map information of a three-dimensional key point, and thus, a key point coordinate in the key map information may refer to a three-dimensional coordinate point.

Step 203: and optimizing the initial pose according to the target map data and the image to be positioned to obtain the target pose of the terminal equipment.

Optionally, in step 203, difference analysis of image features may be specifically performed according to the target map data and the image to be positioned, so as to obtain an optimization result of the initial pose, that is, the target pose.

In the embodiment of the disclosure, the initial pose of the terminal device and the to-be-positioned image acquired by the initial pose can be acquired in response to the pose processing request of the terminal device. And the target map data matched with the initial pose is inquired from the map database, so that the map data is matched with the image to be positioned. The target map data can be used for recording the pose tags and key map information associated with the pose tags, so that the initial pose can be optimized according to the target map data and the image to be positioned, and a more accurate target pose can be obtained. The initial pose can be optimized through the target map data, so that the initial pose and the image to be positioned acquired in real time are optimized, and the accuracy of the target pose is improved.

As shown in fig. 3, a flowchart of another pose processing method for a terminal device according to an embodiment of the present disclosure is provided, where on the basis of the foregoing embodiment, an initial pose is optimized according to target map data and an image to be positioned, so as to obtain a target pose of the terminal device, and the method includes:

step 301: and carrying out discrete sampling on the initial pose from different directions to obtain at least one candidate pose.

Optionally, step 301 may specifically include: and performing discrete sampling on the initial pose from different directions by adopting a pose sampling algorithm to obtain at least one candidate pose. The pose sampling algorithm may be a sampling strategy that defines the direction and each direction of pose sampling. For example, in practical applications, sampling may be performed from four directions, namely, lateral, longitudinal, height, and angle, and each direction may define a corresponding sampling strategy. For example, the horizontal direction may be sampled by 1 horizontal step, and the angle may be sampled by 2 angular steps. The step length in each direction can be set according to actual requirements.

Alternatively, the initial pose may be discretely sampled from four directions, namely, the transverse direction (X), the longitudinal direction (Y), the height (Z), and the angle (ψ), to obtain candidate poses in the four directions. When sampling from four directions, lateral, longitudinal, height and angle, at least one candidate pose may use the notation: vt = { Nx, ny, nz, no }. Where Nx is a horizontal candidate pose, ny is a vertical candidate pose, nz is a height candidate pose, and No is an angle candidate pose. The transverse direction may correspond to an X-axis, the longitudinal direction may correspond to a Y-axis, the height may correspond to a Z-axis, and the angle may correspond to a rotation angle. The angle may include, for example, at least one of a yaw angle, a pitch angle, and a roll angle.

Step 302: and determining pose weights corresponding to at least one candidate pose respectively according to the target map data and the image to be positioned.

Alternatively, the pose weight of each candidate pose may refer to the degree of expression of each candidate pose in the corresponding direction. The larger the pose weight is, the higher the candidate pose importance of the corresponding direction is, and the smaller the pose weight is, the lower the candidate pose importance of the corresponding direction is.

Step 303: and carrying out pose weighted summation on at least one candidate pose according to pose weights respectively corresponding to the at least one candidate pose to obtain the target pose of the terminal equipment.

Optionally, step 303 may specifically include: and multiplying each candidate pose by the pose weight corresponding to the candidate pose to obtain the adjusted pose of each candidate pose, and adding the adjusted poses of each candidate pose to obtain the target position of the terminal equipment.

Of course, after obtaining pose weights corresponding to at least one candidate pose, respectively, on the basis of the step 302, the candidate pose with the largest pose weight may be used as the target pose, so as to achieve pose acquisition with the largest weight.

In the embodiment of the disclosure, the initial pose can be discretely sampled from different directions to obtain at least one candidate pose, and the at least one candidate pose is an extension of the initial pose. The pose weights corresponding to at least one candidate pose can be determined through the target map data and the image to be positioned, so that the weight of each candidate pose is influenced by the target map data and the image to be positioned, and the weight estimation accuracy and timeliness of each candidate pose are higher in real time. And further, by using the pose weight of each candidate pose, weighting and summing the poses of at least one candidate pose to obtain the target pose of the terminal equipment. The initial pose can be expanded from various angles through a discrete sampling mode, the optimization of the initial pose from different angles is realized, and a more accurate target pose is obtained.

Further, on the basis of any of the above embodiments, determining pose weights corresponding to at least one candidate pose according to the target map data and the image to be positioned, respectively, includes:

determining pose errors respectively corresponding to at least one candidate pose according to key map information of the image to be positioned and the target map data;

and performing regression calculation on the pose errors corresponding to the at least one candidate pose respectively to obtain the pose weights corresponding to the at least one candidate pose respectively.

For ease of understanding, as an example map of the object position acquisition shown in fig. 4, at least one piece of map data may be stored in the map database. After the initial pose is determined, the target map data can be searched and obtained from the map database through the initial pose. The target map data can be combined with the image characteristics of the image to be positioned to determine pose weights V corresponding to the candidate poses _w And further based on pose weight V of each candidate pose _w For at least one candidate pose V _T And weighting and summing to obtain the target pose. The target pose may be expressed as:

optionally, performing regression calculation on pose errors corresponding to at least one candidate pose, and obtaining pose weights corresponding to at least one candidate pose may include: and normalizing the pose errors corresponding to at least one candidate pose by using a regression algorithm to obtain the pose weights corresponding to at least one candidate pose.

In this embodiment, when obtaining the pose weight of each candidate pose, the pose error corresponding to at least one candidate pose may be determined according to the key map information of the image to be located and the target map data, and the pose error may measure the accuracy of each pose. By acquiring the pose error corresponding to each candidate pose, the corresponding pose estimation can be realized for each candidate pose, and the pose estimation accuracy is improved.

Further, on the basis of any of the above embodiments, the key map information includes key points and key point features;

determining pose errors respectively corresponding to at least one candidate pose according to key map information of the image to be positioned and the target map data, wherein the determining comprises the following steps:

extracting a first image characteristic of an image to be positioned through a characteristic extraction network, wherein the first image characteristic is used for recording content information of the image to be positioned;

respectively mapping the key points to an image to be positioned according to at least one candidate pose to obtain mapping key points corresponding to at least one candidate pose;

determining mapping characteristics of mapping key points corresponding to at least one candidate pose respectively according to the first image characteristics;

and determining pose errors corresponding to the at least one candidate pose respectively according to the mapping characteristics of the mapping key points in the at least one candidate pose respectively and by combining the characteristics of the key points.

The first image feature may be used to record content information of the image to be positioned, and may characterize key position information of the image to be positioned. Assuming that the image to be positioned is represented by the symbol I, the first image feature may be represented as F = { F (I) }, where F () represents the feature extraction network. The characteristics of a certain pixel point in the image I can be expressed as: fj, j refers to the jth pixel point, and Fj refers to the jth pixel point characteristic.

The key points in the key map information may be three-dimensional key points, and each key point may be associated with a key point feature. The key points are respectively mapped into the image to be positioned according to at least one candidate pose, namely, the key points are respectively mapped into a two-dimensional image coordinate system from a three-dimensional space coordinate system according to at least one candidate pose, and mapping key points corresponding to at least one candidate pose are obtained. Each key point can be mapped at least once according to at least one candidate pose to obtain at least one mapping key point, so that one key point can be associated with the mapping key point corresponding to at least one candidate pose respectively.

According to the first image features, determining mapping features of mapping key points corresponding to at least one candidate pose respectively can refer to extracting the mapping features of the mapping key points from the first image features.

In the embodiment of the disclosure, the specific content information of the image to be positioned can be represented by using the first image feature, so that after the key points of the target map data are respectively mapped to the image to be positioned according to at least one candidate pose, the mapping key points of each candidate pose can be obtained, the mapping key points of each candidate pose can be used for representing the position of each candidate pose mapped to the two-dimensional image, the mapping feature of the mapping key points of each candidate pose can be determined according to the first image feature of the image to be positioned, the extraction of the mapping feature of each candidate pose is realized, and the pose error corresponding to each candidate pose is determined by combining the key point features, so that the pose error is determined according to the mapping feature and the key point feature of each candidate pose, the pose error of each candidate pose is accurately represented by using the error between the features, the accuracy is higher, and the representation effect is better.

Further, on the basis of any one of the above embodiments, the key points include at least one, and the key map information further includes: the weights of at least one key point are respectively corresponding to the key points;

determining pose errors corresponding to at least one candidate pose according to the mapping characteristics of the mapping key points in the at least one candidate pose and by combining the characteristics of the key points, wherein the determining pose errors comprise the following steps:

determining feature differences of at least one key point corresponding to at least one candidate pose respectively according to the mapping feature and the key point feature of the at least one key point at each candidate pose respectively;

determining the feature difference corresponding to each candidate pose at least one key point according to the feature difference corresponding to each candidate pose at least one key point;

and performing weighted aggregation on the feature difference corresponding to each candidate pose at least one key point according to the weight corresponding to each key point to obtain pose errors corresponding to each candidate pose.

For ease of understanding, as in the pose processing example diagram shown in fig. 5, the key points may include N, where N is a positive integer greater than or equal to 1. Referring to 5,N key points, P1, P2 and P3 \8230, where \8230, pn may be mapped respectively by at least one candidate pose to obtain mapping characteristics of each key point corresponding to at least one candidate pose, and it is assumed that 4 candidate poses, which are Nx, ny, nz and No, exist. Then, feature difference calculation can be performed on the key point features and the mapping features, so that feature differences corresponding to at least one candidate pose of each key point can be obtained, and the feature differences can be a multi-dimensional matrix. For any keypoint i, assume that the keypoint feature of that keypoint uses H _i Showing that the mapping feature corresponding to the key point i in the mapping key point of the candidate pose uses F _i Represents, therefore, the feature difference C between this keypoint i and its mapped keypoint correspondence at the candidate pose _i Can be expressed as:

C _i ＝||H _i -F _i ||

and the mapping key point feature Fi is derived from the first image feature F of the image to be positioned. Assuming feature difference usage of a keypoint in each of at least one candidate pose

And (4) showing.

The weights W1, W2 and W3 of each key point \8230Wnare known, the feature difference and the key point weight of at least one key point can be weighted and summed to obtain the weight feature corresponding to each key point, and the weight feature can be represented by the following formula and can also be called a cost volume:

V _c ＝∑W _i T _Ci

the weight characteristic of each key point is the characteristic difference of each key point corresponding to at least one candidate pose, and the weight matrix is expressed based on the dimension of the key points, so that the characteristic difference of each candidate pose corresponding to at least one key point can be obtained from the dimension of the candidate pose. Based on the description, it can be known that, from the dimension of the candidate pose, each candidate pose is aggregated at least one key point to obtain pose errors corresponding to at least one candidate pose respectively.

Optionally, feature differences corresponding to each candidate pose at least one key point may be weighted and aggregated from the pose dimension according to weights corresponding to the at least one key point, so that the feature differences corresponding to the at least one key point are multiplied by the weights to obtain weight features corresponding to the at least one key point, and the weight features corresponding to the at least one key point are added to obtain the pose error corresponding to the candidate pose.

In the embodiment of the disclosure, during pose error acquisition, feature differences corresponding to at least one key point at least one candidate pose can be determined according to the mapping features and the key point features of at least one key point at each candidate pose, and further, the feature differences corresponding to at least one key point at each candidate pose are determined according to the feature differences corresponding to at least one key point at each candidate pose, so that conversion from the feature differences of the key points to the feature differences of the candidate poses is realized. Feature differences corresponding to each candidate pose at least one key point can be aggregated to obtain pose errors corresponding to each candidate pose, the pose errors of each candidate pose are learned to the respective feature errors of at least one key point through the weighted aggregation of the key points in pose dimensions, more comprehensive feature errors can be contained, and the accurate acquisition of the pose errors of the candidate poses is realized.

Further, optionally, the obtaining of the mapping feature of the mapping key point includes:

determining adjacent key points which are associated with the mapping key points in the image to be positioned through an interpolation algorithm;

and determining the mapping characteristics of the mapping key points according to the characteristic values of the adjacent key points in the first image characteristics and the characteristic values of the mapping key points in the first image characteristics.

Alternatively, the interpolation algorithm may include: a bilinear interpolation algorithm, a nearest neighbor interpolation algorithm and the like, and adjacent key points related to the mapping key points can be obtained through the interpolation algorithm. Feature value calculation can be performed on adjacent key points and mapping key points in a pooling (posing) calculation mode to obtain mapping features of the mapping key points. And calculating the characteristic values of the adjacent key points and the mapping key points in a mean value calculation mode to obtain the mapping characteristics of the mapping key points.

In this embodiment, the interpolation algorithm is used to extract the adjacent key points associated with the mapping points in the image to be located, so as to perform feature value calculation of the mapping key points by using the feature values of the adjacent key points in the first image features and the feature values of the mapping key points in the first image features, so that the feature calculation process of the mapping key points is affected by the surrounding feature points, and therefore, the method covers wider image features and has higher accuracy.

Further, on the basis of any of the above embodiments, querying target map data matching the initial pose from a map database includes:

determining a target pose label with the minimum position difference with the initial pose from at least one piece of map data according to the pose labels corresponding to at least one piece of map data in the map database;

and determining the map data corresponding to the target pose tag as target map data.

Optionally, the position in the pose tag corresponding to each of the at least one piece of map data may be compared with the position of the initial pose to obtain a position difference corresponding to each of the at least one piece of map data, so that the pose tag in the map data with the small position difference is used as the target pose tag.

In the embodiment of the disclosure, the position tags corresponding to at least one piece of map data can be used to perform position comparison with the initial position and pose respectively, so as to obtain the position tag with the smallest position difference from the initial position and pose in at least one piece of map data as the target position and pose tag, and the obtained position and pose difference between the target position and pose is the position tag with the highest degree of adaptation to the initial position and pose, thereby obtaining the accuracy of the accurate position and pose tag.

As shown in fig. 6, a flowchart of an embodiment of a pose processing method for a terminal device according to an embodiment of the present disclosure is provided. On the basis of any of the above embodiments, the step of obtaining map data in the map database may include:

step 601: and determining a pose tag.

Alternatively, the pose tag may be a pose captured by a high precision positioning device. The process of establishing the map data corresponding to the pose tag can be executed in advance.

Step 602: and acquiring point cloud data and a map image at the map position indicated by the pose tag.

Alternatively, the point cloud data may refer to three-dimensional point cloud data acquired by Lidar (Laser Radar), and may include a plurality of three-dimensional position points. The map image may refer to an image of a real scene captured by an image sensor. The image sensor corresponding to the map image and the image sensor corresponding to the image to be located may be the same. The three-dimensional location points may include the depth of the point in addition to the coordinate points.

Step 603: and extracting key point information of the pose tag by using the point cloud data and the map image.

Step 604: and determining map data in the map database according to the incidence relation between the pose tag and the key point information.

Alternatively, the kth piece of map data may be represented as:

{T _k ：{P ₁ ，F ₁ ，W ₁ }，…，{P _i ，F _i ，W _i }，…，{P _n ，F _n ，W _n }}

wherein, T _k Can be a pose tag, P ₁ …P _i …P _n N keypoints for this pose tag association may be made. P _i Can be the location of a keypoint, F _i May be a key point feature, W _i May be a keypoint weight. According to the formula, one piece of map data can be composed of position and pose labels and the positions, the characteristics and the weights of a plurality of key points.

In this embodiment, a pose tag may be determined, and point cloud data and a map image may be simultaneously acquired through a map location indicated by the pose tag. The key point information of the pose tag can be extracted through the point cloud data and the map image, namely the key point information is obtained through extraction of three-dimensional point cloud data and a two-dimensional map image, so that the key point information integrates the spatial geometrical characteristics of the pose tag in three-dimensional and two-dimensional angles, and the recording accuracy of the key content of the key point is higher. Therefore, the map data represented by the incidence relation established according to the pose tag and the key point information has higher accuracy.

Further, on the basis of any of the above embodiments, extracting key point information of a pose tag by using the point cloud data and the map image includes:

extracting key points from the point cloud data;

mapping the key points to a map image according to the pose labels to obtain map mapping points;

determining key point characteristics and key point weights of key points according to the map image and the map mapping points;

and determining key point information consisting of key points, key point features and key point weights.

Alternatively, the map mapping point may be a coordinate point in a coordinate system of the map image.

In the embodiment of the disclosure, key points can be extracted from point cloud data, the key points are mapped to a map image according to a pose tag to obtain map mapping points, the map mapping points can enter a two-dimensional space through mapping of the pose tag, key point features and key point weights are obtained in the two-dimensional map image and the map mapping points, and then key point information of key points, key point features and key point weight combinations is determined, so that rapid establishment of the key point information is realized.

Further, on the basis of any of the above embodiments, determining the key point features and the key point weights of the key points according to the map image and the map mapping points includes:

extracting a second image characteristic of the map image through a characteristic extraction network, wherein the second image characteristic is used for recording the content information of the map image;

extracting the characteristics of the map mapping points from the second image characteristics to obtain the key point characteristics of the key points;

according to the second image characteristics, carrying out weight calculation on each pixel point of the map image to obtain a weight map corresponding to the map image;

and extracting the weight of the map mapping point from the weight map to obtain the key point weight of the key point.

For ease of understanding, fig. 7 shows a diagram of an example of establishment of map data in the map database. Referring to fig. 7, the point cloud data may be extracted from the first and second keypoints by combining two keypoint extraction methods, i.e., keypoint prediction and farthest point sampling, to obtain a final keypoint. And the map image may obtain the second image feature through the feature extraction network. The second image feature can be used for acquiring a weight map and a feature map, and further map data can be determined through the weight map, the feature map and key points.

Optionally, performing weight calculation on each pixel point of the map image according to the second image feature to obtain a weight map corresponding to the map image may include performing convolution calculation on the second image feature to obtain a weight value of each pixel point in the map image, so as to obtain a weight map corresponding to the weight value of each pixel point.

Optionally, extracting the weight of the map mapping point from the weight map to obtain the key point weight of the key point, including: and determining a coordinate point of the map mapping point in the weight map, reading the weight of the map mapping point according to the coordinate point of the map mapping point in the weight map, and taking the weight of the map mapping point as the weight of the corresponding key point.

In the embodiment of the disclosure, the second image feature of the map image is extracted, the feature of the map mapping point is extracted from the second image feature, and the feature of the map mapping point is further used as the key point feature of the key point, so that the key point feature is accurately extracted. And performing weight calculation on each pixel point of the map image according to the second image characteristics to acquire a weight map of the map image, extracting the weights of the map mapping points through the weight map, and acquiring the key point weights of the key points based on the corresponding relation between the map mapping points and the key points. Therefore, through the characteristic extraction mode, the weight and the characteristic are accurately and quickly obtained from the characteristic analysis angle by the characteristic extraction and the weight calculation of each characteristic point.

Further, on the basis of any of the above embodiments, performing weight calculation on each pixel point of the map image according to the second image feature to obtain a weight map corresponding to the map image, includes:

and utilizing the map mapping points to constrain the weight calculation process of each pixel point of the map image in the second image characteristic, and obtaining a weight map corresponding to the map image.

As shown in fig. 7, after the extraction of the key points, the learning of the weight map corresponding to the second image feature may be supervised, so that the weight representation of the weight map is more accurate.

Alternatively, a constraint map of the same size may be generated based on map-mapped points in accordance with the size of the map image, specifically, the map-mapped point is set to 1 in the constraint map at the map image location point and the non-map-mapped point is set to 0 in the constraint map to obtain a constraint map marked with the map-mapped point. And constraining the weight calculation process of each pixel point of the map image through the constraint graph to obtain a more accurate weight graph.

In the embodiment of the present disclosure, the weight calculation process of each pixel point in the second image feature may be constrained by using the map mapping point, so as to obtain a weight map of the map image. The weight calculation process is restrained by the map mapping points, so that the obtaining process of the weight map of the map image is influenced by the obtained key points, the weight learning of each pixel point in the map image is more accurate, and the obtained weight map is higher in precision.

Further, optionally, the point cloud data comprises at least one three-dimensional data point, and extracting key points from the point cloud data comprises:

extracting a first key point from at least one three-dimensional data point of the point cloud data by using a key point extraction network;

determining a second key point through a farthest point sampling algorithm;

and the key determining submodule is used for determining the first key point and the second key point as key points in the point cloud data.

Optionally, extracting a first keypoint from at least one three-dimensional data point of the point cloud data using the keypoint extraction network may include: extracting candidate points from at least one three-dimensional data point of the point cloud data by using a key point extraction network; projecting the candidate points to a map image based on the pose tags to obtain projection points corresponding to the candidate points; and determining candidate points corresponding to the projection points meeting the screening condition as first key points.

Optionally, the keypoint extraction network may score the significance of each position point in the point cloud data in the neighborhood, and may obtain the key scores corresponding to the plurality of position points in the point cloud data, respectively, through the keypoint extraction network, so as to select, according to the key scores corresponding to the plurality of position points, a position point whose score is greater than or equal to a score threshold value as a candidate point. In practical applications, the keypoint extraction network may include a classification Convolution network such as a multi-layer KPConv (Kernel Point Convolution) network using a series of local three-dimensional Convolution kernels).

In this embodiment, when the pose tag is represented by T, the camera internal reference K may be used as the camera internal reference known by default. Let the point cloud data be represented by P = { P1, P2, P3 \8230; pn }. The map image is represented using I = { q1, q2, q3 \8230; qn }. Wherein, when projecting the 3D point in the point cloud data to the image plane, the projection point can be represented as:

wherein z is the depth of the point cloud data, T is the pose, K is the camera internal reference, p _k Is a location point in the point cloud data,

k is a positive integer of 1 to n.

And if the distance between the projection point and the nearest pixel point is smaller than the distance threshold, determining that the projection point meets the screening condition, and taking the candidate point corresponding to the projection point as the first projection point.

Optionally, after obtaining the first keypoints, in order to make the keypoints richer, the second keypoints may be determined by a farthest point sampling algorithm. Specifically, the position points in the point cloud data except the first key point may be subjected to farthest point sampling to obtain a second key point.

According to the embodiment of the disclosure, when the key points of the point cloud data are extracted, the network can be extracted from the key points to primarily extract the key points, so that the first key points are obtained. In order to enable the extracted key points to more accurately represent the contour points of the point cloud data, the farthest point sampling can be performed from the remaining three-dimensional data points except the first key point to obtain the second key point. And then taking the first key point and the second key point as key points extracted from the point cloud data. The key points are obtained based on a key point extraction network and a farthest sampling point algorithm, key information of two angles of the inside of the point cloud data and the farthest point is represented, the key points can be represented more accurately, and the key point representation efficiency and accuracy are improved.

Further, on the basis of any of the above embodiments, after obtaining the target pose of the terminal device, the method further includes:

and determining the driving path of the terminal equipment according to the position indicated by the target pose and the destination of the terminal equipment.

According to the position indicated by the target pose and the destination of the terminal equipment, the path navigation can be performed on the terminal equipment to obtain the running path of the terminal equipment, the accuracy of the obtained target pose is higher after the initial pose is optimized on the basis of obtaining the initial pose, more accurate navigation application can be realized, and the navigation efficiency and accuracy of the equipment are improved.

In practical application, the key point prediction network and the farthest point sampling algorithm related to the pose processing method may be obtained by training, and may specifically adopt a supervised or unsupervised training mode, which is not described herein again.

Furthermore, the present solution also relates to: and training networks such as a feature extraction network, a weight graph and a learning network of the feature graph.

As shown in fig. 8, a flowchart of a pose processing method for a terminal device according to another embodiment of the present disclosure may include the following steps:

step 801: and determining a training sample, wherein the training sample can be associated with the initial pose and the real pose.

Step 802: and inquiring target map data matched with the initial pose of the training sample from the map database. Map data in the map database is used to record pose tags and key map information associated with the pose tags.

Step 803: and optimizing the initial pose according to the target map data and the training sample to obtain the target pose of the training sample.

In practical applications, regarding steps 802 to 803, the processing procedure adopted may be the same as the pose processing method shown in the embodiment of fig. 2 and the like, and is not described herein again for the sake of brevity of description.

Step 804: and determining pose loss according to the target pose and the real pose of the training sample by combining a pose loss function.

Step 805: and based on the pose loss, adjusting network parameters of the network subjected to optimization processing on the initial pose until the pose loss meets the convergence condition.

In the embodiment of the disclosure, the network parameters of the network which is optimized for the initial pose are adjusted through the training samples until the pose loss meets the convergence condition. The pose optimization processing process is trained in a pose loss estimation mode, an accurate optimization processing network can be obtained, and pose optimization accuracy and precision are improved.

In the training process, the difference from the above embodiment is that the image to be positioned can be replaced by a training sample, the training sample can be associated with the initial pose and the real pose, the method of the above embodiment can be used to obtain the target pose corresponding to the initial pose, and the pose loss is determined based on the target pose and the real pose in combination with the pose loss function. And adjusting network parameters based on the pose loss and the acquisition of the feature extraction network and the pose weight until the pose loss meets the convergence condition.

The pose loss function referred to in this application is exemplified with at least one candidate pose as lateral, longitudinal, height, and angle 4 directions.

Alternatively, when the error calculation is performed purely by pose, the pose loss function can be expressed as:

wherein, the first and the second end of the pipe are connected with each other,

may be characteristic differences corresponding to a lateral candidate pose, greater or lesser than>

Characteristic difference corresponding to the longitudinal candidate pose can be considered>

Characteristic difference corresponding to the candidate pose of height can be considered>

The feature difference corresponding to the candidate pose of the angle may be.

The target pose and the real pose of the training sample can be obtained through calculation.

Δx ^* ，Δy ^* ，Δz ^* ，Δψ ^* The characteristic difference of the truth value corresponding to the transverse direction, the longitudinal direction, the height and the angle can be obtained by the initial training of the sampleAnd (5) calculating and obtaining the pose and the real pose.

In practical application, the weight of each candidate pose can be also monitored, and the distribution error of the weight of each candidate pose can be used as a pose loss function. The weight distribution of each candidate pose can be expressed as:

L ₂ ＝β(∑ _i P(Δx _i )|Δx _i -Δx ^* |+(∑ _j P(Δy _j )|Δy _j -Δy ^* |

+(∑ _k P(Δz _k )|Δz _k -ΔZ ^* |+(∑ _l P(Δψ _l )|Δψ _l -Δψ ^* |

wherein, P (Δ x) _i ) As a lateral weight, P (Δ y) _j ) As vertical weight, P (Δ z) _k ) For high weight, P (Δ ψ) _l ) Is the angular weight.

In practical application, L1 and L2 may be combined with an error calculation formula as a whole, and the candidate poses and the weights of the candidate poses are learned and adjusted through L1 and L2.

In addition, in practical application, at least one of networks such as a key point prediction network, a farthest point sampling algorithm, a weight graph and a feature graph learning network can be adjusted and trained in real time through pose loss. The weight map can participate in the key point extraction process, namely, the weight map can be used for carrying out feedback adjustment on the key point prediction network, the farthest point sampling network, the weight map and the learning network of the feature map. The adjustment can be made in particular by a penalty function of the weight map.

Alternatively, the penalty function of the weight map may be expressed as: l is a radical of an alcohol ₀ ＝||B-B ^* ||

Wherein, V ^* The weight map is a real weight map, and B is a weight map obtained by learning.

In practical application, the projection features extracted by the feature extraction network can also participate in the training process. The loss function of the projected features can be expressed as: l is ₃ ＝∑ _i |H _i -F _i |

Wherein i represents offKey point, F _i Features of the map representing key points, H _i Representing keypoint features in the map data.

Therefore, in one possible design, the loss function used in the training process of the pose processing method can be expressed as: l0+ L1+ L2+ L3.

Through calculation of the loss function, the integral pose loss can be obtained, and the learning network, the image extraction network and the like of the characteristic diagram and the weight diagram are adjusted through the integral pose loss.

As shown in fig. 9, which is a schematic structural diagram of an embodiment of a pose processing apparatus of a terminal device according to an embodiment of the present disclosure, the pose processing apparatus provided in this embodiment may execute the pose processing method of the terminal device shown in fig. 2. The apparatus 900 may comprise the following units:

the response unit 901 is configured to: responding to a pose processing request of the terminal equipment, and acquiring an initial pose of the terminal equipment and an image to be positioned acquired at the initial pose;

the query unit 902: for: inquiring target map data matched with the initial pose from a map database, wherein the map data in the map database are used for recording pose tags and key map information associated with the pose tags;

an optimization unit 903: for: and optimizing the initial pose according to the target map data and the image to be positioned to obtain the target pose of the terminal equipment.

As an embodiment, the optimization unit comprises:

the pose sampling module is used for carrying out discrete sampling on the initial pose from different directions to obtain at least one candidate pose;

the weight determining module is used for determining pose weights corresponding to at least one candidate pose respectively according to the target map data and the image to be positioned;

and the pose weighting module is used for carrying out pose weighted summation on at least one candidate pose according to pose weights respectively corresponding to the at least one candidate pose to obtain a target pose of the terminal equipment.

As yet another embodiment, the weight determination module includes:

the error submodule is used for determining pose errors corresponding to at least one candidate pose according to key map information of the image to be positioned and the target map data;

and the regression submodule is used for carrying out regression calculation on the pose errors corresponding to the at least one candidate pose respectively to obtain the pose weight corresponding to the at least one candidate pose respectively.

As yet another embodiment, the key map information includes key points and key point features;

an error sub-module, specifically configured to:

As still another embodiment, the key points include at least one, and the key map information further includes: the weights corresponding to at least one key point respectively;

an error submodule, specifically configured to:

and carrying out weighted aggregation on the characteristic difference corresponding to each candidate pose at least one key point according to the weight corresponding to each key point, so as to obtain the pose error corresponding to each candidate pose.

As yet another embodiment, the error submodule is further to:

As yet another embodiment, a query unit, comprising:

the difference query module is used for determining a target pose label with the minimum position difference with the initial pose from at least one piece of map data according to the pose labels respectively corresponding to at least one piece of map data in the map database;

and the target determining module is used for determining the map data corresponding to the target pose tag as target map data.

As still another embodiment, further comprising:

a determination unit configured to determine a pose tag;

the acquisition unit is used for acquiring point cloud data and a map image at the map position indicated by the pose label;

the extraction unit is used for extracting key point information of the pose tag by using the point cloud data and the map image;

and the association unit is used for determining map data in the map database according to the association relationship between the pose tag and the key point information.

As still another embodiment, the extraction unit includes:

the key extraction module is used for extracting key points from the point cloud data;

the key mapping module is used for mapping the key points to the map image according to the pose labels to obtain map mapping points;

the characteristic weight module is used for determining key point characteristics and key point weights of key points according to the map image and the map mapping points;

and the information determining module is used for determining key point information consisting of key points, key point characteristics and key point weights.

As yet another embodiment, a feature weight module includes:

the first extraction submodule is used for extracting a first key point from at least one three-dimensional data point of the point cloud data by using a key point extraction network;

the second extraction submodule is used for determining a second key point through a farthest point sampling algorithm;

As another embodiment, the weight learning sub-module is specifically configured to:

As yet another embodiment, the point cloud data includes at least one three-dimensional data point, and the key extraction module includes:

As still another embodiment, further comprising:

and the navigation unit is used for determining the running path of the terminal equipment according to the position indicated by the target pose and the destination of the terminal equipment.

It should be noted that the user equipment in this embodiment is not specific to a particular user, and cannot reflect personal information of a particular user.

In the technical scheme of the disclosure, the collection, storage, use, processing, transmission, provision, disclosure and other processing of the personal information of the related user are all in accordance with the regulations of related laws and regulations and do not violate the good customs of the public order.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

According to an embodiment of the present disclosure, an electronic device provided by the present disclosure includes:

at least one processor; and

the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the pose processing method of the terminal device corresponding to any one of the above embodiments.

According to an embodiment of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute a pose processing method of a terminal device according to any one of the above embodiments.

According to an embodiment of the present disclosure, the present disclosure also provides a computer program product comprising: a computer program, stored in a readable storage medium, from which at least one processor of the electronic device can read the computer program, the at least one processor executing the computer program causing the electronic device to perform the solution provided by any of the embodiments described above.

FIG. 10 illustrates a schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 10, the apparatus 1000 includes a computing unit 1001 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a Random Access Memory (RAM) 1003. In the RAM1003, various programs and data necessary for the operation of the device 1000 can also be stored. The calculation unit 1001, the ROM1002, and the RAM1003 are connected to each other by a bus 1004. An input/output (I/O) interface 1005 is also connected to bus 1004.

A number of components in device 1000 are connected to I/O interface 1005, including: an input unit 1006 such as a keyboard, a mouse, and the like; an output unit 1007 such as various types of displays, speakers, and the like; a storage unit 1008 such as a magnetic disk, an optical disk, or the like; and a communication unit 1009 such as a network card, a modem, a wireless communication transceiver, or the like. The communication unit 1009 allows the device 1000 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.

Computing unit 1001 may be a variety of general and/or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 1001 executes the respective methods and processes described above, such as the pose processing method of the terminal device. For example, in some embodiments, the pose processing method of the terminal device can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 1000 via ROM1002 and/or communications unit 1009. When the computer program is loaded into the RAM1003 and executed by the computing unit 1001, one or more steps of the posture processing method of the terminal device described above may be executed. Alternatively, in other embodiments, the calculation unit 1001 may be configured to perform the pose processing method of the terminal device by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The Server may be a cloud Server, also called a cloud computing Server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of high management difficulty and weak service extensibility in a traditional physical host and VPS service ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server incorporating a blockchain.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A pose processing method of a terminal device comprises the following steps:

2. The method of claim 1, wherein the optimizing the initial pose according to the target map data and the image to be positioned to obtain a target pose of the terminal device comprises:

carrying out discrete sampling on the initial pose from different directions to obtain at least one candidate pose;

determining pose weights respectively corresponding to at least one candidate pose according to the target map data and the image to be positioned;

and performing pose weighted summation on at least one candidate pose according to pose weights respectively corresponding to the at least one candidate pose to obtain the target pose of the terminal equipment.

3. The method of claim 2, wherein the determining pose weights respectively corresponding to at least one of the candidate poses according to the target map data and the image to be positioned comprises:

determining pose errors respectively corresponding to at least one candidate pose according to the key map information of the image to be positioned and the target map data;

and performing regression calculation on the pose errors corresponding to at least one candidate pose respectively to obtain the pose weight corresponding to at least one candidate pose respectively.

4. The method of claim 3, wherein the key map information includes key points and key point features;

determining pose errors respectively corresponding to at least one candidate pose according to the key map information of the image to be positioned and the target map data, wherein the determining comprises the following steps:

extracting a first image characteristic of the image to be positioned through a characteristic extraction network, wherein the first image characteristic is used for recording content information of the image to be positioned;

mapping the key points to the image to be positioned according to at least one candidate pose respectively to obtain mapping key points corresponding to the at least one candidate pose respectively;

and determining pose errors corresponding to at least one candidate pose according to the mapping characteristics of the mapping key points in at least one candidate pose and by combining the key point characteristics.

5. The method of claim 4, wherein the key points comprise at least one, the key map information further comprising: the weights of at least one key point are respectively corresponding to the key points;

determining pose errors corresponding to at least one candidate pose according to the mapping features of the mapping key points in the at least one candidate pose and by combining the key point features, wherein the determining comprises the following steps:

determining feature differences of at least one key point corresponding to at least one candidate pose according to the mapping feature and the key point feature of the at least one key point at each candidate pose;

and carrying out weighted aggregation on the feature difference corresponding to each candidate pose at least one key point according to the weight corresponding to each key point, so as to obtain a pose error corresponding to each candidate pose.

6. The method according to claim 4 or 5, wherein the step of obtaining the mapping features of the mapping key points comprises:

determining adjacent key points in the image to be positioned, which are associated with the mapping key points, through an interpolation algorithm;

7. The method according to any one of claims 1-6, wherein said querying from a map database for target map data matching the initial pose comprises:

determining a target pose tag with the minimum position difference with the initial pose from at least one piece of map data according to pose tags respectively corresponding to at least one piece of map data in the map database;

and determining the map data corresponding to the target pose tag as the target map data.

8. The method according to any of claims 1-7, wherein the step of obtaining map data in the map database comprises:

determining a pose tag;

simultaneously acquiring point cloud data and a map image at the map position indicated by the pose tag;

extracting key point information of the pose tag by using the point cloud data and the map image;

and determining the map data in the map database according to the incidence relation between the pose tag and the key point information.

9. The method of claim 8, wherein the extracting, using the point cloud data and the map image, the pose tag's keypoint information comprises:

extracting key points from the point cloud data;

mapping the key points to the map image according to the pose labels to obtain map mapping points;

determining key point characteristics and key point weights of the key points according to the map image and the map mapping points;

and determining the key point information consisting of the key points, the key point features and the key point weights.

10. The method of claim 8 or 9, wherein said determining keypoint features and keypoint weights for said keypoints from said map image and said map mapped points comprises:

extracting a second image characteristic of the map image through a characteristic extraction network, wherein the second image characteristic is used for recording content information of the map image;

extracting features of the map mapping points from the second image features to obtain keypoint features of the keypoints;

extracting the weight of the map mapping point from the weight map to obtain the key point weight of the key point.

11. The method according to claim 10, wherein the performing weight calculation on each pixel point of the map image according to the second image feature to obtain a weight map corresponding to the map image includes:

12. The method of any of claims 9-11, wherein the point cloud data comprises at least one three-dimensional data point, and wherein extracting keypoints from the point cloud data comprises:

determining a second key point through a farthest point sampling algorithm;

13. The method according to any one of claims 1-12, wherein after obtaining the target pose of the terminal device, further comprising:

and determining a driving path of the terminal equipment according to the position indicated by the target pose and the destination of the terminal equipment.

14. A pose processing apparatus of a terminal device, comprising:

a response unit for: responding to a pose processing request of a terminal device, and obtaining an initial pose of the terminal device and an image to be positioned acquired at the initial pose;

a query unit to: inquiring target map data matched with the initial pose from a map database, wherein the map data in the map database are used for recording pose tags and key map information associated with the pose tags;

an optimization unit for: and optimizing the initial pose according to the target map data and the image to be positioned to obtain the target pose of the terminal equipment.

15. The apparatus of claim 14, wherein the optimization unit comprises:

the pose sampling module is used for performing discrete sampling on the initial pose from different directions to obtain at least one candidate pose;

the weight determining module is used for determining pose weights corresponding to at least one candidate pose according to the target map data and the image to be positioned;

and the pose weighting module is used for performing pose weighted summation on at least one candidate pose according to pose weights respectively corresponding to the at least one candidate pose to obtain the target pose of the terminal equipment.

16. The apparatus of claim 15, wherein the weight determination module comprises:

the error submodule is used for determining pose errors corresponding to at least one candidate pose according to the key map information of the image to be positioned and the target map data;

and the regression submodule is used for carrying out regression calculation on the pose errors corresponding to at least one candidate pose respectively to obtain the pose weight corresponding to at least one candidate pose respectively.

17. The apparatus of claim 16, wherein the key map information comprises key points and key point features;

the error submodule is specifically configured to:

18. The apparatus of claim 17, wherein the key points comprise at least one, the key map information further comprising: the weights of at least one key point are respectively corresponding to the key points;

the error submodule is specifically configured to:

19. The apparatus of claim 17 or 18, wherein the error sub-module is further configured to:

20. The apparatus according to any of claims 14-19, wherein the querying element comprises:

the gap query module is used for determining a target pose tag with the minimum position gap with the initial pose from at least one piece of map data according to pose tags respectively corresponding to at least one piece of map data in the map database;

and the target determining module is used for determining the map data corresponding to the target pose tag as the target map data.

21. The apparatus of any of claims 14-20, further comprising:

a determination unit for determining a pose tag;

and the association unit is used for determining the map data in the map database according to the association relationship between the pose tag and the key point information.

22. The apparatus of claim 21, wherein the extracting unit comprises:

the characteristic weight module is used for determining key point characteristics and key point weights of the key points according to the map images and the map mapping points;

and the information determining module is used for determining the key point information consisting of the key points, the key point characteristics and the key point weights.

23. The apparatus of claim 21 or 22, wherein the feature weight module comprises:

the characteristic extraction submodule is used for extracting a second image characteristic of the map image through a characteristic extraction network, and the second image characteristic is used for recording the content information of the map image;

a feature mapping sub-module, configured to extract features of the map mapping points from the second image features to obtain key point features of the key points;

the weight learning submodule is used for carrying out weight calculation on each pixel point of the map image according to the second image characteristics to obtain a weight map corresponding to the map image;

and the weight extraction submodule is used for extracting the weight of the map mapping point from the weight map so as to obtain the key point weight of the key point.

24. The apparatus of claim 23, wherein the weight learning submodule is specifically configured to:

and utilizing the map mapping points to constrain the weight calculation process of each pixel point of the map image in the second image characteristics, and obtaining a weight map corresponding to the map image.

25. The apparatus of any of claims 22-24, wherein the point cloud data comprises at least one three-dimensional data point, the key extraction module comprising:

26. The apparatus of any of claims 14-25, further comprising:

and the navigation unit is used for determining a running path of the terminal equipment according to the position indicated by the target pose and the destination of the terminal equipment.

27. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the pose processing method of the terminal device according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the pose processing method of a terminal device according to any one of claims 1 to 13.

29. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the pose processing method of a terminal device according to any one of claims 1 to 13.