WO2022246812A1

WO2022246812A1 - Positioning method and apparatus, electronic device, and storage medium

Info

Publication number: WO2022246812A1
Application number: PCT/CN2021/096828
Authority: WO
Inventors: 宋乐; 郭鑫; 李国林; 谭浩轩; 王世魏; 陈侃; 霍峰; 秦宝星; 程昊天
Original assignee: 上海高仙自动化科技发展有限公司
Priority date: 2021-05-28
Filing date: 2021-05-28
Publication date: 2022-12-01

Abstract

Disclosed in the present application are a positioning method and apparatus, an electronic device, and a storage medium. The method is applied to an electronic device, and the electronic device comprises at least one top view sensor. The method comprises: obtaining sensor data acquired by at least one top view sensor; processing the sensor data; and positioning the electronic device according to the processed sensor data. Or, the method comprises: acquiring a current pose and depth data in at least one preset direction of a robot; determining a feature point cloud on the basis of the depth data; determining an obstacle score of the feature point cloud in a preset global grid map, wherein the preset global grid map is formed on the basis of a historical point cloud; and optimizing the current pose according to the obstacle score to determine positioning information of the robot.

Description

Positioning method, device, electronic device and storage medium

technical field

The present application relates to the technical field of intelligent robots, for example, to a positioning method, device, electronic equipment and storage medium.

Background technique

With the development of society and the advancement of technology, electronic equipment has gradually entered the daily life of human beings from the laboratory. Taking electronic devices as intelligent robots as an example, service robots in daily life include mobile robots and fixed-position robots.

The necessary requirement for the mobile robot to realize the required functions is that the mobile robot accurately knows its own position, that is, accurately locates its own position in the environment, so as to complete the instructions issued by the user.

However, the environment will change in reality, for example, the environment includes more dynamic obstacles. When the environment changes, the positioning accuracy of the electronic device will be affected, so how to improve the positioning accuracy of the electronic device is an urgent problem to be solved.

Contents of the invention

The present application provides a positioning method, device, electronic equipment and storage medium, which effectively improves the positioning accuracy of the electronic equipment.

An embodiment of the present application provides a positioning method, which is applied to an electronic device, where the electronic device includes at least one top-view sensor, and the method includes:

acquiring sensor data collected by the at least one top-looking sensor;

processing said sensor data;

The electronic device is located based on the processed sensor data.

Optionally, the electronic device includes at least two top-view sensors;

The positioning of the electronic device according to the processed sensor data includes: matching the processed sensor data with the local map data in the global map to determine where the electronic device is located. Describe the pose in the global map.

Optionally, the electronic device includes a top-view sensor and at least one head-up sensor, and the sensor data includes the head-up environment data collected by the at least one head-up sensor and the top-view environment data collected by the one top-view sensor;

Positioning the electronic device according to the processed sensor data includes: generating a head-up grid map and a top-view grid map according to the processed sensor data; The head-up grid map locates the electronic device.

Optionally, the electronic device includes at least two top-view sensors and at least one head-up sensor, and the sensor data includes head-up environment data collected by at least one head-up sensor and top-view environment data collected by at least two top-view sensors;

The embodiment of the present application also provides a positioning method, which includes:

Collect the current pose of the robot and depth data in at least one preset direction;

determining a feature point cloud based on the depth data;

determining the obstacle score of the feature point cloud in a preset global grid map, wherein the preset global grid map is formed based on historical point clouds;

Optimizing the current pose according to the obstacle score to determine positioning information of the robot.

Optionally, collecting the current pose of the robot and depth data in at least one preset direction includes: collecting the current pose of the robot, depth data in at least one preset direction, and infrared data in at least one preset direction;

The determining the feature point cloud based on the depth data includes: extracting at least one edge point from the infrared data, and converting the at least one edge point into a feature point cloud according to the depth data.

Optionally, determining the feature point cloud based on the depth data includes: extracting the outer contour points of at least one plane from the depth data and forming the feature point cloud from the outer contour points of the at least one plane.

The embodiment of the present application also provides a positioning device, which is configured in an electronic device, and the device includes:

An acquisition module configured to acquire sensor data collected by at least one top-view sensor;

a processing module configured to process the sensor data;

A positioning module configured to locate the electronic device according to the processed sensor data.

The data collection module is configured to collect the current pose of the robot and depth data in at least one preset direction;

A feature point cloud determination module, configured to determine a feature point cloud based on the depth data;

The obstacle score determination module is configured to determine the obstacle score of the feature point cloud in the preset global grid map, wherein the preset global grid map is formed based on historical point clouds;

A positioning determining module configured to optimize the current pose according to the obstacle score to determine positioning information of the robot.

The embodiment of the present application also provides an electronic device, which includes: at least one processor; a storage device configured to store one or more programs; when the at least one program is executed by the one or more processors Executing, so that the at least one processor implements any positioning method provided in this application.

Optionally, the electronic device further includes: at least two top-view sensors: there is a common-view area in the field of view between the at least two top-view sensors, and the at least two top-view sensors are configured to collect the top-view environment data.

Optionally, the electronic device further includes: a top-view sensor and at least one head-up sensor; the at least one head-up sensor is configured to collect head-up environment data; the one top-view sensor is configured to collect top-view environment data.

Optionally, the electronic device further includes: further comprising: at least two head-up sensors and at least one head-up sensor; the at least one head-up sensor is configured to collect head-up environment data;

The at least two top-view sensors are configured to collect top-view environmental data.

Optionally, the included angles between the at least two top-view sensors and the horizontal direction are different, and there is a common-view area with a set ratio in the field of view between two adjacent top-view sensors.

Beneficial technical effect of the present invention, painstakingly added

Description of drawings

FIG. 1 is a schematic flowchart of a positioning method provided in an embodiment of the present application;

Fig. 2 is a schematic diagram of attitude definition in robotics in the related art;

FIG. 3 is a schematic flowchart of another positioning method provided in the embodiment of the present application;

FIG. 4 is a schematic flowchart of another positioning method provided in the embodiment of the present application;

FIG. 5 is a schematic flowchart of another positioning method provided in the embodiment of the present application;

Fig. 6 is a schematic flow chart of a positioning method for a dual top-looking time-of-flight (Time of Fight, TOF) camera provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of the installation of a dual top-view TOF camera provided in the embodiment of the present application;

FIG. 8 is a schematic flowchart of another positioning method provided in the embodiment of the present application;

FIG. 9 is a schematic structural diagram of an indoor robot provided by an embodiment of the present application;

FIG. 10 is a schematic flowchart of another positioning method provided in the embodiment of the present application;

FIG. 11 is a schematic flowchart of another positioning method provided in the embodiment of the present application;

FIG. 12 is a schematic flowchart of a multi-layer grid map positioning method provided by an embodiment of the present application;

FIG. 13 is a schematic flowchart of another positioning method provided in the embodiment of the present application;

Fig. 14 is a schematic structural diagram of another indoor robot provided by the embodiment of the present application;

FIG. 15 is a schematic flowchart of another positioning method provided by the embodiment of the present application;

FIG. 16 is a schematic flowchart of another positioning method provided in the embodiment of the present application;

FIG. 17 is a schematic flowchart of another multi-layer grid map positioning method provided by the embodiment of the present application;

Fig. 18 is a schematic flowchart of another positioning method provided by the embodiment of the present application;

Fig. 19 is a schematic flowchart of another positioning method provided by the embodiment of the present application;

Fig. 20 is an example diagram of a pose provided by an embodiment of the present application;

FIG. 21 is an example diagram of a preset global grid map provided by an embodiment of the present application;

Fig. 22 is a schematic flowchart of another positioning method provided by the embodiment of the present application;

Fig. 23 is an example diagram of a coordinate transformation provided by the embodiment of the present application;

Fig. 24 is a schematic flowchart of another positioning method provided by the embodiment of the present application;

Fig. 25 is an example diagram of a positioning method provided by an embodiment of the present application;

Fig. 26 is a schematic flowchart of another positioning method provided by the embodiment of the present application;

Fig. 27 is a schematic flowchart of another positioning method provided by the embodiment of the present application;

Fig. 28 is a schematic flowchart of another positioning method provided by the embodiment of the present application;

Fig. 29 is an example diagram of another positioning method provided by the embodiment of the present application;

Fig. 30 is a schematic structural diagram of a positioning device provided by an embodiment of the present application;

Fig. 31 is a schematic structural diagram of another positioning device provided by the embodiment of the present application;

Fig. 32 is a schematic structural diagram of another positioning device provided by the embodiment of the present application;

Fig. 33 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

FIG. 34 is a schematic structural diagram of a storage medium provided by an embodiment of the present application.

Detailed ways

The application will be described below in conjunction with the accompanying drawings and embodiments. The specific embodiments described here are only used to explain the present application, but not to limit the present application. In addition, for the convenience of description, only some structures related to the present application are shown in the drawings but not all structures.

Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart depicts operations (or steps) as sequential processing, many of the operations may be performed in parallel, concurrently, or simultaneously. Additionally, the order of multiple operations can be rearranged. The process may be terminated when multiple operations are complete, but may also have additional steps not included in the figure. The processing may correspond to a method, function, procedure, subroutine, subroutine, or the like.

The term "comprising" and its variants used in this application are open-ended, ie "including but not limited to". The term "based on" is "based at least in part on". The term "one embodiment" means "at least one embodiment."

The modifications of "one" and "multiple" mentioned in this application are illustrative and not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, "one" and "multiple" should be understood as "one or more".

FIG. 1 is a schematic flowchart of a positioning method provided in an embodiment of the present application. The method is applicable to the situation of locating an electronic device, and the method can be executed by a locating device, wherein the device can be implemented by software and/or hardware, and is generally integrated on the electronic device. The electronic device includes at least one top-view sensor. A top-view sensor can be thought of as a sensor that sits on top of an electronic device. The data collected by the top-view sensor during the mapping phase is used to build a global map. The data collected by the top-view sensor in the positioning phase is used in combination with the global map for positioning. The electronic device in this application can perform indoor positioning, and can perform outdoor positioning when the outdoor top-view environment is not uniform. The electronic device in this application may be a mobile electronic device. Exemplarily, the electronic device is a robot.

The technical terms involved in this application are described below: a reference coordinate system is also called a reference coordinate system, which is a reference standard for describing the positions or angles of points, lines, planes, and coordinate systems. The camera extrinsics are used to represent the pose of the camera in three-dimensional space relative to the reference coordinate system. Pose refers to position and attitude. FIG. 2 is a schematic diagram of attitude definition in robotics in the related art. Referring to FIG. 2 , the position includes x, y, z, and the attitude includes yaw, pitch, and roll.

Coordinate transformation in this application: is the process of transforming from one coordinate system to another. Statistical filter: calculate the average distance from each point to the nearest k points of each point, and the distances of all points in the point cloud should form a Gaussian distribution. Calculate the mean and variance, and remove noise points through the 3σ principle, and k is a positive integer. Nonlinear optimization method: In a given objective function f(x), find the optimal set of numerical mappings to make f(x) maximum or minimum. When f(x) is a nonlinear function, the solution method is called In the nonlinear optimization method, the dimension of x in this application is 3, f(x1, y1, θ), that is, the objective function is a function about x1, y1 and θ. Among them, x1 and y1 can represent the relative distance between the top-looking sensors, such as the distance between the top-looking sensors, and the distance between the top-looking sensor and the roof of the house. x1 and y1 may be values in the coordinate system where the top-view sensor is located. θ can be considered as the angle between the top-view sensor and the vertical direction. The TOF camera continuously sends light pulses to the target, then uses the sensor to receive the light returned from the target, and obtains the distance of the target by detecting the time-of-flight of the light pulse. In this application, a target may be considered as an object on top of an electronic device, such as a roof in a house.

The prerequisite for realizing navigation and positioning is to establish an accurate map in the environment, which is the basic condition for realizing positioning and navigation. Common methods for mapping and positioning electronic devices include cameras and lidar. The mapping and positioning method based on lidar has high precision and strong anti-interference ability of data precision. It is a relatively mature and widely used technology, but the cost of a single laser is high, and it faces the problem of high cost in the process of widespread promotion. The use of cameras for positioning has the advantages of low hardware cost and rich information. Cameras mainly include monocular cameras, binocular cameras, depth cameras, infrared cameras and other types. Using camera mapping and positioning generally requires the fusion of multiple sensor data such as wheel odometers or inertial measurement units (Inertial Measurement Unit, IMU). The binocular camera needs to restore the depth information in the scene through the parallax calculation of the two cameras, the depth camera directly obtains the depth in the scene through TOF, structured light and other technologies, and the infrared camera obtains the infrared image in the scene. The depth camera is used to restore the point in the two-dimensional space to the point cloud in the three-dimensional space, that is, the pixel point of the visual data can be converted into the virtual radar technology of the point cloud data, which is also applicable to the laser positioning algorithm. Laser-like point cloud information, visual image information, and infrared image information can be simultaneously obtained through the above-mentioned camera, which combines the advantages of both laser and camera sensors.

Positioning first requires the construction of an accurate indoor map, which is used for the calculation of the electronic device's own pose in the absolute coordinate system and the path planning for the subsequent movement of the electronic device. The direction of traditional sensors is set to the horizontal direction to collect environmental data in the horizontal direction. Since the environment changes rapidly in reality, and it takes a long time to establish a map, the accuracy of the established map is poor, and the subsequent robot positioning process is easy. Loss of positioning data, that is, when drastic changes occur in the environment, the positioning accuracy will drop sharply. In addition, in the real environment, the data collected by the forward-looking sensor (head-up sensor) is easily affected by dynamic objects such as the flow of people, such as: flow of people, pets, mobile robots blocking the sensor, which will also affect the accuracy of positioning, thus restricting the electronic equipment in the aircraft. The use effect in the field, station, shopping mall or supermarket environment.

In order to improve the positioning accuracy, as shown in Figure 1, a positioning method provided in the embodiment of the present application includes the following steps:

S10. Acquire sensor data collected by at least one top-view sensor.

S20. Process sensor data.

Exemplarily, processing the sensor data may include removing noise data in the sensor data.

S30. Position the electronic device according to the processed sensor data.

Exemplarily, the processed sensor data may be matched with local map data in the global map, and the pose of the electronic device in the global map may be determined according to the local map data in the global map matched with the processed sensor data.

Exemplarily, determining the pose of the electronic device in the global map according to the local map data matched with the processed sensor data in the global map includes: according to the local map data matched with the processed sensor data in the global map and the global The pose transformation relationship of the map, the pose transformation relationship between the processed sensor data and the local map data, determines the pose of the electronic device in the global map.

This embodiment locates the electronic device based on the data collected by the top-view sensor. Since the environment above the computer device is not easy to change, the positioning method provided by this embodiment effectively improves the positioning accuracy of the electronic device.

Fig. 3 is a flow chart of another positioning method provided by the embodiment of the present application. In this embodiment, the computer device includes at least two top-view sensors. As shown in Figure 3, the method provided in this embodiment includes the following steps:

S110. Acquire sensor data collected by at least two top-view sensors.

In this embodiment, sensor data can be considered as data collected by sensors. A sensor may be a detection device on an electronic device. The content of the sensor data is not limited here, and may be determined based on the type of sensor included in the electronic device. For example, the sensor data includes the top-view environment data collected by the top-view sensor. The top-view environment data may be data collected by a top-view sensor, for example, the environment data on top of an electronic device is collected as top-view environment data. The content of the environmental data is determined based on the collected equipment and is not limited here.

Before realizing the positioning, in this step, the sensor data collected by the sensor on the electronic device may be obtained first, so as to process the sensor data and locate the electronic device. The method of obtaining the sensor data is not limited here.

S120. Process the sensor data.

After the sensor data is acquired, the sensor data can be processed so that localization can be performed based on the processed sensor data. How to process the sensor data is not limited here, and different sensor data corresponds to different processing means.

In one embodiment, the time stamps of the sensor data collected by at least two top-looking sensors can be aligned so as to locate the electronic device. When the sensor data includes top-view environment data, since the top-view environment data is collected by at least two top-view sensors, this embodiment can position the electronic device based on the top-view environment data corresponding to each top-view sensor, or The top-view environment data of at least two top-view sensors are spliced to locate the electronic device based on the spliced top-view environment data.

In one embodiment, when positioning the electronic device based on the top-view environment data, this embodiment can extract the point cloud information of the surface edge based on the top-view environment data, so as to perform global map matching based on the point cloud information, so as to realize electronic The location of the device. The global map may be a grid map established based on the positioning scene. The positioning scene can be regarded as a scene where the electronic device is currently located and needs to be positioned. A global map can be constructed during the mapping phase.

S130. When the positioning instruction is acquired, match the processed sensor data with the local map data in the global map, and determine the pose of the electronic device in the global map.

The positioning instruction can be regarded as an instruction to trigger the electronic device to perform positioning. The positioning instruction can be triggered through a human-computer interaction interface, which is not limited here. The local map data can be regarded as the data of the local map in the global map. A global map can be composed of local maps.

In this step, the pose transformation relationship between the processed sensor data and the local map data may be determined, and the electronic device positioning may be performed in combination with the pose transformation relationship between the local map and the global map. After the processed sensor data is matched with the local map data, the pose transformation relationship between the two can be obtained, and then the electronic device can be positioned by combining the pose transformation relationship between the local map and the global map. The pose transformation relation is the pose relation.

In one embodiment, the global map can be constructed based on the top-view environment data collected by multiple top-view sensors, so in the positioning phase, based on the processed sensor data, the corresponding local map data can be matched in the global map, and then Ability to determine the pose of the electronic device in the global map.

In a positioning method provided by an embodiment of the present application, sensor data is firstly obtained; secondly, the sensor data is processed; and then, when a positioning instruction is obtained, the processed sensor data is matched with the local map data in the global map, Determine the pose of the electronic device in the global map. Utilizing the above technical solution, it is possible to locate the electronic device based on the sensor data collected by the top-view sensor and the local map data in the global map, and effectively avoid the technology of poor positioning accuracy when the environment changes during the positioning of the electronic device. problem, effectively improving the positioning accuracy.

On the basis of the above-mentioned embodiments, modified embodiments of the above-mentioned embodiments are proposed. It should be noted here that, for the sake of brevity, only differences from the above-mentioned embodiments are described in the modified embodiments.

In one embodiment, the sensor data includes top-view environment data; processing the sensor data includes: aligning the top-view environment data collected by the at least two top-view sensors based on time stamps; preprocessing the aligned top-view environment data environmental data.

This embodiment refines the operation of processing sensor data to ensure that the processed sensor can achieve accurate positioning.

In one embodiment, matching the processed sensor data with local map data in the global map to determine the pose of the electronic device in the global map includes: The point cloud information of the edge is converted into raster data; determine the local map data matching the raster data in the global map; according to the local map data and the raster data, determine the pose in .

In this embodiment, the processed sensor data is matched with the local map data in the global map to determine the pose of the electronic device, and accurately locate the electronic device.

Fig. 4 is a schematic flow chart of another positioning method provided by the embodiment of the present application. Referring to Fig. 4, this method comprises the steps:

S210. Acquire sensor data of at least two top-view sensors.

Sensor data includes top-looking environment data.

S220. Align the top-view environment data of the at least two top-view sensors based on the time stamps.

Aligning based on time stamps can be considered as establishing a corresponding relationship between sensor data collected by multiple sensors based on time. The technical means of alignment are not limited here.

S230. Preprocess the aligned top-view environment data.

Preprocessing the aligned top-view environment data can be considered as processing the aligned top-view environment data to facilitate positioning. The technical means of preprocessing is not limited, as long as it is convenient to match the preprocessed visual environment data with the local map data in the global map.

Exemplarily, the means of preprocessing include but not limited to denoising and splicing.

S240. Convert the point cloud information of the surface edge included in the processed sensor data into grid data when the positioning instruction is acquired.

There is no limitation on how to convert the point cloud information of the surface edge into raster data. Exemplarily, when converting the point cloud information of the edge of the surface into raster data, the mapping conversion can be performed based on the Cartesian coordinate system, such as converting the point cloud information of the edge of the surface into the Cartesian coordinate system, and then converting to The point cloud information of the surface edge in the Cartesian coordinate system is projected into the grid coordinate system to obtain the grid data.

S250. Determine local map data matching the raster data in the global map.

The global map can be regarded as a grid map. To facilitate matching, this embodiment performs matching with the global map based on the grid data, so as to obtain local map data corresponding to the grid data.

Since the global map is generated based on the top-view environment data collected by the top-view sensor, when positioning electronic equipment, the corresponding local map data can be matched from the global map based on the raster data, and the local map data can be determined based on the matched local map data. The pose of the electronic device. The matching technical means are not limited here.

S260. Determine the pose of the electronic device in the global map according to the local map data and the grid data.

After the local map data is obtained by matching the grid data with the global map, this embodiment can determine the pose of the electronic device based on the pose transformation relationship between the local map data obtained through matching and the grid data.

Exemplarily, this embodiment determines the pose of the electronic device in the global map through the pose transformation relationship between the local map and the global map corresponding to the local map data, and the pose transformation relationship between the grid data and the local map data.

This embodiment refines the operations of processing sensor data and determining the pose of an electronic device. Using the positioning method of this embodiment, the sensor data is effectively processed, and the electronic device is positioned based on the processed sensor data and the global map. Improved positioning accuracy.

In one embodiment, the preprocessing of the aligned top-view environment data includes: removing noise points in the aligned top-view environment data; splicing the top-view environment data after denoising; extracting the spliced top-view environment data Point cloud information of the edge of the midplane.

This embodiment refines the operation of preprocessing the aligned top-view environment data. By splicing and denoising the top-view environment data, the field of view angle of the top-view sensor is increased, and the positioning is improved by extracting point cloud information at the edge of the surface. efficiency.

When preprocessing the top-view environment data, the noise in the aligned top-view environment data can be removed first to improve the positioning accuracy. The method for removing noise is not limited here.

After the noise is removed, in this embodiment, the noise-removed top-view environment data can be spliced, so as to expand the field of view of the top-view sensor and improve positioning accuracy. The method of splicing the noise-removed top-view environment data is not limited, for example, splicing the noise-removed top-view environment data by means of coordinate transformation.

After splicing the noise-removed top-view environment data, point cloud information of surface edges in the spliced top-view environment data may be extracted, so as to locate the electronic device based on the extracted point cloud information. The technical means for extracting point cloud information is not limited here.

In one embodiment, the splicing the noise-removed top-view environment data includes: converting the noise-removed top-view environment data into the coordinate system where the target top-view sensor is located, and the target top-view sensor is the at least One of the two top-looking sensors.

The target top-looking sensor is any one of the at least two top-looking sensors. During the conversion of the coordinate system, the conversion can be performed based on the external parameters of the top-view sensor, and the conversion method is not limited here.

Taking the number of top-looking sensors as an example, in this example, the top-looking environment data collected by the two top-looking sensors can be transformed into the Cartesian coordinate system, and then based on the external parameters of the two top-looking sensors, the two The noise-removed top-view environment data corresponding to one top-view sensor among the top-view sensors is converted to the coordinate system where the other top-view sensor is located.

This embodiment refines the operation of splicing and denoising the top-view environment data, and effectively increases the field of view angle of the top-view sensor through the technical means of coordinate system conversion.

In one embodiment, the determining the pose of the electronic device in the global map according to the local map data and the grid data includes: according to the relationship between the local map data and the global map and the pose relationship between the raster data and the local map data to determine the pose of the electronic device in the global map.

In this embodiment, based on the pose relationship between the grid data and the local map data and the pose relationship between the local map data and the global map, the pose relationship between the grid data and the global map can be determined, that is, the global pose in the map.

This embodiment refines the technical means for locating electronic devices through local map data and global maps, effectively locating electronic devices.

In one embodiment, the method further includes: if the mapping instruction is obtained, adding the point cloud information of the edge of the surface included in the processed sensor data to the matched local map data; adding the point cloud information of the edge of the surface The local map data after the cloud information is updated to the global map.

The mapping instruction can be regarded as an instruction that triggers the electronic device to perform mapping. The acquisition of the mapping instruction is not limited, for example, it can be acquired through a human-computer interaction interface.

After obtaining the mapping instruction, this embodiment can add point cloud information of the edge of the surface to the matched local map data, and then use the local map data after adding the point cloud information of the edge of the surface to update the global map.

In this embodiment, after obtaining the mapping instruction, the mapping is effectively performed based on the processed sensor data, so as to facilitate the positioning of the electronic device.

FIG. 5 is a schematic flow chart of another positioning method provided by the embodiment of the present application. Referring to Fig. 5, this method comprises the steps:

S310. Acquire sensor data collected by at least two top-view sensors.

S320. Process the sensor data.

S330. When the positioning instruction is acquired, match the processed sensor data with the local map data in the global map, and determine the pose of the electronic device in the global map.

S340. If the mapping instruction is obtained, add the point cloud information of the surface edge included in the processed sensor data to the matched local map data.

S350. Update the local map data added with the point cloud information of the surface edge into the global map.

The execution sequence of S340 and S330 is not limited here, and may be executed in parallel or sequentially.

In one embodiment, the electronic device can acquire sensor data in real time, and after monitoring the positioning instruction, it can perform positioning based on the sensor data; after monitoring the mapping instruction, it can build a map based on the sensor data.

In an embodiment, the electronic device may also acquire sensor data after receiving a mapping instruction or a positioning instruction.

The following is an exemplary description of the present application. The positioning method provided by the embodiment of the present application can be regarded as a solution for robot positioning and mapping based on dual top-view TOF cameras. The positioning method provided by this application can make the robot maintain stable positioning in the scene where obstacles on the ground change frequently or there are many dynamic obstacles. Install a wheel encoder on the driving wheel of the robot, and install a top-view sensor on the top of the robot, such as TOF camera 1 and TOF camera 2. The angle range between TOF camera 1 and TOF camera 2 and the vertical direction is 0°-30° , the field of view of a single TOF camera is small, and the field of view is easily blocked, resulting in fluctuations in positioning. The purpose of installing two TOF cameras is to expand the field of view. The intersection of the two fields of view is to calibrate the external parameters between the two TOF cameras. . The distance parameter between TOF camera 1 and TOF camera 2 is based on the robot's own parameters (such as the size of the robot, the area of the top surface of the robot, etc.) combined with the camera field of view (Field of View, FOV) Adjust to the value under the installation method with the widest field of view. For example, the position difference in the y1 direction is 20cm, the difference in the x1 direction is 0, and the two cameras deviate from the vertical direction by 25 degrees to the first side and the second side, or the position difference in the x1 direction is 20cm, and the difference in the y1 direction is 0. 20 degrees from vertical. Due to the installation method of the double top-view TOF camera, the features on the ceiling are less likely to change than those on the ground, and are not easily affected by dynamic obstacles such as crowds on the ground, and the double top-view TOF camera overcomes the single TOF The effect of poor positioning stability when the camera field of view is too small. Using dual TOF camera data and wheel encoder data, the pose transformation between the robot TOF camera and the local map and the pose transformation between the local map and the global map are determined through feature extraction, brute force matching and nonlinear optimization methods. Pose transformation can be understood as a relative position relationship. After obtaining the relative positional relationship between two objects, the positional relationship between different objects can be deduced. For example, the robot can calculate the relative pose of the robot at two different times through the same object seen by the sensor at two different times, and can also obtain the position of the object relative to the robot. The above steps can be considered as map building. By matching the sensor data with the objects in the map, the pose of the robot is obtained, that is, localization.

x1, y1 and the angle can be determined according to the field of view angle of the top-view sensor and the distance between the top-view sensor and the roof of the house. The specific values are not limited, as long as multiple top-view sensors can be seen together. Angle is the angle between the top-looking sensor and the vertical direction. For example, the distance between two top-looking sensors

Wherein, h may be the distance between the top-view sensor and the roof of the house. H is the field of view angle, and θ can be considered as the angle between the top-view sensor and the vertical direction.

This application can be applied to the field of robot automatic control technology. The positioning method has strong anti-interference ability, and is suitable for dynamic environments or environments with many moving obstacles, and overcomes the small field of view of the camera when the robot is in a changing environment or when there are many people. The problem of robot positioning failure when it is susceptible to interference.

Taking the electronic device as a robot and the top-view sensor as a double-top-view TOF camera as an example, FIG. 6 is a schematic flowchart of a positioning method for a double-top-view TOF camera provided by an embodiment of the present application, and FIG. 7 is a schematic diagram of a positioning method provided by an embodiment of the present application. A schematic diagram of the installation of a double top-view TOF camera, see Figure 6. When the double top-view TOF camera is used for robot positioning, it includes the following steps:

Step s1: collect data.

As shown in Figure 6, sensors are installed on the machine, that is, the robot: 1. TOF camera 1 and TOF camera 2, 2. Wheel encoder, wherein TOF camera 1 and TOF camera 2 are double top-view TOF cameras.

The step s1 described in the embodiment of the present application also includes the following steps: when installing the wheel encoder, the wheel encoder can be installed on the wheel shaft of the driving wheel of the robot, and the wheel encoder is used for track deduction to calculate the walking of the robot The speed and distance are used for pose transformation and nonlinear optimization.

When installing the depth TOF camera 1, the depth TOF camera 1 needs to be installed on the top of the robot. The angle between the depth TOF camera 1 and the vertical direction ranges from 0 to 30°, and the specific angle can be adjusted according to the camera model.

When installing the depth TOF camera 2, the depth TOF camera 2 needs to be installed on the top of the robot. The angle between the depth TOF camera 2 and the vertical direction ranges from 0 to 30°, and the specific angle can be adjusted according to the camera model.

After the sensor is installed, the sensor data can be obtained, that is, the data collected by the sensor can be obtained.

Step s13: Align the timestamps of the encoder data and the two TOF camera data.

Time stamp alignment of multiple sensors means aligning sensor data collected by multiple sensors based on time stamps.

Step s2: Preprocessing TOF camera data.

Preprocess the collected point cloud information, that is, preprocess the aligned top-view environment data.

The step s2 provided in the embodiment of the present application also includes the following steps: using a statistical filter to remove noise from the original data of the TOF camera 1 and TOF camera 2, that is, remove the noise in the aligned top-view environment data; Using the external parameters between TOF camera 1 and TOF camera 2, the data of TOF camera 2 is converted to the coordinate system of TOF camera 1, that is, the top-view environment data after splicing and removing noise.

Transforming the coordinate system also includes the following steps: converting the point clouds acquired by TOF camera 1 and TOF camera 2 into the Cartesian coordinate system, and the coordinates of any point in the TOF camera 1 after converting to the Cartesian coordinate system are expressed as

After converting to the Cartesian coordinate system, the coordinates of any point in the TOF camera 2 are expressed as

The external parameters of TOF camera 1 and TOF camera 2 (which can indicate the relative position and direction of TOF camera 1 and TOF camera 2) are expressed as

Converting the point cloud acquired under TOF camera 2 to TOF camera 1 (that is, the target top-view sensor) is expressed as:

After converting all points obtained under TOF camera 2, all points under the field of view of TOF camera 2 are converted to under TOF camera 1, thereby expanding the field of view angle of TOF camera 1.

Step s23: Extract point cloud information of the edge of the surface.

Extract the point cloud information of the surface edge and convert the extracted point cloud information of the surface edge into two-dimensional (2-Dimension, 2D) grid data, that is, extract the point cloud data of the surface edge in the top-view environment data (point cloud information ) and convert the point cloud data of the surface edge in the extracted top-view environment data into raster data.

Extracting the point cloud information of the edge of the surface further includes the following steps: screening the points with edge features in the point cloud of the current frame, where the point cloud of the current frame can be the data of the enlarged TOF camera 1 .

The plane grid coordinate system is established with the origin of the TOF camera, and the point cloud information at the edge of the surface is converted to the Cartesian coordinate system.

Project the point cloud information of the surface edge into the grid coordinate system according to x-y to obtain 2D grid data, that is, grid data.

Step s3: Match the point cloud information of the surface edge with the local map data.

Calculate the pose transformation between the collected point cloud information of the surface edge and the local map data, that is, determine the local map data matching the raster data in the global map.

Step s4: Add the point cloud information of the surface edge to the local map data that is successfully matched.

Using the pose obtained by matching the point cloud information of the edge of the face with the local map data, the current frame point cloud is added to the local map, that is, the point cloud information is added to the matched local map data.

Step s5: Match the local map data with the global map to obtain the pose.

Using the coordinates of all points in the local map, calculate the pose transformation between the local map and the global map to achieve positioning, that is, determine the pose of the electronic device in the global map based on the local map data and raster data.

Step s6: Determine whether the current mode is the positioning mode, if the current mode is the positioning mode, execute s1, and determine the position of the electronic device on the global map based on the pose between the local map data and the raster data and the pose obtained in step s5 pose in ; if the current mode is not positioning mode, go to s7.

It is judged that the current mode is the positioning mode/mapping mode, and if the current mode is the mapping mode, step s7 is executed. Select the mapping mode or positioning mode on the application (Application, APP). If the mapping mode is selected, a map will be generated. If the positioning mode is selected, the robot will be positioned on the existing map.

Step s7: Add the local map data to the global map.

When the current mode is not the positioning mode, use the result obtained by s5 to update the points in the local map to the global map, that is, update the global map.

Step s7 includes the following steps: Transform the point cloud in the local map into the global map coordinate system through the pose obtained in step s5, and the position of the local map relative to the robot is known, and the position of the robot in the global map is known, then The position of the local map on the global map can be calculated through pose transformation. Similar to A is 1m north of B, and B is 1m north of C, then the distance between A and C is 2m.

Use the x-y value of the coordinates of the point cloud in the local map in the global coordinate system to determine the grid coordinates in the global map, and then use the z value to update the height distribution in the current grid.

In this application, the global map may be a grid map. If the position of the local map relative to the robot is known, and the position of the robot on the global map is known, the position of the local map on the global map can be calculated through pose conversion. Similar to A is 1m north of B, and B is 1m north of C, then the distance between A and C is 2m.

The positioning method proposed in the embodiment of the present application can be considered as a method of extracting the point cloud features of the ceiling edge for mapping and positioning. This method realizes the function of the robot using the three-dimensional (3-Dimension, 3D) point cloud information of the ceiling for mapping and positioning. . It overcomes the positioning problem of the robot in a scene with a lot of people, that is, solves the positioning problem in the case of a large-scale change in the scene.

Fig. 8 is a schematic flow chart of another positioning method provided by the embodiment of the present application. This method is applicable to indoor positioning of electronic equipment. This method can be performed by a positioning device, wherein the device can be implemented by software and/or hardware. Realized and generally integrated on electronic equipment. An electronic device may be a device capable of active or passive movement. Exemplarily, the electronic device may be a robot. The application scenario of the robot is not limited, and can be indoor or outdoor. In this embodiment, an indoor robot is taken as an example to describe the electronic equipment, and the electronic equipment is not limited here, and the implementation manner of the electronic equipment other than the indoor robot is the same as or similar to that of the indoor robot.

The robots in this embodiment include indoor robots and outdoor robots. Outdoor robots can be considered as robots that work outdoors and can move. Indoor robots can be considered as robots that work indoors and can move. Since the working scenes of indoor robots are generally in high-dynamic environments such as shopping malls, garages, and supermarkets, the head-up lidar (such as two-dimensional lidar) of indoor robots is often blocked, resulting in lidar data containing many dynamic objects or The lidar data are all blocked to become invalid. At the same time, the scene of the indoor robot changes frequently, and the traditional head-up laser mapping and positioning scheme is not robust.

The electronic device in the embodiment of the present application includes a top-view sensor, and the top-view sensor is used to collect top-view environment data. The structure of the electronic equipment will be described by taking the electronic equipment as an indoor robot as an example. FIG. 9 is a schematic structural diagram of an indoor robot provided by an embodiment of the present application. Referring to FIG. 9 , a top-view sensor, such as a depth camera, is located on the top of the indoor robot and is used to collect top-view environmental data on the top of the indoor robot. The sensors included in the indoor robot include at least: a depth camera, a laser radar, and a wheel odometer. The depth camera faces upwards and is used to collect top-view environmental data. The angle between the depth camera and the horizontal direction is not limited here, as long as it can It is enough to collect the top-view environment data on the top of the indoor robot. For example, the angle between the depth camera and the horizontal direction is within the range of 90°±10°. Optionally, the depth camera faces straight up. LiDAR, that is, the installation of the head-up sensor is not limited, as long as the head-up environment data can be collected. Exemplarily, the orientation of the lidar is horizontally forward, and the angle between the positive direction and the horizontal direction is 0°.

When working with an indoor robot, first complete the external parameter calibration of the wheel odometer, lidar, and depth camera; then establish a horizontal environment map and a top-view environment map, that is, a head-up grid map and a top-view grid map; , the indoor robot can be positioned through the head-up grid map and the top-view grid map. As shown in Figure 8, a positioning method provided in the embodiment of the present application includes the following steps:

S410. Acquire sensor data, where the sensor data includes head-up environment data collected by at least one head-up sensor and top-view environment data collected by a top-view sensor.

In this embodiment, the sensor data may be regarded as data collected by a sensor located on the electronic device. The content of the sensor data is not limited here, and may be determined according to the type of sensor included in the electronic device. For example, the sensor data may include: environment data collected by the top-view sensor (ie, top-view environment data), environment data collected by the head-up sensor (ie, head-up environment data), and data collected by the wheel odometer. The environment data includes head-up environment data and top-view environment data, and the environment data can be regarded as the data collected by the sensor representing the surrounding environment of the electronic device.

The head-up environmental data can be considered as the environmental data of the running direction of the electronic equipment collected by the head-up sensor, and the head-up sensor can be considered as the sensor that collects the environmental data of the running direction of the electronic equipment, such as lidar. The top-view environment data can be considered as the environment data on the top of the electronic device collected by the top-view sensor, and the top-view sensor can be considered as a sensor that collects the environment data on the top of the electronic device, such as a depth camera.

The top-view environment data, such as ceiling information is relatively fixed, and the probability of being blocked is relatively small, so the top-view environment data can maintain stable positioning when the head-up environment data is blocked.

Before realizing indoor positioning, the sensor data collected by the sensors on the indoor robot can be obtained first, so as to process the sensor data for indoor positioning, and the method of obtaining the sensor data is not limited here. For example, the processor of the indoor robot communicates with the sensor to obtain the sensor data collected by the sensor.

S420. Process the sensor data to obtain processed sensor data.

Once the sensor data has been acquired, it can be processed so that a raster map can be built. A grid map is a map expression method. The grid map divides the spatial plane into grids with a certain resolution, and the value in the grid is the probability that the current grid is occupied. In this embodiment, the grid map includes a top-view grid map and a horizontal grid map.

How to process the sensor data is not limited here, and different sensor data corresponds to different processing methods.

In one embodiment, since the top-view sensor data is collected by the top-view sensor, in order to improve the positioning efficiency, when processing the top-view environment data, the point cloud information of the surface edge in the top-view environment data can be extracted; Extract the point cloud information of the face edge in the head-up environment data. For accurate positioning, when processing sensor data, time stamps can be aligned between top-looking sensor data and head-down sensor data. When the head-up environment data is collected by multiple head-up sensors, the head-up environment data collected by the multiple head-up sensors can be spliced when processing the sensor data.

S430. Generate a head-up grid map and a top-view grid map according to the processed sensor data.

In this embodiment, the head-up grid map can be regarded as a map established based on the environment of the head-up direction of the electronic device (the running direction of the electronic device), and the top-view grid map can be regarded as a map established based on the environment of the electronic device's top-view direction. Exemplarily, the head-up grid map may be a grid map generated based on processed head-up environment data. The top-view grid map may be a grid map generated based on the processed top-view environment data. The data used in the establishment of the head-up grid map and the top-view grid map are not limited here.

When generating a grid map, an empty grid map is first generated, and after the point cloud information (that is, the processed top-view environment data and the processed head-up environment data in the processed sensor data) is obtained, the point cloud information can be Map to raster map to update the raster map and calculate the probability corresponding to the raster.

When the environment in the top-view direction of the electronic device is single, this embodiment can combine the head-up grid map to eliminate duplicate top-view environment data, so as to avoid the problem of low positioning efficiency caused by the single environment in the top-view direction of the electronic device. In the environment of the head-up direction of the electronic device, the mobility of people is high, or the environment of the head-up direction of the electronic device changes dynamically, this embodiment can combine the top-view grid map to improve the positioning accuracy.

S440. Position the electronic device according to the processed sensor data, the top-view grid map, and the head-up grid map.

After the top-view grid map and the head-down grid map are generated, the electronic device can be positioned based on the processed sensor data, the top-view grid map and the head-up grid map.

In one embodiment, this embodiment can perform pose prediction based on the wheel odometer data in the processed sensor data, and then realize global pose determination based on the top-view grid map and the flat-view grid map.

When locating electronic devices based on the top-view grid map and the top-view grid map, loop closure detection can be performed on the top-view grid map and the top-view grid map, and then the global pose can be optimized and output based on the global pose graph.

Exemplarily, the closed-loop detection may be calculating the matching rate between the point cloud information corresponding to the sensor data and the grid map, and the specific formula is as follows:

In the formula: k: represents the number of preprocessed laser points in the current frame; T: represents the pose of the current frame in the map; h _i : represents the i-th laser point in the current frame after preprocessing The height value of the laser point transformed by the pose T; μ _i , σ _i : Indicates that the i-th laser point among the preprocessed laser points in the current frame is projected to the grid where the grid map falls according to the pose T The parameters of the Gaussian distribution of the height values of all laser points.

Among them, Score represents the matching situation between the current point cloud information and the raster map, that is, the matching rate. The Score is in the range of 0-1. The larger the score, the better the matching, and the more likely it is a closed loop. After the Score is determined, the relative positions of the two nodes can be obtained, and then the global pose graph can be optimized based on the relative positions of the two nodes.

A node is an abstract concept, which represents the information encapsulated by a measurement, such as point cloud information and pose information; the relative position of the pose between two nodes obtained based on the closed-loop information can be used for global pose graph optimization, global pose graph optimization After that, the pose information of the nodes will be adjusted, and finally the optimized global pose result will be output.

When optimizing the global pose graph, the cost function is as follows:

The function h is as follows, which is used to calculate the relative position of the pose between two nodes:

In the formula: c _i , c _j : represent the pose information of nodes i and j respectively; R _i : represent the rotation vector of node i; t _i , t _j : represent the pose translation vectors of nodes i and j respectively; θ _i , θ _j : represent the pose angle vectors of nodes i and j respectively; Z _ij : represent the laser matching between nodes i and j, that is, the observed pose transformation between two laser frames; X ² : pose Graph residual; Λij is the part corresponding to ij in the information matrix, which represents the amount of observation information between i and j, and is used as the weight when optimizing the global pose graph.

In a positioning method provided by an embodiment of the present application, first, sensor data is obtained, and the sensor data includes head-up environment data and top-view environment data; then the sensor data is processed; secondly, a head-up grid map and a head-up grid map are generated according to the processed sensor data A top-view grid map; finally, the electronic device is positioned according to the processed sensor data, the top-view grid map, and the head-up grid map. With the above technical solution, since the environment above the computer equipment is not easy to change, the head-up grid map generated by combining the top-view environment data collected by the top-view sensor and the head-up environment data collected by the head-up sensor avoids The impact of the environment on the positioning of electronic devices improves the robustness of positioning.

In one embodiment, the processing of the sensor data includes: preprocessing the sensor data; transforming the preprocessed sensor data into the body coordinate system; optimizing the sensor data after the coordinate system transformation; according to the optimized The sensor data is processed top-view environment data and processed head-up environment data.

When processing sensor data, the sensor data may be preprocessed first, and the preprocessing means are not limited. The preprocessing means may be determined based on the content of the sensor data. For example, the corresponding relationship between sensor data can be established based on time; point cloud information that is convenient for positioning in the top-view environment data can also be extracted, and point cloud information in the head-up environment data can be extracted.

Exemplarily, the preprocessing means of the head-up environment data in the sensor data includes but not limited to: time stamp alignment, extracting the features in the aligned head-up environment data, and segmenting the point cloud information of the surface edge in the head-up environment data .

In order to realize positioning, the preprocessed sensor data can be transformed into the body coordinate system, so as to locate the electronic device in the body coordinate system. The conversion means is not limited here.

After the coordinate system conversion, the sensor data after the coordinate system conversion can be optimized for better positioning. The optimization methods are not limited, such as Iterative Closest Point (ICP), ICP variants and brute force matching.

After optimizing the sensor data after transforming the coordinate system, the processed top-view environment data and the processed head-up environment data can be obtained according to the optimized sensor data, such as reading the optimized top-view environment data in the optimized sensor data as For the processed top-view environment data, the optimized head-up environment data in the optimized sensor data is read as the processed head-up environment data.

This embodiment refines the operation of processing sensor data to ensure that the processed sensor data can be positioned.

In one embodiment, the optimization of the sensor data after the coordinate system transformation includes: processing the sensor data after the coordinate system transformation by brute force matching.

The sensor data after transforming the coordinate system is processed by brute force matching, which can eliminate the influence of initial value sensitivity. The method of violent matching is not limited here, such as optimizing the sensor data after transforming the coordinate system through the Correlation Scan Match (CSM) algorithm. CSM can calculate the relative pose between the laser and the map. Optimizing the sensor data after transforming the coordinate system through CSM can make the positioning result more accurate.

This embodiment refines the technical means of optimizing the sensor data after the coordinate system is optimized, so that the top-view grid map and the horizontal grid map generated based on the sensor data after the coordinate system optimization conversion can be positioned more accurately.

In one embodiment, the preprocessing of the sensor data includes: aligning the head-up environment data and the top-view environment data based on the time stamp; extracting the point cloud information of the surface edge in the aligned top-view environment data.

This embodiment refines the technical means of preprocessing the sensor. When preprocessing the sensor data, the top-view environment data and the head-down environment data may be aligned based on the time stamp first. .

This embodiment can effectively associate sensor data based on time, and effectively extract point cloud information of surface edges in the top-view environment data, so as to generate a top-view grid map.

In order to facilitate the generation of the top-view grid map in this embodiment, the point cloud information of the surface edge in the aligned top-view environment data can be extracted, so that the extracted point cloud information can be mapped to the top-view grid map to realize mapping and Positioning: In order to facilitate the generation of a head-up grid map, the point cloud information of the edge of the face in the aligned head-up environment data can be extracted, so that the point cloud information can be mapped to the head-up grid map for mapping and positioning.

Fig. 10 is a schematic flowchart of another positioning method provided in the embodiment of the present application. Referring to Fig. 10, the method includes the following steps:

S510. Acquire sensor data, where the sensor data includes head-up environment data collected by at least one head-up sensor and top-view environment data collected by a top-view sensor.

S520. Align the head-up environment data with the top-view environment data based on the time stamp.

Aligning head-up environment data and top-view environment data based on time stamps can be considered as establishing the corresponding relationship of data collected by multiple sensors based on time, so as to process the aligned top-view environment data and realize positioning.

S530. Extract point cloud information of surface edges in the aligned top-view environment data.

In this embodiment, the point cloud information of the surface edge in the aligned top-view environment data can be extracted for positioning. The method for extracting the point cloud information of the surface edge is not limited here.

S540. Transform the preprocessed sensor data into the body coordinate system.

This example only shows the process of processing the top-view environment data, and the technical means for preprocessing the rest of the sensor data are not limited here. The technical means of coordinate system conversion is not limited here.

S550. Optimizing the sensor data after the coordinate system transformation.

S560. Obtain processed top-view environment data and processed head-up environment data according to the optimized sensor data.

S570. Generate a head-up grid map and a top-view grid map according to the processed sensor data.

S580. Position the electronic device according to the processed sensor data, the top-view grid map, and the head-up grid map.

In one embodiment, the positioning of the electronic device according to the processed sensor data, the top-view grid map and the head-up grid map includes: positioning the head-up view according to the processed head-up environment data Perform closed-loop detection on the grid map to obtain a head-up matching rate; perform closed-loop detection on the top-view grid map according to the processed top-view environment data to obtain a top-view matching rate; when the head-up matching rate is greater than the first set threshold and/or Or when the top-view matching rate is greater than a second set threshold, determine the global pose of the electronic device.

This embodiment refines the positioning technical means, and determines the global pose of the electronic device based on the head-up matching rate and the top-looking matching rate, which can ensure more accurate positioning results.

Fig. 11 is a schematic flowchart of another positioning method provided in the embodiment of the present application. Referring to Fig. 11, the method includes the following steps:

S610. Acquire sensor data, where the sensor data includes head-up environment data collected by the head-up sensor and top-view environment data collected by the top-view sensor.

S620. Process the sensor data.

S630. Generate a head-up grid map and a top-view grid map according to the processed sensor data.

S640. Perform closed-loop detection on the head-up grid map according to the processed head-up environment data to obtain a head-up matching rate.

The head-up matching rate can be considered as the probability of matching the head-up grid map with the head-up environment data. Here, there is no limitation on the technical means of loop closure detection, as long as the head-up matching rate can be determined.

In this embodiment, based on the head-up grid map and the head-up environment data, the head-up matching rate can be obtained by performing closed-loop detection on the head-up grid map.

S650. Perform loop closure detection on the top-view grid map according to the processed top-view environment data to obtain a top-view matching rate.

The top-view matching rate can be considered as the probability of matching the top-view grid map with the top-view environmental data. Here, no limitation is imposed on the technical means of loop closure detection, as long as the top-view matching rate can be determined.

In this embodiment, based on the top-view grid map and the top-view environment data, the closed-loop detection of the top-view grid map can be performed to obtain the top-view matching rate.

When determining the head-up matching rate and the top-view matching rate, loop closure detection, also known as loop-back detection, can be performed on the head-up grid map and the top-view grid map to determine the corresponding top-view matching rate and horizontal matching rate. Here, the execution order of determining the head-up matching rate and the top-view matching rate is not limited. The head-up matching rate and the top-view matching rate may be determined in parallel, or the head-up matching rate and the top-view matching rate may be determined sequentially.

S660. Determine the global pose of the electronic device when the head-up matching rate is greater than a first set threshold and/or the top-view matching rate is greater than a second set threshold.

When the head-up matching rate is greater than the first set threshold and/or the top-view matching rate is greater than the second set threshold, this embodiment may determine the global pose of the electronic device based on the top-view grid map and the head-up grid map. For example, the technical means based on pose graph optimization determines the residual error of the pose graph to determine the global pose. The first set threshold and the second set threshold are not limited. The head-up matching rate and the top-view matching rate may have different setting thresholds.

In one embodiment, generating the head-up grid map and the top-view grid map according to the processed sensor data includes: generating a head-up grid map based on the processed head-up environment data; generating a top-view grid map based on the processed top-view environment data grid map.

This embodiment refines the technical means for generating a head-up grid map and a top-view grid map. This embodiment can effectively combine the head-up and top-view environments of electronic devices for precise positioning.

In this embodiment, the head-up grid map is generated based on the head-up environment data, and the top-view grid map is generated based on the top-view environment data.

In the initial stage of mapping, you can first create an empty top-view grid map and an empty head-up grid map. After obtaining the sensor data, you can update the head-up grid map based on the processed head-up environment data. Based on the processed The top-view environment data updates the top-view raster map.

In this embodiment, the fusion of top-view environmental data and horizontal-view environmental data is used to construct maps, that is, map building and positioning are performed based on the fusion of top-view sensors and horizontal-view sensors. Top-view sensor) measurement data characteristics and application scene characteristics, give full play to the advantages of data collected by multiple sensors, and achieve accurate real-time mapping and highly robust positioning.

12 is a schematic flowchart of a multi-layer grid map positioning method provided in the embodiment of the present application. FIG. 12 takes the number of top-view sensors as an example, and the method includes:

S1. Obtain sensor data.

The acquired sensor data includes lidar data, wheel odometer data, and depth camera data, namely top-view environment data and head-up environment data.

S2. Data preprocessing.

S2.1. Time stamp alignment of sensor data collected by multiple sensors.

S2.2. Extract the features of the top-view environment data, and segment the point cloud information of the surface edge.

S3. Prediction of pose by wheel odometer.

S4. Coordinate system conversion.

The point cloud information of the lidar and the point cloud information of the surface edge of the top-view environment data are transferred to the body coordinate system to obtain the point cloud information after coordinate transformation. The body coordinate system can be customized or the coordinate system of the wheel odometer.

S5. Optimizing point cloud information.

Based on the converted point cloud information, the current pose is optimized through the CSM algorithm, where the optimized pose can be considered as the position of the coordinate transformed point cloud information in the grid map in the current body coordinate system, and the current point cloud information In the body coordinate system, through the matching of the current point cloud information and the map, the pose of the point cloud information after coordinate transformation in the body coordinate system can be corrected in the map.

S6. Updating the probability corresponding to the grid in the grid map.

S6.1. Update the head-up grid map based on the head-up laser data;

S6.2. Update the top-view grid map based on the top-view edge point cloud.

S7. Closed-loop detection.

S7.1. Detect head-up closed loop through head-up laser data and head-up grid map;

S7.2. Detect top-view closed loops through the top-view surface edge point cloud and top-view grid map.

S8. Global pose graph optimization, output global pose.

The head-up laser data can be considered as the processed head-up environment data, and the top-view surface edge point cloud can be considered as the processed top-view environment data.

Fig. 13 is a schematic flow chart of another positioning method provided by the embodiment of the present application. This method is applicable to indoor positioning of electronic equipment. This method can be performed by a positioning device, wherein the device can be implemented by software and/or hardware. Realized and generally integrated on electronic equipment. An electronic device may be a device capable of active or passive movement. Exemplarily, the electronic device may be a robot. The application scenario of the robot is not limited, and can be indoor or outdoor. In this embodiment, an indoor robot is taken as an example to describe the electronic equipment, and the electronic equipment is not limited here, and the implementation manner of the electronic equipment other than the indoor robot is the same as or similar to that of the indoor robot.

The robots in this embodiment include indoor robots and outdoor robots. Outdoor robots can be considered as robots that work outdoors and can move. Indoor robots can be considered as robots that work indoors and can move. Since the working scenes of indoor robots are generally in highly dynamic environments such as shopping malls, garages, and supermarkets, the head-up lidar (especially 2D lidar) of indoor robots is often blocked, resulting in many dynamic objects or The lidar data are all blocked to become invalid. At the same time, the scene of the indoor robot changes frequently, and the traditional head-up laser mapping and positioning scheme is not robust.

The electronic device in the embodiment of the present application includes at least two top-view sensors, and the top-view sensors are used to collect top-view environment data. The structure of the electronic equipment will be described by taking the electronic equipment as an indoor robot as an example. Figure 14 is a schematic structural diagram of another indoor robot provided by the embodiment of the present application, see Figure 9 and Figure 14, top-view sensors, such as dual depth cameras, that is, two depth cameras are located on the top of the indoor robot, used to collect the top of the indoor robot Top-view environment data for different regions. The sensors included in the indoor robot include at least: a depth camera, a laser radar, and a wheel odometer. The depth camera faces upwards and is used to collect top-view environmental data. The angle between the depth camera and the horizontal direction is not limited here, as long as the adjacent The field of view between the two depth cameras only needs to have a common view area. For example, the angle between the depth camera and the horizontal direction is 90°±20°. At the same time, ensure that the two cameras have a 1/3 common view area, which is convenient for camera calibration and Image stitching. LiDAR, that is, the installation of the head-up sensor is not limited, as long as the head-up environment data can be collected. Exemplarily, the orientation of the lidar is horizontally forward, and the angle between the positive direction and the horizontal direction is 0°.

When working with an indoor robot, first complete the external parameter calibration of the wheel odometer, lidar, and top-view depth camera; then establish a horizontal environment map and a top-view environment map, that is, a head-up grid map and a top-view grid map; complete After the map is built, the indoor robot can be positioned through the head-up grid map and the top-view grid map. As shown in Figure 13, a positioning method provided in the embodiment of the present application includes the following steps:

S710. Acquire sensor data, where the sensor data includes head-up environment data collected by at least one head-up sensor and top-view environment data collected by at least two top-view sensors.

In this embodiment, the sensor data can be regarded as the data collected by the sensor on the electronic device. The content of the sensor data is not limited here, and may be determined according to the type of sensor included in the electronic device. For example, the sensor data may include: environment data collected by the top-view sensor (ie, top-view environment data), environment data collected by the head-up sensor (ie, head-up environment data), and data collected by the wheel odometer. The environment data includes head-up environment data and top-view environment data, and the environment data can be regarded as the data collected by the sensor representing the surrounding environment of the electronic device.

Before realizing the positioning, in this step, the sensor data collected by the sensor on the electronic device may be obtained first, so as to process the sensor data for positioning, and the acquisition method is not limited here. For example, a processor of an electronic device communicates with a sensor to obtain sensor data collected by the sensor.

S720. Process the sensor data.

In one embodiment, since the top-view sensor data (top-view environment data) is collected by at least two top-view sensors, processing the top-view sensor data may include splicing the top-view sensors collected by at least two top-view sensors Data manipulation for better positioning. Similarly, when multiple heads-up sensors collect head-up environment data, when processing the head-up environment data, the head-up environment data collected by multiple heads-up sensors can be spliced.

In addition, in order to improve the positioning efficiency, when processing the top-view environment data, the point cloud information of the surface edge in the top-view environment data can be extracted; the point cloud information of the surface edge in the head-up environment data can also be extracted. For accurate positioning, when processing sensor data, time stamps can be aligned between top-looking sensor data and head-up sensor data.

S730. Generate a head-up grid map and a top-view grid map according to the processed sensor data.

In this embodiment, the head-up grid map can be regarded as a map established based on the environment of the head-up direction of the electronic device (the running direction of the electronic device), and the top-view grid map can be regarded as a map established based on the environment of the electronic device's top-view direction. Exemplarily, the head-up grid map may be a grid map generated based on head-up environment data. The top-view grid map may be a grid map generated based on top-view environment data. The data used in the establishment of the head-up grid map and the top-view grid map are not limited here.

S740. Position the electronic device according to the processed sensor data, the top-view grid map, and the head-up grid map.

Exemplarily, the closed-loop detection can be calculated by calculating the matching rate of the point cloud information corresponding to the sensor data and the grid map, and the specific formula is as follows:

In the formula: k: represents the number of preprocessed laser points in the current frame; T: represents the pose of the current frame in the map; h _i : among the preprocessed laser points in the current frame The height value of the laser point converted from attitude T; μ _i , σ _i : indicates the i-th laser point in the laser point after preprocessing in the current frame is projected to the grid where the grid map falls according to the pose T Gaussian distribution parameter for the height values of all laser points.

Score represents the matching situation between the current point cloud information and the grid map, that is, the matching rate. The Score is in the range of 0-1. The larger the score, the better the matching, and the more likely it is a closed loop. After the Score is determined, the relative positions of the two nodes can be obtained, and then the global pose graph can be optimized based on the relative positions of the two nodes.

A node is an abstract concept, which represents the information encapsulated by a measurement, such as point cloud information and pose information; the global pose graph can be made based on the relative position of the pose between two nodes obtained from the closed-loop information. After the global pose graph is optimized The pose information of the node will be adjusted, and finally the optimized global pose result will be output.

When optimizing the global pose graph, the cost function is as follows:

In the formula: c _i , c _j : represent the pose information of nodes i and j respectively; Ri: represent the rotation vector of node i; t _i , t _j : represent the pose translation vectors of nodes i and j respectively; θ _i , θ _j : represent the pose angle vectors of nodes i and j respectively; Z _ij : laser matching between nodes i and j, that is, the pose transformation between two observed laser frames; X2: pose graph residual , Λij is the part corresponding to ij in the information matrix, which represents the amount of observation information between i and j, and is used as the weight when optimizing the global pose graph.

In a positioning method provided by an embodiment of the present application, first, sensor data is obtained, and the sensor data includes head-up environment data and top-view environment data; then the sensor data is processed; secondly, a head-up grid map and a head-up grid map are generated according to the processed sensor data A top-view grid map; finally, the electronic device is positioned according to the processed sensor data, the top-view grid map, and the head-up grid map. With the above technical solution, since the environment above the computer equipment is not easy to change, the head-up grid map generated by combining the top-view environment data collected by multiple top-view sensors and the head-up environment collected by the head-up sensor The influence of the environment on the positioning of the electronic equipment is avoided, and the robustness of the positioning is improved.

When processing sensor data, the sensor data may be preprocessed first, and the preprocessing means are not limited. The preprocessing means may be determined based on the content of the sensor data. For example, the corresponding relationship between sensor data can be established based on time; the top-view environment data can also be spliced; the point cloud information that is convenient for positioning in the top-view environment data can be extracted, and the point cloud information in the head-up environment data can be extracted.

Exemplarily, the preprocessing means of the head-up environment data in the sensor data includes but not limited to: time stamp alignment, extracting the features in the head-up sensor data after alignment, and segmenting the point cloud information of the surface edge in the head-up sensor data.

After optimizing the sensor data after converting the coordinate system, the processed top-view environment data and the processed head-up environment data can be obtained according to the sensor data, such as reading the optimized top-view environment data in the sensor data as the processed top-view environment Data, read the optimized head-up environment data in the sensor data as the processed head-up environment data.

This embodiment refines the operation of processing sensor data to ensure that the processed sensor data can be positioned

The sensor data after transforming the coordinate system is processed by brute force matching, which can eliminate the influence of initial value sensitivity. The method of violent matching is not limited here, such as optimizing the converted data through the Correlation Scan Match (CSM) algorithm. CSM can calculate the relative pose between the laser and the map. Optimizing the sensor data after transforming the coordinate system through CSM can make the positioning result more accurate.

This embodiment refines the specific technical means for optimizing the sensor data after the coordinate system conversion, so that the top-view grid map and the horizontal grid map generated based on the sensor data after the coordinate system optimization conversion can be positioned more accurately.

In one embodiment, the preprocessing of the sensor data includes: aligning the head-up environment data and the top-view environment data based on time stamps; splicing the aligned top-view environment data; extracting the top-view environment data The point cloud information of the face edge in .

This embodiment refines the technical means of preprocessing the sensor. When preprocessing the sensor data, the top-view environment data and the head-down environment data may be aligned based on the time stamp first.

Since the top-view environment data is collected by at least two top-view sensors, when preprocessing the sensor data, the aligned top-view environment data can be spliced to perform positioning based on the spliced top-view environment data. The method of splicing is not limited here.

After splicing the aligned top-view environment data, in order to facilitate the generation of the top-view grid map, this embodiment can extract the point cloud information of the surface edge in the top-view environment data, so as to map the point cloud information to the top-view grid In order to facilitate the generation of the head-up grid map, the point cloud information of the edge of the face in the aligned head-up environment data can be extracted, so that the point cloud information can be mapped to the head-up grid map to realize the mapping and positioning.

Fig. 15 is a schematic flowchart of another positioning method provided in the embodiment of the present application. Referring to Fig. 15, the method includes the following steps:

S810. Acquire sensor data, where the sensor data includes head-up environment data collected by at least one head-up sensor and top-view environment data collected by at least two head-up sensors.

S820. Align the head-up environment data with the top-view environment data based on the time stamp.

Aligning the head-up environment data and the top-view environment data based on the time stamp can be considered as establishing at least one sensor data correspondence based on time, so as to process the aligned top-view environment data, and then realize positioning.

S830. Stitching the aligned top-view environment data.

The method of splicing the aligned top-view environment data is not limited. For example, the splicing can be realized based on the top-view environment data in the common-view area, so as to splice the top-view environment data collected by multiple top-view sensors. When there is no common-view area, the aligned top-view environment data can be spliced based on the positions of multiple top-view sensors in the electronic device.

S840. Extract point cloud information of surface edges in the spliced top-view environment data.

After splicing and aligning the top-view environment data, this embodiment may extract point cloud information of surface edges in the spliced top-view environment data for positioning. The specific means for extracting the point cloud information of the surface edge is not limited here.

S850. Transform the preprocessed sensor data into the body coordinate system.

This example only shows the process of processing the top-view environment data, and the technical means for preprocessing the rest of the sensor data are not limited here.

S860. Optimizing the sensor data after the transformed coordinate system, and obtaining processed top-view environment data and processed head-up environment data according to the sensor data.

S870. Generate a head-up grid map and a top-view grid map according to the processed sensor data.

S880. Position the electronic device according to the processed sensor data, the top-view grid map, and the head-up grid map.

In one embodiment, the positioning of the electronic device according to the processed sensor data, the top-view grid map, and the head-up grid map includes: performing closed-loop detection on the head-up grid map to obtain head-up matching rate; performing closed-loop detection on the top-view grid map to obtain the top-view matching rate; when the head-up matching rate is greater than the first set threshold and/or the top-view matching rate is greater than the second set threshold, determine the The global pose of the electronic device.

Fig. 16 is a schematic flowchart of another positioning method provided in the embodiment of the present application. Referring to Fig. 16, the method includes the following steps:

S910. Acquire sensor data, where the sensor data includes head-up environment data collected by at least one head-up sensor and top-view environment data collected by at least two head-up sensors.

S920. Process the sensor data.

S930. Generate a head-up grid map and a top-view grid map according to the processed sensor data.

S940. Perform loop closure detection on the head-up grid map to obtain a head-up matching rate.

S950. Perform loop closure detection on the top-view grid map to obtain a top-view matching rate.

In this embodiment, based on the top-view grid map and the top-view environment data, the top-view matching rate can be obtained by performing closed-loop detection on the top-view grid map.

When determining the head-up matching rate and the top-view matching rate, loop closure detection, also known as loop-back detection, can be performed on the head-up grid map and the top-view grid map to determine the corresponding top-view matching rate and horizontal matching rate. Here, the execution order of determining the head-up matching rate and the top-view matching rate is not limited. The head-up matching rate and the top-view matching rate can be determined in parallel, or can be determined sequentially.

S960. When the head-up matching rate is greater than a first set threshold and/or the top-view matching rate is greater than a second set threshold, determine the global pose of the electronic device.

Fig. 17 is a schematic flowchart of a multi-layer grid map positioning method provided in the embodiment of the present application. Fig. 17 takes two top-view sensors as an example, and the method includes:

S1. Obtain sensor data.

S2. Data preprocessing.

S2.1. Align the time stamps of the sensor data collected by multiple sensors, and splicing the double top-view depth camera data, that is, the top-view environment data collected by two top-view sensors;

S3. Prediction of pose by wheel odometer.

S4. Coordinate system conversion.

Transfer the point cloud information of the lidar and the point cloud information of the surface edge of the top-view environment data to the body coordinate system, and obtain the point cloud information after coordinate transformation, where the body coordinate system can be customized or wheeled The coordinate system of the odometry.

S5. Optimizing point cloud information.

S6. Updating the probability corresponding to the grid in the grid map.

S6.1. Update the head-up grid map based on the head-up laser data;

S6.2. Update the top-view grid map based on the top-view edge point cloud.

S7. Closed-loop detection.

S7.2. Detect top-view closed loops through the top-view face edge point cloud and top-view grid map.

S8. Global pose graph optimization, output global pose.

With the development of automation control technology, mobile service robots are gradually being applied in industrial production and commercial services. The important working premise of mobile service robot is accurate location information acquisition. A common way to obtain location information in the related art is that the mobile server relies on a laser radar sensor to obtain the location information. Although lidar has the ability of positioning accuracy and anti-interference, but because mobile server robots often run in more complex scenes, the application of lidar is limited, for example, the feature frame identified by lidar in crowded scenes The large space-time transformation makes the pose determined by the motion server robot have a large error, and the field of view of the lidar is restricted by the crowd, resulting in limited data collection by the lidar. Mobile service robots need a high-precision localization method when operating in this densely populated scene.

As mentioned above, the service quality of robots mainly depends on accurate location information. Mobile robots mostly use lidar sensors to measure location information, but this is limited by the scene. The characteristics of the variable are large and cannot be precisely located. The technical solution of the present application improves the accuracy of robot positioning by performing mapping and positioning based on depth data in a preset direction (for example, basically unchanged indoor ceiling features) in a short period of time.

Fig. 18 is a schematic flow chart of another positioning method provided by the embodiment of the present application. This embodiment is applicable to robot positioning in crowded scenes. The method can be performed by a positioning device, which can use hardware and/or It can be implemented by means of software. Referring to FIG. 18, the positioning method provided by the embodiment of the present application includes the following steps:

S1010. Collect the current pose of the robot and depth data in at least one preset direction.

The current pose can be information representing the current position and state of the robot, and can include the position coordinates in the world coordinate system and the angle between the robot direction and the X-axis of the world coordinate system.

At least one preset direction may be at least one of a horizontal direction and a vertical direction.

Depth data in at least one preset direction can be collected based on sensors preset on the robot.

S1020. Determine a feature point cloud based on the depth data.

A feature point cloud is a point cloud that characterizes features in the depth data.

After collecting the depth data in at least one preset direction, the feature point cloud can be determined based on the depth data, for example, extracting the outer contour points of at least one plane from the depth data and forming the feature point cloud from the outer contour points of the at least one plane .

S1030. Determine an obstacle score of the feature point cloud in a preset global grid map, wherein the preset global grid map is formed based on historical point clouds.

The preset global grid map may include one or more grids, and each grid may include a probability value that the robot is located in the grid.

The preset global grid map is divided into obstacle areas, barrier-free areas and unknown areas.

Exemplarily, each point in the feature point cloud can be mapped to a grid in the preset global grid map, and the probability corresponding to the grid is used as the obstacle score of the point, and the feature point cloud The sum of the obstacle scores of all the points mapped to the obstacle region in , is used as the obstacle score of the feature point cloud.

S1040. Optimizing the current pose according to the obstacle score to determine positioning information of the robot.

The positioning information can reflect the position information of the robot at the current moment, and can include the coordinates of the robot in the world coordinate system and the angle between the robot and the X-axis of the world coordinate system.

Exemplarily, the obstacle score of the feature point cloud can be used as the constraint condition of the current pose, the current pose can be optimized on the basis of the above obstacle score, and the optimized current pose can be used as the positioning information of the robot. It can be understood that the way to optimize the current pose can be optimized by nonlinear least square method and Lagrange multiplier method.

The technical solution of the present application improves the accuracy of robot positioning by performing mapping and positioning based on depth data in a preset direction within a short period of time.

Fig. 19 is a schematic flowchart of another positioning method provided by the embodiment of the present application. Referring to FIG. 19 , the method provided by the embodiment of the present application includes the following steps.

S1110. Collect the current pose of the robot, depth data in at least one preset direction, and infrared data in at least one preset direction.

The current pose can be information representing the current position and state of the robot, and can include the position coordinates in the world coordinate system and the angle between the robot direction and the X-axis of the world coordinate system. Fig. 20 is an example diagram of a pose provided by the embodiment of the present application. Referring to Fig. 20, the current pose of the robot in the embodiment of the present application may contain three unknown quantities, namely x, y and yaw, where x represents The robot is used as the abscissa in the world coordinate system, y represents the ordinate in the world coordinate system, and yaw represents the angle between the robot direction and the X-axis of the world coordinate system. The preset direction can be a preset data collection direction, which can be set by the user or the service provider. The preset direction can be any direction in space, for example, the vertical direction of the robot and the horizontal direction of 45 degrees upward of the robot. The depth data can be data reflecting the distance from the object to the robot, and the depth data can be collected by sensors installed on the robot. The infrared data can be the data collected by the infrared sensor, and the infrared data can be generated by collecting objects in the preset direction of the robot by the infrared sensor, and can indicate the distance between the robot and the object.

In the embodiment of the present application, the current pose of the robot can be acquired by using the sensors arranged on the robot, for example, the moving distance of the robot can be collected by inertial navigation or displacement sensors to determine the current pose of the robot. Depth data and infrared data can be collected in a preset direction of the robot, for example, depth data of obstacles can be collected in the direction of the top of the robot using a Time Of Flight (TOF) camera. The robot can collect depth data and infrared data in multiple preset directions to further improve the accuracy of robot positioning.

S1120. Extract at least one edge point from the infrared data, and convert the at least one edge point into a feature point cloud according to the depth data.

The edge point can be the position point at the edge position in the image composed of infrared data, and the edge point can be detected in the infrared data through differential edge detection, Reborts operator edge detection, Sobel edge detection, Laplacian edge detection, Prewitt operator edge detection, etc. detection.

Exemplarily, the edge points can be extracted from the infrared data according to the edge detection method, and the coordinates of at least one edge point can be converted from two-dimensional coordinates to three-dimensional coordinates according to the depth value of the depth data. For example, each edge point can be extracted from the depth data. The depth value corresponding to each edge point can be used as the third dimension of the three-dimensional coordinates of each edge point. The edge points after coordinate transformation can be composed into a feature point cloud.

S1130. Determine the obstacle score of the feature point cloud in the preset global grid map, where the preset global grid map is formed based on the historical point cloud.

The obstacle score may be the total probability score of the grid mapped to the edge point of the grid in the obstacle area when all the edge points in the feature point cloud are mapped to the preset global grid map. The preset global grid map can be composed of historical point clouds, which can reflect the situation in the space where the robot is located. The preset global grid map can include one or more grids, and each grid can include grid probability value. The preset global grid map can be gradually improved during the movement of the robot. The obstacle score may be the sum of the probability values that the edge point is located at the obstacle position in the preset global grid map. FIG. 21 is an example diagram of a preset global grid map provided by an embodiment of the present application. Referring to Fig. 21, the preset global grid map can include three parts: unknown area, obstacle area and unobstructed area, and the obstacle score can be determined by the sum of the probability values of the grids in the obstacle area mapped to the statistical feature point cloud.

In this embodiment of the application, all the edge points in the feature point cloud can be mapped to the preset global grid map one by one, the probability value of the corresponding grid where each edge point is located can be determined, and the edge points can be counted. The sum of the probability values of the grids in the obstacle area is used as the obstacle score of the feature point cloud in the preset global grid map.

S1140. Optimizing the current pose according to the obstacle score to determine the positioning information of the robot.

The obstacle score of the feature point cloud can be used as the constraint condition of the current pose, the current pose can be optimized on the basis of the above obstacle score, and the optimized current pose can be used as the positioning information of the robot. The way to optimize the current pose can be nonlinear least square optimization and Lagrange multiplier optimization.

In the embodiment of the present application, by acquiring the current pose of the robot and the depth data and infrared data in the preset direction, extracting at least one edge point from the infrared data, processing at least one edge point according to the depth data to form a feature point cloud, and determining the feature point The obstacle score of the cloud in the preset global grid map, using the obstacle score to optimize the current pose to obtain the robot's positioning information, realizes the accurate acquisition of the robot's position information, reduces the impact of complex environments on pose determination, and can enhance the robot Service quality helps to improve user experience.

Fig. 22 is a schematic flowchart of another positioning method provided by the embodiment of the present application, and the embodiment of the present application is described on the basis of the foregoing embodiments. Referring to Fig. 22, the method provided by the embodiment of the present application includes the following steps.

S1210. Obtain the current pose of the robot in the world coordinate system, wherein the current pose at least includes an abscissa, a ordinate, and an angle between the robot direction and the X-axis of the world coordinate system.

The world coordinate system can be the absolute coordinate system of the robot, and the origin of the world coordinate system can be determined when the robot is initialized.

In the embodiment of the present application, the current pose may include three elements, the abscissa, the ordinate, and the angle between the robot direction and the X-axis of the world coordinate system, and the current pose may be acquired by sensors set in the robot. The data collected by the sensor of the robot can be obtained, and based on the data collected by the sensor, the abscissa and ordinate of the robot in the world coordinate system at the current moment and the angle between the robot direction and the X-axis of the world coordinate system are determined as the current pose.

S1220. Use at least one depth data sensor pre-set on the robot to collect depth data in at least one preset direction and at least one infrared data sensor to collect infrared data in at least one preset direction, wherein the preset direction includes at least a horizontal direction and a vertical direction At least one of the vertical directions.

The depth data sensor can be a device that collects depth data. It can collect the distance from the robot to the object to be collected, and can perceive the depth of objects in space. The depth data sensor can include structured light depth sensors, camera array depth sensors, and time-of-flight depth sensors. Infrared Data sensors, etc. Infrared data sensors can be devices that generate thermal images of objects to be collected. Infrared data sensors can include infrared imagers and time-of-flight cameras. Depth data sensors and infrared data sensors on robotic devices can be used for integrated data acquisition A device, such as a TOF camera, can directly control the depth data and infrared data of the collected object collected by the TOF camera. At least one preset direction may be at least one of a horizontal direction and a vertical direction of the robot.

In the embodiment of the present application, a depth data sensor and an infrared data sensor are pre-installed on the robot, and the depth data sensor and the infrared data sensor can be used to respectively collect horizontal or vertical data of the robot. A plurality of depth data sensors and a plurality of infrared data sensors can be preset on the robot, and the preset direction of data collection preset by multiple depth data sensors can be different, and the preset direction of data collection preset by multiple infrared data sensors can be Differently, the preset direction may be a preset direction set by a user or a service provider, for example, it may be a direction convenient for collecting indoor ceiling features, and the preset direction may include directions such as vertically upward or horizontally upward at 45 degrees.

S1230. Filter out noise in the infrared data.

In the embodiment of this application, due to the influence of the environment during the infrared data collection process, there is noise in the data, which reduces the accuracy of edge point collection. In order to improve the accuracy of robot positioning, the noise in the infrared data can be filtered . The filtering method may include Gaussian filtering, bilateral filtering, median filtering, mean filtering and other methods. The Gaussian filter can be a linear smoothing filter, which can eliminate Gaussian noise during image processing and achieve noise reduction of image data.

Exemplarily, taking Gaussian filtering on the collected infrared data as an example to eliminate noise in the infrared data, a template (convolution or mask) can be used to scan each pixel in the infrared data, and the template can be used to determine the The weighted average gray value replaces the value of the center pixel of the template to achieve noise filtering of infrared data.

S1240. Extract at least one edge point from the noise-filtered infrared data.

In the embodiment of the present application, after the infrared data is filtered out, edge points may be extracted from the image formed by the infrared data. All the edge points in the infrared data can be extracted, or an edge point can be extracted at a certain distance to further improve the efficiency of robot positioning. The way of extracting edge points can be through image recognition, for example, the points in the image formed by the infrared data that have a large color difference from other surrounding pixel points can be regarded as edge points.

In an exemplary embodiment, the infrared data can be processed according to the Canny edge detection algorithm, and the infrared data is sequentially subjected to Gaussian filtering to reduce noise, using the finite difference of the first-order partial derivative to calculate the magnitude and direction of the gradient, and the gradient magnitude Perform non-maximum suppression and use a double-threshold algorithm to detect and connect edges to obtain edge points in infrared data. The Canny edge detection algorithm can be an algorithm for detecting the edge of an image, for example, it can include image Gaussian filtering to reduce noise, use the finite difference of the first-order partial derivative to calculate the magnitude and direction of the gradient, and perform non-maximum suppression and Steps such as detecting and connecting edges using a double-threshold algorithm. You can also use: Sobel edge detection, Prewitt edge detection, Roberts edge detection, Canny edge detection, Marr-Hildreth edge detection and other methods to detect edge points in infrared data.

S1250. Convert the two-dimensional coordinates of at least one edge point into three-dimensional coordinates by using the camera model of the robot and the depth data to form a feature point cloud.

The camera model can be a camera model established by the robot for three-dimensional transformation, and can be used to correct the coordinates of edge points using depth data to obtain a feature point cloud with less distortion. The depth data may include depth information of at least one edge point, and the depth information may serve as third-dimensional information corresponding to the three-dimensional coordinates of the edge point. The camera model may include one or more of an Euler camera model, a UVN camera model, a pinhole camera model, a fisheye camera model, and a wide-angle camera model.

The coordinates of each edge point can be converted using the preset camera model as the reference system, so that the two-dimensional coordinates of the edge point and the coordinates of the world coordinate system are in the same reference system, and at least one edge point corresponding to the depth data can be determined For depth information, the two-dimensional coordinates and depth information of each edge point can be used to form three-dimensional coordinates, and multiple edge points with three-dimensional coordinates can be used to form a feature point cloud. Exemplarily, the conversion of the two-dimensional data of the edge points to the three-dimensional point cloud can be realized in the following conversion manner:

Among them, Z is the depth information of the edge point, (u, v) is the two-dimensional coordinates of the edge point, K is the internal parameter matrix of the camera, which can be determined by the camera model, and P is the coordinate of the three-dimensional point cloud (feature point cloud).

S1260. Transform the coordinates of at least one edge point in the feature point cloud into the world coordinate system, and map each edge point after the coordinate transformation to the target grid of the preset global grid map.

Coordinate system transformation may be performed on the coordinates of at least one edge point in the feature point cloud, so that the coordinates of at least one edge point are based on the world coordinate system. Fig. 23 is an example diagram of a coordinate transformation provided by the embodiment of the present application. See Figure 23. The collected edge points are located in the robot coordinate system, and the camera model corresponds to a coordinate system. The depth data can be added to the edge points in the robot coordinate system according to the coordinate system of the camera model to achieve undistorted or low-distorted coordinates. Transform, and then convert the three-dimensional coordinates to world coordinates. Exemplarily, the conversion process can be expressed by the following formula: w _i =Proj(T*p _i ), wherein, w _i represents the coordinates of the i-th edge point in the world coordinate system, and p _i represents the coordinates of the i-th edge point in the robot The coordinates under the coordinates, the Proj() function maps the three-dimensional coordinates to the two-dimensional coordinates, T is the current pose of the robot, represented by x, y and yaw, and T can be expressed as follows:

In the embodiment of the present application, after determining the coordinates of at least one edge point in the world coordinate system, at least one edge point can be mapped to the target grid of the preset global grid map in sequence according to the coordinates, for example, the preset global grid Different grids in the grid map have different coordinate ranges, at least one edge point can be mapped to the corresponding grid according to their respective coordinate ranges, and the grid with the edge point mapping can be marked as the target grid.

S1270. If the target grid to which an edge point is mapped is a grid within the obstacle area in the preset global grid map, acquire the probability value of the target grid as the obstacle score of the one edge point.

Grids within obstacle areas can be identified by information.

The target grid with edge point mapping in the preset global grid map can be tested. If the target grid is in the obstacle area, the probability value stored in the target grid will be used as the obstacle of the corresponding edge point Score.

If the target grid mapped to an edge point is not a grid in the obstacle area, for example, the grid mapped to the edge point is an unobstructed area or an unknown area in the preset global grid, then the edge point can not be The corresponding probability value is counted, and the edge point can be deleted or the probability value stored in the grid corresponding to the edge point can not be acquired.

S1280. Count the sum of the obstacle scores of all edge points in the feature point cloud as the obstacle score of the feature point cloud.

In the embodiment of the present application, the obstacle scores of all edge points in the feature point cloud can be counted, and the sum of the obstacle scores can be used as the obstacle score of the feature point cloud.

S1290. Construct a residual function according to the current pose and the obstacle score of the feature point cloud.

In the embodiment of the present application, the residual function for optimizing the current pose can be constructed according to the current pose and the obstacle score of the feature point cloud, where the residual function can be the functional relationship of the optimized robot pose, which can represent the robot's current The relationship between the pose and the obstacle score, the residual function can be an optimization function formed by a nonlinear least squares problem, for example,

Among them, e ₁ is the residual, p _k is the kth edge point in the feature point cloud, M(T,p _k ) is the kth edge point p _k is projected to the preset global grid when the robot pose is T The obstacle score calculated by the map, n is the number of edge points in the feature point cloud.

S12100. Adjust the parameter information in the residual function to minimize the result value of the residual function. The parameter information in the residual function is at least one of the abscissa and ordinate of the current pose and the angle between the robot direction and the X-axis of the world coordinate system. The value of the parameter.

The value of the current pose in the residual function can be adjusted to minimize the result value of the residual formula, and the methods for adjusting the current pose can include gradient descent method, Newton method, and quasi-Newton method.

S12110, taking the abscissa and ordinate of the current pose and the angle between the robot direction and the X-axis of the world coordinate system when the result value is the minimum as positioning pose information.

The positioning pose information may be pose information used for robot positioning, and the positioning pose information may indicate the most likely current state of the robot.

In the embodiment of the present application, when the result value of the residual function is the smallest, it can be determined that the current pose is optimally optimized, and the abscissa, ordinate, and robot direction of the adjusted current pose can be compared with the world coordinates The included angle of the X-axis of the system is used as the output positioning pose information.

In this embodiment of the application, by acquiring the current pose of the robot in the world coordinate system and using the depth sensor installed on the robot to collect depth data and infrared data in a preset direction, filtering out the noise contained in the infrared data, using the camera model of the robot And the depth data process the infrared data into a feature point cloud, convert the coordinates of at least one edge point in the feature point cloud to the world coordinate system, and map each edge point to the target grid of the preset global grid map, in the target grid Obtain the probability value of the target grid as the obstacle score of the corresponding edge point when it is the obstacle position, and the sum of the obstacle scores of at least one edge point of the statistical feature point cloud is used as the obstacle score of the feature point cloud, which is constructed using the obstacle score and the current pose Residual function, adjust the value of the current pose in the residual function, so that the result value of the residual function is the smallest, and the current pose when the result value is the smallest is used as the positioning information of the robot. In the embodiment of the present application, the infrared data corresponding to The edge points form a feature point cloud, which reduces the amount of data calculation without changing the accuracy of positioning information, improves the efficiency of positioning information determination, optimizes the current pose by using obstacle scores, improves the accuracy of robot position information determination, and enhances the service quality of robots , which helps to improve the user experience.

On the basis of the above embodiments, the method further includes: optimizing the positioning information according to the moving speed and obstacle score of the robot.

On the basis of the embodiment of the present application, the moving speed of the robot can also be obtained, and the current pose can be optimized by using the moving speed and the obstacle score to obtain positioning information to further improve the positioning accuracy of the robot. For example, the robot's moving speed can be collected Moving speed, constituting a restriction condition based on the moving speed, based on the restriction condition, a nonlinear least squares problem can be constructed for the positioning information obtained after optimizing the current pose based on the obstacle score, and the problem can be solved according to the gradient descent method. In the nonlinear least squares When the result value of the multiplication problem is the smallest, the final positioning information can be obtained.

Fig. 24 is a flowchart of another positioning method provided by the embodiment of the present application. The implementation of the present application is described on the basis of the above embodiment. Referring to Fig. 24, the method provided by the embodiment of the present application includes the following steps.

S1310. Collect the current pose of the robot, depth data in at least one preset direction, and infrared data in at least one preset direction.

S1320. Extract at least one edge point from the infrared data, and convert the at least one edge point into a feature point cloud according to the depth data.

S1330. Determine the obstacle score of the feature point cloud in the preset global grid map, where the preset global grid map is formed based on the historical point cloud.

S1340. Construct a first residual item according to the current pose and the obstacle score of the feature point cloud.

In the embodiment of the present application, the first residual term corresponding to the nonlinear least squares problem can be constructed based on the current pose and the obstacle score of the feature point cloud, for example,

Among them, e ₁ is the residual, p _k is the kth edge point in the feature point cloud, M(T,p _k ) is the kth edge point in the feature point cloud, p _k is projected to the grid when the robot pose is T The obstacle score calculated by the grid map, n is the number of edge points in the feature point cloud.

S1350. Determine the predicted pose based on the moving speed, and use the difference between the predicted pose and the historical pose at the previous moment as a second residual term.

The predicted pose may be the pose of the robot determined according to the moving speed. For example, the moving position of the robot may be determined through the moving speed, and the pose of the robot may be determined according to the moving position as the predicted pose.

The moving speed of the robot can be collected, the moving speed can be used to generate the position of the robot, and the predicted pose can be determined according to the position, based on the difference between the predicted pose and the historical pose at the previous moment as the second residual term, where , the historical pose can be the pose information determined by the robot at the last moment, which can include the coordinates and the angle between the robot's direction and the abscissa axis of the world coordinate system.

S1360. Adjust the parameter information in the predicted pose in the first residual item and the second residual item and/or the parameter information in the current pose so that the sum of the first residual item and the second residual item is the smallest, and the parameter information Including the value of at least one parameter among the abscissa, ordinate and the angle between the robot direction and the X-axis of the world coordinate system.

In the embodiment of the present application, at least one parameter among the abscissa, ordinate, and the angle between the robot direction and the X-axis of the world coordinate system in the predicted pose and/or the current pose can be adjusted separately, so that the first residual term and the second residual term have the smallest value, and the adjustment methods can include gradient descent method, Newton method and quasi-Newton method, etc.

S1370, taking the predicted pose and/or the abscissa and ordinate of the current pose when the sum of the first residual term and the second residual term is the smallest, and the angle between the robot direction and the X-axis of the world coordinate system as positioning pose information.

When the sum of the first residual term and the second residual term is the smallest, the current pose of the robot can be optimized, and the adjusted predicted pose and/or current pose information can be used as the robot’s The positioning pose information can be used as the positioning information used in the robot positioning process, wherein the relevant information includes the abscissa, the ordinate, and the angle between the robot direction and the X-axis of the world coordinate system.

S1380. Update the pose of the robot corresponding to the positioning information to the preset global grid map.

The final pose of the robot can be determined according to the abscissa, ordinate, and the angle between the robot direction and the X-axis of the world coordinate system in the positioning information. Based on the final pose, multiple The probability value of the grid, and the probability value is added to the corresponding grid in the corresponding preset global grid map, so as to realize the update of the preset global grid map.

In the embodiment of the present application, by collecting the depth data and infrared data of the robot's current pose and preset direction, extracting at least one edge point from the infrared data, and processing at least one edge point according to the depth data, the processed at least one edge constitutes a feature Point cloud, determine the obstacle score of the feature point cloud in the preset global grid map, construct the first residual item based on the obstacle score and the current pose, and construct the second residual item based on the predicted pose determined by the moving pose and the historical pose The difference term, adjust the current pose and the predicted pose so that the sum of the first residual term and the second residual term is the smallest, and the abscissa and ordinate of the current pose and the predicted pose when the sum is minimized, as well as the direction of the robot and the world The angle between the X-axis of the coordinate system is used as the positioning pose information of the robot, and the positioning pose information is used as the positioning information, and the pose corresponding to the positioning information is updated to the preset global grid map, which realizes accurate acquisition of robot positioning information and reduces The influence of complex environment on pose determination can enhance robot service quality and help improve user experience.

In an exemplary embodiment, FIG. 25 is an example diagram of a positioning method provided in the embodiment of the present application. Referring to FIG. 25 , the robot positioning and mapping method based on a top-view TOF camera may include the following steps:

Step 1: Obtain the top-view feature point cloud:

1. Perform Gaussian filter noise reduction processing on the collected infrared data.

2. Carry out Canny edge detection on infrared data to obtain edge points, mainly including steps such as finite difference calculation amplitude and direction, non-maximum suppression, detection and connection of edges with double threshold algorithm.

3. The edge point extracted in 2 is the coordinate point of the two-dimensional image layer, and the depth information of at least one edge point in the depth data and the camera model are used to obtain the three-dimensional feature point cloud. The conversion method is as follows:

Among them, Z is the depth information of the edge point, (u, v) is the two-dimensional coordinates of the edge point, K is the internal parameter matrix of the camera, which can be determined by the depth camera model, and P is the coordinate of the three-dimensional point cloud.

Step 2. Point cloud matching:

When the robot performs positioning and mapping, it can use the historical feature point cloud converted to the world coordinate system to construct a grid map, and use the newly generated feature point cloud to match the grid map. The matching process can include:

1. Convert the extracted feature point cloud P to the world coordinate system through the robot pose T. The conversion formula is w _i =Proj(T*p _i ), where w _i means that the i-th edge point is in the world coordinate system The coordinates below, p _i represents the coordinates of the i-th edge point under the robot coordinates, the Proj() function maps the three-dimensional coordinates to two-dimensional coordinates, T is the current pose of the robot, represented by x, y and yaw, T can be The formula is as follows:

2. Each edge point in the feature point cloud is mapped to a grid map, and the probability that the grid to which the edge point is mapped is an obstacle is taken as the matching score (obstacle score) of the edge point, and the sum of the matching scores of all points is recorded as The score of the feature point cloud:

s=1*p _cell

Among them, s is the matching score of an edge point, and p _cell is the occupancy probability of the grid to which the edge point is mapped.

3. Build a cost function model to optimize the pose T of the robot, and minimize the cost function by adjusting the pose T. The equation is as follows:

Among them, e ₁ is the residual, p _k is the kth edge point of the feature point cloud, M(T,p _k ) is the kth edge point of the feature point cloud p _k is projected to the grid when the robot pose is T The obstacle score calculated by the grid map, n is the number of edge points in the feature point cloud.

Step 3. Robot pose optimization

For the feature point cloud in step 2 and the vehicle speed (moving speed of the robot) as constraints to construct a nonlinear least squares problem together, jointly optimize the pose of the robot, including the following steps:

1. Build a cost function model to optimize the pose T of the robot, and adjust the pose T to minimize the error term of the cost function. The error term is as follows:

2. The error item _e3 obtained by constructing the vehicle speed:

e ₃ =LL _last

Among them, L is the pose of the robot at the current moment pushed by the vehicle speed, and L _last is the pose of the robot obtained at the previous moment.

3. For all error terms, construct the following optimization problem, and use the optimization library to find (Google ceres) solution to get the pose at the current moment:

(x,y,yaw)＝argmin∑|e ₁ |+|e ₃ |

4. For the feature point cloud obtained in step 2, transform it into the world coordinate system through the optimized pose, and update the grid map based on the feature point cloud.

Fig. 26 is a schematic flow chart of another positioning method provided by the embodiment of the present application. This embodiment is applicable to robot positioning in crowded scenes. The method can be performed by a positioning device, which can use hardware and/or It is realized by software, referring to FIG. 26 , the positioning method provided by the embodiment of the present application includes the following steps.

S1410. Collect the current pose of the robot and depth data in at least one preset direction.

The current pose can be information representing the current position and state of the robot, and can include the position coordinates in the world coordinate system and the angle between the robot direction and the X-axis of the world coordinate system. Referring to Fig. 20, the current pose of the robot in the embodiment of the present application may contain three unknown quantities, namely x, y and yaw, where x represents the abscissa of the robot in the world coordinate system, and y represents the vertical in the world coordinate Coordinates, yaw indicates the angle between the robot direction and the X-axis of the world coordinate system. The preset direction can be a preset data collection direction, which can be set by the user or the service provider. The preset direction can be any direction in space, for example, the vertical direction of the robot and the horizontal direction of 45 degrees upward of the robot. The depth data can be the data reflecting the distance from the object to the robot. The depth data can include the position information of the object in space and the distance from the depth data acquisition device. The depth data can be collected by sensors installed on the robot.

In the embodiment of the present application, the current pose of the robot can be collected using sensors installed on the robot. For example, inertial navigation or displacement sensors can be used to collect the moving distance to determine the current pose of the robot. Depth data may be collected in at least one preset direction of the robot, for example, depth data of obstacles may be collected in the direction of the top of the robot or in a 45-degree horizontal direction. It can be understood that the robot collects depth data in multiple preset directions to further improve the accuracy of robot positioning.

S1420. Extract the outer contour points of at least one plane from the depth data, and form the feature point cloud with the extracted outer contour points of at least one plane.

The plane can be one or more planes included in the point cloud composed of depth data. The plane can be divided and generated along the vertical or horizontal direction in the point cloud composed of depth data. The outer contour points can be the set of position points that constitute the outer contour of the plane. , through the outer contour points to represent all the position points of the plane, which can reduce the amount of data used by the robot positioning. The feature point cloud can be a collection of position points reflecting the characteristics of the depth data.

The depth data can be divided into one or more planes in different directions, and the position points at the outer contour points in each plane can be extracted, and the extracted multiple position points can form a feature point cloud, which can reflect the depth data through the feature point cloud Characteristics.

S1430. Determine the obstacle score of the feature point cloud in the preset global grid map, where the preset global grid map is formed based on the historical point cloud.

The obstacle score may be the total probability score of the grid that is mapped to the grid in the obstacle area when all the position points in the feature point cloud are mapped to the preset global grid map. The preset global grid map can be composed of historical point clouds, which can reflect the situation in the space where the robot is located. The preset global grid map can include one or more grids, and each grid can include grid probability value. The preset global grid map can be gradually improved during the movement of the robot. The obstacle score may be the sum of probability values that the location point is located at the obstacle location in the preset global grid map. Referring to Fig. 21, the preset global grid map may consist of three parts: unknown area, obstacle area and unobstructed area, and the obstacle score may be determined by the sum of the probability values of the statistical feature point cloud mapped to the grid in the obstacle area.

In the embodiment of this application, all the position points in the feature point cloud can be mapped to the preset global grid map one by one, the probability value of the corresponding grid where each position point is located can be determined, and the obstacles where the position point is located can be counted The sum of the probability values of the grids in the area is used as the obstacle score of the feature point cloud in the preset global grid map.

S1440. Optimizing the current pose according to the obstacle score to determine the positioning information of the robot.

In the embodiment of the present application, by acquiring the current pose of the robot and the depth data in the preset direction, extracting the outer contour points of at least one plane from the depth data, and using the outer contour points to form a feature point cloud, it is determined that the feature point cloud is in the preset The obstacle score in the global grid map, using the obstacle score to optimize the current pose to obtain the robot's positioning information, realizes the accurate acquisition of the robot's position information, reduces the impact of complex environments on the pose determination, and can enhance the robot's service quality. Help improve user experience.

Fig. 27 is a flowchart of another positioning method provided by the embodiment of the present application. The embodiment of the present application is described on the basis of the above-mentioned embodiments. Referring to Fig. 27, the method provided by the embodiment of the present invention specifically includes the following steps:

S1510. Obtain the current pose in the world coordinate system of the robot, wherein the current pose at least includes an abscissa, a ordinate, and an included angle between the robot direction and the X-axis of the world coordinate system.

S1520. Use at least one depth data sensor preset on the robot to collect depth data in the at least one preset direction, where the at least one preset direction includes at least one of a horizontal direction and a vertical direction.

A depth data sensor can be a device that collects depth data. It can collect the distance from the robot to the object to be collected, and can perceive the depth of objects in space. The depth data sensor can include a structured light depth sensor, a camera array depth sensor, and a time-of-flight depth sensor. At least one preset direction may be at least one of a horizontal direction and a vertical direction of the robot.

In the embodiment of the present application, a depth camera is preset installed on the robot, and the data collection direction of the depth camera can be the horizontal direction or the vertical direction of the robot, and the data collection direction of the depth camera can be a preset value set by the user or the service provider. direction. The robot can use the depth camera to collect the depth data of the object in the corresponding preset direction. A plurality of depth data sensors can be preset on the robot, and the preset directions of collecting data set by the multiple depth data sensors can be different, and the preset directions can be preset directions set by users or service providers, for example, it can be convenient for collecting For the direction of ceiling features, the preset direction can include vertical upward or horizontal upward 45 degrees, etc. The reliability of depth data is further improved by means of multi-direction and multi-data collection sources to enhance the accuracy of positioning information.

S1530. Filter out noise in the depth data.

In the embodiment of this application, due to the influence of the environment during the infrared data collection process, there is noise in the data, which reduces the accuracy of edge point collection. In order to improve the accuracy of robot positioning, the noise in the infrared data can be filtered remove. The filtering method may include Gaussian filtering, bilateral filtering, median filtering, mean filtering and other methods. The Gaussian filter can be a linear smoothing filter, which can eliminate Gaussian noise during image processing and achieve noise reduction of image data.

S1540. Convert the depth data into a three-dimensional point cloud based on the camera model of the robot, where the depth data at least includes position point information and depth information.

The depth data is composed of position point information and depth information. The position point information can form an image in a plane. The depth information can be the depth distance between the position point corresponding to the position point information and the acquisition device. The depth data can be a depth image. Each The three dimensions of a pixel point are abscissa, ordinate and depth information respectively. The camera model can be a camera model established by a robot for three-dimensional conversion, and can be used to correct the coordinates of edge points using depth data to obtain a feature point cloud with less distortion. The depth data can include depth information of multiple edge points. The depth The information may be used as third-dimensional information corresponding to the three-dimensional coordinates of the edge points. The camera model may include one or more of an Euler camera model, a UVN camera model, a pinhole camera model, a fisheye camera model, and a wide-angle camera model.

In the embodiment of the present application, the location point information and depth information in the depth data can be extracted, and one or more three-dimensional location points can be determined in space based on the location point information and depth information in the depth data, and the robot can be used to predict The established camera model converts the obtained 3D position points to reduce the distortion of the position points caused by the depth data acquisition device and improve the accuracy of positioning. The converted 3D position points can be used to construct a 3D point cloud. Exemplarily, the conversion from depth data to 3D point cloud can be realized in the following conversion manner:

Among them, Z is the depth information of the position point, (u, v) is the position point information of the depth data, K is the camera internal reference matrix, which can be determined by the camera model, and P is the coordinate of the 3D point cloud.

S1550. Divide the 3D point cloud into at least one plane according to the depth information and the preset normal vector information.

The depth information may be the depth information included in the depth data in the process of segmenting the plane, and the preset normal vector information may include the normal information and normal vector information used for segmenting the 3D point cloud. The preset normal vector information can be preset inside the system, or can be input by the user.

The depth information and the preset normal vector information can be obtained, and the 3D point cloud can be divided into multiple planes according to the depth information and the preset normal vector information. The division of the plane can be based on the normal vector information, and the maximum iteration depth of the plane division does not exceed the acquired depth. Exemplarily, the SACMODEL_PLANE model in the Point Cloud Library (PCL) function can be used to divide the obtained 3D point cloud, and the depth and normal vector information and the 3D point cloud can be input to the SACMODEL_PLANE model to obtain multiple flat.

S1560. Extract the outer contour points of each plane as a feature point cloud.

In the embodiment of the present application, in order to reduce the amount of calculation, the outer contour points in each plane may be selected to represent each plane, and the set of all extracted outer contour points may be used as a feature point cloud.

S1570. Convert the coordinates of each position point in the feature point cloud to the world coordinate system, and map each position point after coordinate conversion to the target grid of the preset global grid map.

The coordinates of all the position points in the feature point cloud can be transformed into a coordinate system, so that the coordinates of the feature point cloud are based on the world coordinate system. Referring to Figure 23, the extracted outer contour points are located in the robot coordinate system, and the camera model corresponds to a coordinate system. The depth data can be added to the outer contour points in the robot coordinate system according to the coordinate system of the camera model to achieve no distortion or low Distorted coordinate conversion, and then convert the three-dimensional coordinates to world coordinates. For example, the conversion process can be expressed by the following formula:

w _i ＝Proj(T*p _i ), where, w _i represents the coordinates of the i-th position point in the world coordinate system, p _i represents the coordinates of the i-th position point in the robot coordinates, and the Proj() function converts the three-dimensional Coordinates are mapped to two-dimensional coordinates, T is the current pose of the robot, represented by x, y and yaw, T can be formulated as follows:

In this embodiment of the application, after determining the coordinates of each location point in the world coordinate system, all location points can be mapped to the target grid of the preset global grid map in sequence according to the coordinates, for example, the preset grid map Different grids have different coordinate ranges, and all position points can be mapped to corresponding grids according to their respective coordinate ranges, and the grid with position point mapping can be marked as the target grid.

Step S1580, if the target grid to which an edge point is mapped is a grid within the obstacle area in the preset global grid map, obtain the probability value of the target grid as the obstacle score of the one location point.

Grids within obstacle areas can be identified by information.

It is possible to check the target grid with location point mapping in the preset global grid map. If the target grid is a grid in the obstacle area, the probability value stored in the target grid will be used as the obstacle of the corresponding location point Score.

If the target grid is not a grid in the obstacle area, for example, the grid mapped to the location point is an obstacle-free area or an unknown area in the preset global grid, then the probability value corresponding to the location point may not be counted. The location point can be deleted or the probability value stored in the grid corresponding to the location point can not be acquired.

S1590. Count the sum of the obstacle scores of all the position points in the feature point cloud as the obstacle score of the feature point cloud.

In the embodiment of the present application, the obstacle scores of all position points in the feature point cloud can be counted, and the sum of the obstacle scores can be used as the obstacle score of the feature point cloud.

S15100. Construct a residual function according to the current pose and the obstacle score of the feature point cloud.

Among them, e ₁ is the residual, p _k is the kth edge point in the feature point cloud, M(T,p _k ) is the kth edge point p _k is projected to the preset global grid when the robot pose is T The obstacle score calculated by the map, m is the number of location points in the feature point cloud.

S15110. Adjust the parameter information in the residual function to minimize the result value of the residual function. The parameter information in the residual function is at least one of the abscissa and ordinate of the current pose and the angle between the robot direction and the X-axis of the world coordinate system The value of the parameter.

Step 15120, take the abscissa and ordinate of the current pose when the result value is the smallest, and the angle between the robot direction and the X-axis of the world coordinate system as the positioning pose information.

In the embodiment of the present application, when the result value of the residual function is the smallest, it can be determined that the current pose is optimally optimized, and the abscissa, ordinate, and robot direction of the adjusted current pose can be compared with the world coordinates The included angle of the X-axis of the system is used as the positioning pose information.

In the embodiment of the present application, by obtaining the current pose of the robot in the world coordinate system and using the depth sensor installed on the robot to collect depth data in a preset direction, filtering out the noise contained in the depth data, using the robot's camera model and depth information Generate a 3D point cloud, divide the 3D point cloud into multiple planes according to the depth information and normal vector information processing, extract the outer contour points of each plane as a feature point cloud, and convert the coordinates of all position points in the feature point cloud to world coordinates System, map all position points to the target grid of the preset global grid map, obtain the probability value of the target grid as the obstacle score of the position point when the target grid is an obstacle position, and count the obstacles of all position points in the feature point cloud The sum of the scores is used as the obstacle score of the feature point cloud, using the obstacle score and the current pose to construct the residual function, adjusting the value of the parameter information in the current pose in the residual function, so that the result value of the residual function is the smallest, and the result The current pose when the value is the smallest is used as the positioning pose information of the robot. In the embodiment of the present application, the outline points of the plane in the 3D point cloud corresponding to the depth data are selected to form the feature point cloud, and the data calculation is reduced without changing the accuracy of the positioning information. Quantity, improving the efficiency of positioning information determination, using obstacle scores to optimize the current pose, improving the accuracy of robot position information determination, can enhance the service quality of the robot, and help improve the user experience.

On the basis of the embodiment of the present application, the moving speed of the robot can also be obtained, and the current pose can be optimized by using the moving speed and the obstacle score to obtain positioning information to further improve the positioning accuracy of the robot. For example, the robot's moving speed can be collected Moving speed, constituting a restriction condition based on the moving speed, based on the restriction condition, a nonlinear least squares problem can be constructed for the positioning information obtained after optimizing the obstacle score, and the problem can be solved according to the gradient descent method. In the nonlinear least squares problem When the result value is minimum, the final positioning information can be obtained.

Fig. 28 is a flow chart of another positioning method provided by the embodiment of the present application. The implementation of the present application is described on the basis of the above embodiments. Referring to Fig. 28, the method provided by the embodiment of the present application includes the following steps.

S1610. Collect the current pose of the robot and depth data in at least one preset direction.

S1620. Extract the outer contour points of at least one plane from the depth data, and form the outer contour points of at least one plane into a feature point cloud.

S1630. Determine the obstacle score of the feature point cloud in the preset global grid map, where the preset global grid map is formed based on the historical point cloud.

S1640. Construct a first residual item according to the current pose and the obstacle score of the feature point cloud.

In the embodiment of this application, the first residual term corresponding to the nonlinear least squares problem can be constructed based on the current pose and the obstacle score of the feature point cloud, for example,

Among them, e ₁ is the residual, p _k is the kth position point in the feature point cloud, M(T,p _k ) is the kth position point in the feature point cloud, p _k is projected to the grid when the robot pose is T The obstacle score calculated by the grid map, m is the number of position points in the feature point cloud.

S1650. Determine the predicted pose based on the moving speed, and use the difference between the predicted pose and the historical pose at the previous moment as a second residual term.

, the moving speed of the robot can be collected, the moving speed can be used to generate the position of the robot, and the predicted pose can be determined according to the position, based on the difference between the predicted pose and the previous historical pose as the second residual term, Wherein, the historical pose may be the pose information determined by the robot at the last moment, and may include the coordinates and the angle between the robot direction and the robot direction abscissa axis.

S1660. Adjust the parameter information in the predicted pose and/or the parameter information in the current pose in the first residual item and the second residual item to minimize the sum of the first residual item and the second residual item, the The parameter information includes the value of at least one parameter among the abscissa, ordinate and the angle between the robot direction and the X-axis of the world coordinate system.

In the embodiment of the present application, at least one parameter among the abscissa, ordinate, and the angle between the robot direction and the X-axis of the world coordinate system in the speed prediction pose and/or the current pose can be adjusted respectively, so that the first residual The sum of the term and the second residual term is the smallest, and the adjustment methods can include gradient descent method, Newton method, and quasi-Newton method.

S1670, taking the predicted pose and/or the abscissa and ordinate of the current pose when the sum of the first residual term and the second residual term is the smallest, and the angle between the robot direction and the X-axis of the world coordinate system as positioning pose information.

S1680. Update the robot pose corresponding to the positioning information to the preset global grid map.

In the embodiment of the present application, by collecting the depth data of the robot's current pose and preset direction, extracting the outer contour points of different planes from the depth data to form a feature point cloud, and determining the obstacle score of the feature point cloud in the preset global grid map , the first residual term is constructed based on the obstacle score and the current pose, the second residual term is constructed based on the predicted pose determined according to the mobile pose and the historical pose, and the current pose and the predicted pose are adjusted so that the first residual term The sum of the second residual term is the smallest, and the abscissa and ordinate of the current displacement and the predicted pose when the sum is the smallest, and the angle between the robot direction and the X-axis of the world coordinate system are used as the positioning information of the robot, and the positioning information corresponds to The pose of the robot is updated to the preset global grid map, which realizes accurate acquisition of robot positioning information, reduces the impact of complex environments on pose determination, enhances the quality of robot service, and helps improve user experience.

In an exemplary embodiment, FIG. 29 is an example diagram of another positioning method provided in the embodiment of the present application. Referring to FIG. 29, the robot positioning and mapping method based on a top-view TOF camera may include the following steps:

Step 1. Obtain the top-view feature point cloud:

1. Perform Gaussian filter noise reduction processing on the collected depth data.

2. Generate 3D point cloud based on depth data, and use depth information and camera model to obtain 3D point cloud. The conversion method is as follows:

Among them, Z is the depth information of the position point, (u, v) is the two-dimensional coordinates of the depth data (position point information), K is the camera internal reference matrix, which can be determined by the camera model, and P is the coordinate of the three-dimensional point cloud.

3. Segment the 3D point cloud in 2 according to the depth and normal vector information extraction, and obtain the equation of each plane. The segmentation and equation calculation methods will not be described in detail, such as the point based on random sampling consistency in the PCL library function Cloud plane segmentation method. Take the outer contour points of each plane as the feature point cloud.

Step 2. Point cloud matching:

1. Convert the extracted feature point cloud P to the world coordinate system through the robot pose T. The conversion formula is w _i =Proj(T*p _i ), where w _i represents the coordinates of the position point in the world coordinate system, p _i represents the coordinates of the position point under the robot coordinates. The Proj() function maps the three-dimensional coordinates to two-dimensional coordinates. T is the current pose of the robot, represented by x, y and yaw. T can be expressed as follows:

2. Each location point in the feature point cloud is mapped to the grid map, and the probability that the grid to which the edge point is mapped is an obstacle is taken as the matching score (obstacle score) of the location point, and the sum of the matching scores of all points is recorded as The score of the feature point cloud:

s=1*p _cell

Among them, s is the matching score of a location point, and p _cell is the occupancy probability of the grid to which the location point is mapped.

Among them, e ₁ is the residual, p _k is the kth position point of the feature point cloud, M(T,p _k ) is the kth position point of the feature point cloud, p _k is projected to the grid when the robot pose is T The obstacle score calculated by the grid map, m is the number of position points in the feature point cloud.

Step 3. Robot pose optimization

Among them, e ₁ is the residual, p _k is the kth position point of the feature point cloud, M(T,p _k ) is the kth position point of the feature point cloud, p _k is projected to the grid when the robot pose is T The obstacle score calculated by the grid map, m is the number of location points in the feature point cloud.

2. The error item _e3 obtained by constructing the vehicle speed:

e ₃ =LL _last

Among them, L is the pose of the robot at the current moment pushed by the speed of the vehicle, and L _last is the pose of the robot obtained after optimization at the last moment.

(x,y,yaw)＝argmin∑|e ₁ |+|e ₃ |

FIG. 30 is a schematic structural diagram of a positioning device provided in an embodiment of the present application. The device is applicable to the positioning of electronic equipment, and the device is configured in the electronic equipment. As shown in Figure 30, the device includes: an acquisition module 21, configured to acquire sensor data collected by at least one top-view sensor; a processing module 22, configured to process the sensor data; The data locates the electronic device.

The positioning device provided in this embodiment locates the electronic equipment based on the data collected by the top-view sensor. Since the environment above the computer equipment is not easy to change, the positioning method provided in this embodiment effectively improves the positioning accuracy of the electronic equipment.

Optionally, the electronic device includes at least two top-view sensors; the positioning module 23 is configured to match the processed sensor data with the local map data in the global map to determine the The pose of the electronic device in the global map.

In this embodiment, the device first acquires sensor data through the acquisition module 21; secondly, processes the sensor data through the processing module 22; Match the local map data in the global map to determine the pose of the electronic device in the global map.

This embodiment provides a positioning device, which effectively avoids the technical problem of poor positioning accuracy when the environment changes during positioning, and effectively improves the positioning accuracy.

In one embodiment, the processing module 22 is configured to: align the top-view environment data collected by at least two top-view sensors based on time stamps; and preprocess the aligned top-view environment data.

In one embodiment, the processing module 22 is configured to preprocess the aligned top-view environment data in the following manner: remove the noise points in the aligned top-view environment data; stitch the top-view environment data after the noise removal; extract the stitched top-view environment data The point cloud information of the surface edge in the top-view environment data of .

In one embodiment, the processing module 22 is configured to splice the top-view environment data after noise removal in the following manner: convert the top-view environment data after noise removal to the coordinate system where the target top-view sensor is located, and the target top-view The sensor is one of the at least two top-looking sensors.

In one embodiment, the positioning module 23 is configured to: convert the point cloud information of the edge of the surface included in the processed sensor data into raster data; determine the local map data in the global map that matches the raster data; The local map data and the grid data determine the pose of the electronic device in the global map.

In one embodiment, the determining module 23 is configured to determine the pose of the electronic device in the global map according to the local map data and the grid data in the following manner: according to the local map data and the grid data The pose relationship between the global maps, and the pose relationship between the grid data and the local map data determine the pose of the electronic device in the global map.

In one embodiment, the device further includes a mapping module, and the mapping module is configured to: if the mapping instruction is obtained, add the point cloud information of the edge of the surface included in the processed sensor data to the processed sensor In the local map data for data matching; update the local map data after adding the processed sensor data to the global map.

Optionally, the sensor data includes head-up environment data collected by at least one head-up sensor and top-view environment data collected by a top-view sensor; the positioning module is configured to generate a head-up grid map and a top-view grid map based on the processed sensor data A viewing grid map: positioning the electronic device according to the processed sensor data, the top-viewing grid map, and the head-up viewing grid map.

In this embodiment, the device first obtains sensor data through the acquisition module 21, and the sensor data includes head-up environment data and top-view environment data; secondly, the sensor data is processed through the processing module 22; The head-up grid map and the top-view grid map are generated from the sensor data, and the electronic device is positioned according to the processed sensor data, the top-view grid map, and the head-up grid map.

This embodiment provides a positioning device, which avoids the influence of the environment on the positioning of electronic devices by combining the top-view grid map and the flat-view grid map, thereby improving the robustness of positioning.

Referring to FIG. 31 , in one embodiment, the processing module 22 includes: a preprocessing unit 221 configured to preprocess the sensor data; a conversion unit 222 configured to convert the preprocessed sensor data into the body coordinate system; The optimization unit 223 is configured to optimize the sensor data after the coordinate system transformation, and obtain processed top-view environment data and processed head-up environment data according to the sensor data.

In one embodiment, the preprocessing unit 221 is configured to: align the head-up environment data and the top-view environment data based on the time stamp; extract the point cloud information of the surface edge in the aligned top-view environment data.

In one embodiment, the optimization unit 223 is configured to: process the sensor data after the coordinate system transformation by brute force matching.

In one embodiment, the positioning module 23 is configured to locate the electronic device according to the processed sensor data, the top-view grid map and the head-up grid map in the following manner: according to the processed head-up data set The head-up grid map is subjected to closed-loop detection to obtain a head-up matching rate; according to the processed top-view grid map, a closed-loop detection is performed on the top-view grid map to obtain a top-view matching rate; when the head-up matching rate is greater than a first set threshold And/or when the top-view matching rate is greater than a second set threshold, determine the global pose of the electronic device.

In one embodiment, the positioning module 23 is configured to generate a head-up grid map and a top-view grid map according to the processed sensor data in the following manner: generate a head-up grid map based on the processed head-up environment data; Generate top-view raster maps based on environment data.

Optionally, the sensor data includes head-up environment data collected by at least one head-up sensor and top-view environment data collected by at least two top-view sensors. The viewing grid map is used to locate the electronic device according to the processed sensor data, the top-viewing grid map and the head-up viewing grid map.

In one embodiment, the processing module 22 includes: a preprocessing unit 221, configured to preprocess the sensor data; a conversion unit 222, configured to convert the preprocessed sensor data into the body coordinate system; an optimization unit 223, It is set to optimize the sensor data after the coordinate system transformation, and obtain the processed top-view environment data and the processed head-up environment data according to the sensor data.

In one embodiment, the preprocessing unit 221 is configured to: align the head-up environment data and the top-view environment data based on the time stamp; splice the aligned top-view environment data; extract the spliced top-view environment data Point cloud information of the edge of the midplane.

In one embodiment, the positioning module 24 is configured to locate the electronic device according to the processed sensor data, the top-view grid map and the head-up grid map in the following manner: according to the processed head-up data set Perform closed-loop detection on the head-up grid map to obtain a head-up matching rate; perform closed-loop detection on the top-view grid map according to the processed top-view data set to obtain a top-view matching rate; when the head-up matching rate is greater than the first setting When the threshold and/or the top-view matching rate is greater than a second set threshold, the global pose of the electronic device is determined.

In one embodiment, the positioning module is configured to generate a head-up grid map and a top-view grid map according to the processed sensor data in the following manner: generate a head-up grid map based on the processed head-up environment data; generate a head-up grid map based on the processed top-view Environment data generate a top-view raster map.

The above-mentioned positioning device can execute the positioning method provided by any embodiment of the present application, and has corresponding functional modules for executing the method.

Fig. 32 is a schematic structural diagram of another positioning device provided by this magical embodiment, which can execute the positioning method provided by any embodiment of this application, and has corresponding functional modules for executing the method. The device can be implemented by software and/or hardware, including: a data acquisition module 31 , a feature point cloud determination module 32 , an obstacle score determination module 33 and a location determination module 34 .

The data acquisition module 31 is set to: collect the current pose of the robot and the depth data of at least one preset direction; the feature point cloud determination module 32 is set to: determine the feature point cloud based on the depth data; the obstacle score determination module 33 is set to: Determine the obstacle score of the feature point cloud in the preset global grid map, wherein the preset global grid map is formed based on the historical point cloud; the positioning determination module 34 is configured to optimize the current Pose to determine the positioning information of the robot.

Optionally, the data collection module 31 is configured to: collect the current pose of the robot and depth data and infrared data in at least one preset direction; the feature point cloud module 32 is configured to extract at least one edge point from the infrared data , and converting the at least one edge point into a feature point cloud according to the depth data.

In the embodiment of the present application, the current pose of the robot and the depth data and infrared data in the preset direction are obtained through the data acquisition module, and the feature point cloud determination module extracts at least one edge point from the infrared data, and processes at least one edge point according to the depth data To form a feature point cloud, the obstacle score determination module determines the obstacle score of the feature point cloud in the preset global grid map, and the positioning determination module uses the obstacle score to optimize the current pose to obtain the positioning information of the robot, realizing the accuracy of the robot position information Acquisition, reducing the impact of complex environments on pose determination, can enhance the quality of robot services and help improve user experience.

Optionally, on the basis of the above embodiments, the location determining module 34 includes: a comprehensive optimization unit configured to optimize the location information according to the moving speed and obstacle score of the robot.

Optionally, on the basis of the above embodiments, the data acquisition module 31 includes: a pose acquisition unit configured to acquire the current pose in the world coordinate system of the robot, wherein the current pose includes at least the abscissa, The ordinate and the angle between the direction of the robot and the X-axis of the world coordinate system; the data acquisition unit is configured to use at least one depth data sensor pre-set on the robot to collect depth data in the preset direction and at least one infrared data The sensor collects infrared data in the preset direction, wherein the preset direction includes at least one of a horizontal direction and a vertical direction.

Optionally, on the basis of the above-mentioned application embodiments, the feature point cloud determination module 32 includes: a noise processing unit configured to filter out noise in the infrared data; an edge extraction unit configured to filter out noise At least one edge point is extracted from the infrared data; a point cloud generation unit is configured to use the camera model of the robot and the depth data to convert the at least one edge point into three-dimensional coordinates to form a feature point cloud.

Optionally, on the basis of the above-mentioned embodiments, the obstacle score determination module 33 includes: a position mapping unit, configured to transform the coordinates of at least one edge point in the feature point cloud into the world coordinate system, and after mapping the transformed coordinates Each edge point of each edge point is to the target grid of the preset global grid map; the score determination unit is set to the grid where the target grid mapped to a position point is the grid of the obstacle area in the preset global grid map In this case, the probability value of the target grid is obtained as the obstacle score of the one edge point; the score statistics unit is configured to count the sum of the obstacle scores of the at least one edge point in the feature point cloud as The obstacle score of the feature point cloud.

Optionally, on the basis of the above-mentioned embodiments, the positioning determination module 34 further includes: a first residual unit configured to construct a residual function according to the current pose and the obstacle score of the feature point cloud; the parameter The adjustment unit is set to adjust the parameter information in the residual function so that the result value of the residual function is the smallest, and the parameter information in the residual function is the abscissa and ordinate of the current pose and the robot direction and the X axis of the world coordinate system The value of at least one parameter in the included angle; The location determination unit is set to the abscissa, ordinate, and the included angle location of the X-axis of the robot direction and the world coordinate system when the result value is minimized. Posture information.

Optionally, on the basis of the above embodiment, the comprehensive optimization unit is configured to: construct a first residual item according to the current pose and the obstacle score of the feature point cloud; determine a prediction based on the moving speed pose, and the difference between the predicted pose and the previous moment’s historical pose as the second residual item; adjust the parameter information in the speed prediction pose in the first residual item and the second residual item And/or the parameter information in the current pose makes the sum of the first residual term and the second residual term the smallest value, the parameter information includes the abscissa, ordinate, robot direction and world coordinate system The value of at least one parameter in the included angle of the X axis; the speed prediction pose and/or the current pose when the sum of the first residual term and the second residual term is minimized The abscissa, ordinate, and the angle between the robot direction and the X-axis of the world coordinate system are used as the optimized positioning pose information.

Optionally, on the basis of the above embodiments, the device further includes: a map update module configured to update the robot pose corresponding to the positioning information to the preset global grid map. Optionally, on the basis of the above-mentioned embodiments, the data collection module 31 is set to collect the current pose of the robot and the depth data of at least one preset direction; the feature point cloud determination module 32 is set to extract the depth data The outer contour points of at least one plane and the outer contour points of at least one plane form a feature point cloud.

In the embodiment of the present application, the current pose of the robot and the depth data in at least one preset direction are obtained through the data acquisition module, and the feature point cloud determination module extracts at least one plane's outer contour points from the depth data, and uses the outer contour points to form features Point cloud, the obstacle score determination module determines the obstacle score of the feature point cloud in the preset global grid map, and the positioning determination module uses the obstacle score to optimize the current pose to obtain the positioning information of the robot, realizing the accurate acquisition of the robot position information, reducing The influence of complex environment on pose determination can enhance robot service quality and help improve user experience.

Optionally, on the basis of the above embodiments of the invention, the location determination module 44 includes: a comprehensive optimization unit configured to optimize the location information according to the moving speed and obstacle score of the robot.

Optionally, on the basis of the above embodiments, the data acquisition module 31 includes: a pose acquisition unit configured to acquire the current pose in the world coordinate system of the robot, wherein the current pose includes at least the abscissa, The ordinate and the angle between the direction of the robot and the X-axis of the world coordinate system; the depth acquisition unit is configured to use at least one depth data sensor preset on the robot to collect depth data in the at least one preset direction, wherein, The at least one preset direction includes at least one of a horizontal direction and a vertical direction; a depth processing unit configured to filter out noise in the depth data.

Optionally, on the basis of the above-mentioned embodiments, the feature point cloud determination module 32 includes: a point cloud generation unit configured to convert the depth data into a three-dimensional point cloud based on the camera model of the robot, and the depth The data includes at least position point information and depth information; the plane division unit is configured to divide the three-dimensional point cloud into at least one plane according to the depth information and preset normal vector information; the feature extraction unit is configured to extract the Outline points are used as feature point cloud.

Optionally, on the basis of the above embodiments, the obstacle scoring module 403 includes: a position mapping unit configured to convert the coordinates of at least one position point in the feature point cloud into a world coordinate system, and map the transformed coordinates to Each location point is to the target grid of the preset global grid map; the score determination unit is configured to map a location point to the target grid as the grid of the obstacle area in the preset global grid map In the case of , the probability value of the target grid is obtained as the obstacle score of the one location point; the score statistics unit is set to count the sum of the obstacle scores of all the location points in the feature point cloud as the Obstacle scores for feature point clouds.

Optionally, on the basis of the above-mentioned embodiments, the positioning determination module 34 further includes: a first residual unit configured to construct a residual function according to the current pose and the obstacle score of the feature point cloud; the parameter The adjustment unit is configured to adjust the parameter information in the residual function so that the result value of the residual function is the smallest, and the parameter information in the residual function is the abscissa and ordinate of the current pose and the X of the robot direction and the world coordinate system The value of at least one parameter in the included angle of the axis; the positioning determination unit is set to the abscissa, ordinate and the included angle of the robot direction and the X-axis of the world coordinate system when the result value is the smallest. The output positioning pose information.

Optionally, on the basis of the above embodiment, the comprehensive optimization unit is configured to: construct a first residual item according to the current pose and the obstacle score of the feature point cloud; determine a prediction based on the moving speed pose, and the difference between the predicted pose and the previous moment’s historical pose as the second residual item; adjust the parameter information and the parameter information in the predicted pose in the first residual item and the second residual item /or the parameter information in the current pose makes the sum of the first residual term and the second residual term the smallest value, the parameter information includes the abscissa, ordinate, and the robot direction and the world coordinate system The value of at least one parameter in the included angle of the X axis; the velocity prediction pose and/or the transverse direction of the current pose when the sum of the first residual term and the second residual term is minimized Coordinates, ordinates, and the angle between the robot direction and the X-axis of the world coordinate system are used as positioning pose information.

Optionally, on the basis of the above embodiments, the device further includes: a map update module configured to update the robot pose corresponding to the positioning information to the preset global grid map.

FIG. 33 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As shown in FIG. 34 , the electronic equipment provided by the embodiment of the present application includes: one or more processors 41 and storage devices 42; there may be one or more processors 41 in the electronic equipment, and one processor is used in FIG. 34 41 as an example; the storage device 42 is set to store one or more programs; the one or more programs are executed by the one or more processors 41, so that the one or more processors 41 realize the implementation of the present application The positioning method described in any one of the examples.

The electronic device may further include: an input device 43 and an output device 44 .

In one embodiment, the electronic device further includes a sensor, the sensor includes at least two top-view sensors and a wheel encoder, the field of view between the at least two top-view sensors has a common viewing area, and the at least two top-view sensors A top-view sensor is configured to collect top-view environmental data above the electronic device; and the wheel encoder is configured to determine the speed and distance at which the electronic device travels.

The speed and distance collected by the wheel encoder can be used in positioning. The top-view environmental data collected by the top-view sensor can be used for mapping and positioning.

The position of the top-view sensor on the electronic device is not limited here, as long as the top-view sensor can collect top-view environmental data, and the field of view between multiple top-view sensors has a common viewing area.

The technical means that there is a common-view area in the field of view between a plurality of top-view sensors is not limited, it can be considered that there is a common-view area between two adjacent top-view sensors in the at least two top-view sensors; or the at least two There is a target top-view sensor in the top-view sensor, and the top-view sensors except the target top-view sensor among the at least two top-view sensors have a common view area with the field angle of the target top-view sensor.

The common viewing area of the field angles of the at least two top-view sensors can be used for external parameter calibration of the top-view sensors, and then the top-view environment data collected by the at least two top-view sensors can be spliced.

In one embodiment, the electronic device further includes: a top-view sensor and a head-up sensor; the head-up sensor is configured to collect the head-up environment data of the running direction of the electronic device; the top-view sensor is configured to collect the above-mentioned electronic device top-view environment data.

In one embodiment, the number of the top-view sensors is at least two, and the angle between the at least two top-view sensors and the horizontal direction is different, and there is a set ratio of the field of view between two adjacent top-view sensors. common view area.

The top-view sensor in this embodiment can collect top-view environmental data in the same area to improve collection accuracy. In this embodiment, multiple top-view sensors can collect top-view environmental data in different areas above the electronic device, so as to improve positioning accuracy by expanding the collection range.

In this application, multiple top-view sensors may not have a common-view area to collect more top-view environment data. In this embodiment, there may be a common-view area with a set ratio in the field of view between adjacent top-view sensors, so that the top-view environment data can be spliced effectively. The setting ratio is not limited, such as one-third or one-fourth.

In one embodiment, the number of the top-view sensor is one.

The location of the top-view sensor in this embodiment is not limited, as long as it can collect top-view environment data. In this embodiment, the number of top-view sensors is one, which reduces the cost of electronic equipment.

Optionally, the electronic device further includes a depth data sensor and an infrared data sensor, the depth data sensor is configured to collect depth data in a preset direction, and the infrared data sensor is configured to collect infrared data in the preset direction.

Optionally, the electronic device further includes a depth data sensor, and the depth data sensor is configured to collect depth data in a preset direction.

The processor 41 , storage device 42 , input device 43 and output device 344 in the electronic device may be connected via a bus or in other ways. In FIG. 3 , connection via a bus is taken as an example.

The storage device 42 in the electronic device, as a computer-readable storage medium, can be used to store one or more programs, and the programs can be software programs, computer-executable programs and modules, such as the positioning method provided in the embodiment of the present application Corresponding program instructions/modules (for example, modules in the positioning device shown in FIG. 30 include: acquisition module 21, processing module 22, generation module 23 and positioning module 24 or data acquisition module 31 and feature point cloud in FIG. 31 determination module 32, obstacle score determination module 33 and positioning determination module 34). The processor 41 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the storage device 42 , that is, implements the positioning method in the above method embodiments.

The storage device 42 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the electronic device, and the like. In addition, the storage device 42 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices. In some examples, storage device 42 may include memory located remotely from processor 31, and such remote memory may be connected to the device via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input device 33 can be configured to receive input numbers or character information, and generate key signal input related to user settings and function control of the electronic device. The output device 34 may include a display device such as a display screen.

In an exemplary embodiment, the electronic device may specifically be a robot, or a positioning and navigation device installed on the robot. A robot or a positioning and navigation device can accurately determine the position of the robot through the positioning method provided by the embodiment of the present invention.

When one or more programs included in the above-mentioned electronic device are executed by the one or more processors 31, for example, the programs perform the following operations:

acquiring sensor data collected by the at least one top-looking sensor; processing the sensor data; locating the electronic device based on the processed sensor data; or

Collect the current pose of the robot and depth data in at least one preset direction; determine the feature point cloud based on the depth data; determine the obstacle score of the feature point cloud in the preset global grid map, wherein the preset The global grid map is constructed based on the historical point cloud; the current pose is optimized according to the obstacle score to determine the positioning information of the robot.

The embodiment of the present application provides a computer-readable storage medium. FIG. 34 is a schematic structural diagram of a storage medium provided in the embodiment of the present application. A computer program 53 is stored on the computer-readable storage medium 51, and the program is executed by a processor 52. Execution is used to execute the positioning method, which includes:

Optionally, when the computer program 53 is executed by the processor 52, it may also be used to execute the positioning method provided by any embodiment of the present application.

The computer-readable storage medium 51 in the embodiment of the present application may use any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium 51 . The computer-readable storage medium 51 may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media 51 include: electrical connections with one or more wires, portable computer disks, hard disks, Random Access Memory (RAM), read-only Memory (Read Only Memory, ROM), Erasable Programmable Read Only Memory (Erasable Programmable Read Only Memory, EPROM), flash memory, optical fiber, portable CD-ROM (Compact Disc-Read-Only Memory, CD-ROM), An optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium 51 may be any tangible medium that contains or stores a program for use by or in conjunction with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a data signal carrying computer readable program code in baseband or as part of a carrier wave. Such propagated data signals may take many forms, including but not limited to: electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium 51, which can send, propagate, or transmit information for use by or in conjunction with an instruction execution system, apparatus, or device. program.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wires, optical cables, radio frequency (Radio Frequency, RF), etc., or any suitable combination of the above.

Computer program code for carrying out the operations of the present invention may be written in one or more programming languages, or combinations thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional A procedural programming language, such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or it may be connected to an external computer such as use an Internet service provider to connect via the Internet).

Claims

A positioning method applied to an electronic device, the electronic device comprising at least one top-view sensor, the method comprising:

acquiring sensor data collected by the at least one top-looking sensor;

processing said sensor data;

The electronic device is located based on the processed sensor data.
The positioning method according to claim 1, wherein the electronic device comprises at least two top-view sensors;

The positioning of the electronic device according to the processed sensor data includes: matching the processed sensor data with the local map data in the global map to determine where the electronic device is located. Describe the pose in the global map.
The method of claim 2, wherein the sensor data comprises top-view environment data;

Processing the sensor data includes: aligning the top-view environment data of the at least two top-view sensors based on time stamps; and preprocessing the aligned top-view environment data.
The method according to claim 3, wherein said preprocessing the aligned top-view environment data comprises:

Remove the noise in the aligned top-view environment data;

Splicing the top-view environment data after denoising;

Extract the point cloud information of the surface edge in the stitched top-view environment data.
The method according to claim 4, wherein said splicing the noise-removed top-view environment data comprises:

The top-view environment data after denoising is transformed into the coordinate system where the target top-view sensor is located, and the target top-view sensor is one top-view sensor among the at least two top-view sensors.
The method according to claim 2, wherein matching the processed sensor data with local map data in the global map to determine the pose of the electronic device in the global map comprises:

Convert the point cloud information of the surface edge included in the processed sensor data into raster data;

determining local map data matching the raster data in the global map;

Determine the pose of the electronic device in the global map according to the local map data and the grid data.
The method according to claim 6, wherein the determining the pose of the electronic device in the global map according to the local map data and the grid data comprises:

Determine the pose of the electronic device in the global map according to the pose relationship between the local map data and the global map, and the pose relationship between the grid data and the local map data .
The method according to claim 1, further comprising:

In the case of obtaining the mapping instruction, adding the point cloud information of the surface edge included in the processed sensor data to the local map data matched with the processed sensor data;

The local map data after adding the point cloud information of the surface edge is updated to the global map.
The positioning method according to claim 1, wherein said electronic device includes a top-view sensor and at least one head-up sensor; said sensor data includes head-up environment data collected by said at least one head-up sensor and said one top-view sensor Collected top-view environment data;

Positioning the electronic device according to the processed sensor data includes: generating a head-up grid map and a top-view grid map according to the processed sensor data; The head-up grid map locates the electronic device.
The method of claim 9, wherein said processing said sensor data comprises:

preprocessing said sensor data;

Transform the preprocessed sensor data into the body coordinate system;

Optimize the sensor data after transforming the coordinate system;

According to the optimized sensor data, the processed top-view environment data and the processed head-up environment data are obtained.
The method of claim 10, wherein said preprocessing said sensor data comprises:

aligning the head-up environment data and the top-view environment data based on a timestamp;

Extract the point cloud information of the face edges in the aligned top-view environment data.
The method according to claim 10, wherein said optimizing the sensor data after transforming the coordinate system comprises:

The sensor data after transforming the coordinate system is processed by brute force matching.
The method according to claim 9, wherein the positioning of the electronic device according to the processed sensor data, the top-view grid map and the head-up grid map comprises:

Carrying out closed-loop detection to the head-up grid map according to the processed head-up environment data to obtain the head-up matching rate;

According to the top-view environment data after processing, described top-view grid map is carried out closed-loop detection and obtains top-view matching rate;

When at least one of the head-up matching rate greater than a first set threshold and the top-view matching rate greater than a second set threshold is satisfied, determine the global pose of the electronic device.
The method according to claim 9, wherein generating a head-up grid map and a top-view grid map according to the processed sensor data comprises:

Generate a head-up grid map based on the processed head-up environment data;

Generate a top-view raster map based on the processed top-view environment data.
The positioning method according to claim 1, wherein the electronic device includes at least two top-view sensors and at least one head-up sensor; the sensor data includes the head-up environment data collected by the at least one head-up sensor and the at least two head-up sensors. Top-view environmental data collected by a top-view sensor;

The positioning of the electronic device according to the processed sensor data includes: generating a head-up grid map and a top-view grid map according to the processed sensor data; according to the processed sensor data, the top-view grid map and The head-up grid map locates the electronic device.
The method of claim 15, wherein said processing said sensor data comprises:

preprocessing said sensor data;

Transform the preprocessed sensor data into the body coordinate system;

Optimize the sensor data after transforming the coordinate system;

The processed top-view environment data and the processed head-up environment data are obtained according to the optimized sensor data.
The method of claim 16, wherein said preprocessing said sensor data comprises:

aligning the head-up environment data and the top-view environment data based on a timestamp;

Stitch the aligned top-view environment data;

Extracting point cloud information of surface edges in the spliced top-view environment data.
The method according to claim 16, wherein said optimizing the sensor data after transforming the coordinate system comprises:

The sensor data after transforming the coordinate system is processed by brute force matching.
The method according to claim 15, wherein said positioning the electronic device according to the processed sensor data, the top-view grid map and the head-up grid map comprises:

Carrying out closed-loop detection to the head-up grid map according to the processed head-up environment data to obtain the head-up matching rate;

Perform closed-loop detection on the top-view grid map according to the processed top-view environment data to obtain a top-view matching rate;

When at least one of the head-up matching rate greater than a first set threshold and the top-view matching rate greater than a second set threshold is satisfied, determine the global pose of the electronic device.
The method according to claim 15, wherein generating a head-up grid map and a top-view grid map according to the processed sensor data comprises:

Generate a head-up grid map based on the processed head-up environment data;

Generate a top-view raster map based on the processed top-view environment data.
A positioning method, comprising:

Collect the current pose of the robot and depth data in at least one preset direction;

determining a feature point cloud based on the depth data;

determining the obstacle score of the feature point cloud in a preset global grid map, wherein the preset global grid map is formed based on historical point clouds;

Optimizing the current pose according to the obstacle score to determine positioning information of the robot.
The positioning method according to claim 21, wherein collecting the current pose of the robot and depth data in at least one preset direction comprises: collecting the current pose of the robot, depth data in at least one preset direction, and the at least one preset direction Infrared data for setting direction;

The determining the feature point cloud based on the depth data includes: extracting at least one edge point from the infrared data, and converting the at least one edge point into a feature point cloud according to the depth data.
The method according to claim 22, wherein said optimizing said current pose according to said obstacle score to determine positioning information of said robot comprises:

Optimizing the current pose according to the moving speed of the robot and the obstacle score to determine positioning information of the robot.
The method according to claim 22, wherein the collecting the current pose of the robot, depth data in at least one preset direction, and infrared data in at least one preset direction comprises:

Acquiring the current pose of the robot in the world coordinate system, wherein the current pose at least includes an abscissa, a ordinate, and an angle between the direction of the robot and the X-axis of the world coordinate system;

Using at least one depth data sensor preset on the robot to collect depth data in the at least one preset direction and using at least one infrared data sensor preset on the robot to collect infrared data in the at least one preset direction data, wherein the at least one preset direction includes at least one of a horizontal direction and a vertical direction.
The method according to claim 22, wherein said extracting at least one edge point from said infrared data, and converting said at least one edge point into a feature point cloud according to said depth data comprises:

filtering out noise in the infrared data;

extracting at least one edge point in the noise-filtered infrared data;

converting the two-dimensional coordinates of the at least one edge point into three-dimensional coordinates by using the camera model of the robot and the depth data to form a feature point cloud.
The method according to claim 22, wherein said determining the obstacle score of said feature point cloud in a preset global grid map comprises:

transforming the coordinates of the at least one edge point in the feature point cloud into a world coordinate system, and mapping each edge point after the transformed coordinates to the target grid of the preset global grid map;

In the case that the target grid to which each edge point is mapped is a grid in the obstacle area in the preset global grid map, obtain the probability value of the target grid to which each edge point is mapped as The obstacle score of each edge point;

The sum of the obstacle scores of the at least one edge point in the feature point cloud is counted as the obstacle score of the feature point cloud.
The method according to claim 26, wherein the positioning information includes positioning pose information, and the positioning pose information is pose information used for robot positioning;

The optimizing the current pose according to the obstacle score to determine the positioning information of the robot includes:

Constructing a residual function according to the current pose and the obstacle score of the feature point cloud;

Adjust the parameter information in the residual function so that the result value of the residual function is the smallest, and the parameter information in the residual function is the abscissa, ordinate and the angle between the robot direction and the X-axis of the world coordinate system The value of at least one parameter;

The abscissa and ordinate of the current pose when the result value is the smallest, and the angle between the robot direction and the X-axis of the world coordinate system are used as positioning pose information.
The method according to claim 23, wherein the positioning information includes positioning pose information, and the positioning pose information is pose information used for robot positioning;

The optimizing the current pose according to the moving speed of the robot and the obstacle score to determine the positioning information of the robot includes:

Constructing a first residual term according to the current pose and the obstacle score;

Determine the predicted pose based on the moving speed, and use the difference between the predicted pose and the historical pose at the previous moment as a second residual term;

Adjusting at least one of the parameter information in the predicted pose and the parameter information in the current pose in the first residual item and the second residual item so that the first residual item and the second residual item The sum of the residual items is the smallest value, and the parameter information includes the value of at least one parameter in the angle between the abscissa, the ordinate and the robot direction and the X-axis of the world coordinate system;

The abscissa and ordinate of at least one of the predicted pose and the current pose when the sum of the first residual term and the second residual term is minimized and the world coordinate system The included angle of the X axis is used as positioning pose information.
The positioning method according to claim 21, wherein,

Determining the feature point cloud based on the depth data includes: extracting the outer contour points of at least one plane from the depth data and forming the feature point cloud from the outer contour points of the at least one plane.
The method according to claim 29, wherein said optimizing said current pose according to said obstacle score to determine positioning information of said robot comprises:

Optimizing the current pose according to the moving speed of the robot and the obstacle score to determine positioning information of the robot.
The method according to claim 29, wherein the collecting the current pose of the robot and the depth data of at least one preset direction comprises:

Acquiring the current pose of the robot in the world coordinate system, wherein the current pose at least includes an abscissa, a ordinate, and an angle between the direction of the robot and the X-axis of the world coordinate system;

collecting depth data in the at least one preset direction using at least one depth data sensor preset on the robot, wherein the at least one preset direction includes at least one of a horizontal direction and a vertical direction;

Noise in the depth data in the at least one preset direction is filtered out.
The method according to claim 29, wherein said extracting the outer contour points of at least one plane from the depth data and forming the feature point cloud from the outer contour points of the at least one plane comprises:

converting the depth data into a three-dimensional point cloud based on the camera model of the robot, the depth data at least including position point information and depth information;

segmenting the 3D point cloud into at least one plane according to the depth information and preset normal vector information;

Extracting the outer contour points of the at least one plane as a feature point cloud.
The method according to claim 29, wherein said determining the obstacle score of said feature point cloud in a preset global grid map comprises:

Converting the coordinates of multiple position points in the feature point cloud to the world coordinate system, and mapping each position point after the transformed coordinates to the target grid of the preset global grid map;

In the case that the target grid to which each position point is mapped is a grid in the obstacle area in the preset global grid map, obtain the probability value of the target grid to which each position point is mapped as The handicap score for each location point;

The sum of the obstacle scores of the plurality of position points in the feature point cloud is counted as the obstacle score of the feature point cloud.
The method according to claim 33, wherein the positioning information includes positioning pose information, and the positioning pose information is pose information used for robot positioning;

The optimizing the current pose according to the obstacle score to determine the positioning information of the robot includes:

Constructing a residual function according to the current pose and the obstacle score of the feature point cloud;

Adjust the parameter information in the residual function so that the result value of the residual function is the smallest, and the parameter information in the residual function is at least the value of a parameter;

When the result value is minimum, the abscissa and ordinate of the current pose, and the angle between the robot direction and the X-axis of the world coordinate system are used as positioning pose information.
The method according to claim 30, wherein the positioning information includes positioning pose information, and the positioning pose information is pose information used for robot positioning;

The optimizing the current pose according to the moving speed of the robot and the obstacle score to determine the positioning information of the robot includes:

Constructing a first residual term according to the current pose and the obstacle score;

Determine the predicted pose based on the moving speed, and use the difference between the predicted pose and the historical pose at the previous moment as a second residual term;

Adjusting at least one of the parameter information in the predicted pose and the parameter information in the current pose in the first residual item and the second residual item so that the first residual item and the second residual item The sum of the residual items is the smallest value, and the parameter information includes the value of at least one parameter in the angle between the abscissa, the ordinate and the robot direction and the X-axis of the world coordinate system;

The abscissa and ordinate of at least one of the predicted pose and the current pose when the sum of the first residual term and the second residual term is minimized and the world coordinate system The included angle of the X axis is used as positioning pose information.
The method according to any one of claims 22-35, further comprising:

The robot pose corresponding to the positioning information is updated to the preset global grid map.
A positioning device configured on electronic equipment, the device comprising:

An acquisition module configured to acquire sensor data collected by at least one top-view sensor;

a processing module configured to process the sensor data;

A positioning module configured to locate the electronic device according to the processed sensor data.
A positioning device, comprising:

The data collection module is configured to collect the current pose of the robot and depth data in at least one preset direction;

A feature point cloud determination module, configured to determine a feature point cloud based on the depth data;

The obstacle score determination module is configured to determine the obstacle score of the feature point cloud in the preset global grid map, wherein the preset global grid map is formed based on historical point clouds;

A positioning determining module configured to optimize the current pose according to the obstacle score to determine positioning information of the robot.
An electronic device comprising:

at least one processor;

a storage device configured to store one or more programs;

When the at least one program is executed by the one or more processors, the at least one processor implements the positioning method according to any one of claims 1-36.
The electronic device according to claim 39, further comprising: at least two top-view sensors: there is a common viewing area in the field of view between the at least two top-view sensors, and the at least two top-view sensors are configured to collect top-view sensors. depending on the environment data.
The electronic device of claim 39, further comprising: a top-view sensor and at least one head-up sensor;

The at least one head-up sensor is configured to collect head-up environment data;

The one top-view sensor is configured to collect top-view environment data.
The electronic device of claim 39, further comprising: at least two top-view sensors and at least one head-up sensor;

The at least one head-up sensor head-up sensor is configured to collect head-up environment data;

The at least two top-view sensors are configured to collect top-view environmental data.
The electronic device according to claim 42, wherein the angles between the at least two top-view sensors and the horizontal direction are different, and there is a common-view area of a set ratio in the field of view between two adjacent top-view sensors.
A computer-readable storage medium storing a computer program, which implements the positioning method according to any one of claims 1-36 when the computer program is executed by a processor.