CN114474061B

CN114474061B - Cloud service-based multi-sensor fusion positioning navigation system and method for robot

Info

Publication number: CN114474061B
Application number: CN202210148066.XA
Authority: CN
Inventors: 何丽; 刘钰嵩; 齐继超; 刘志强; 李顺; 李可新; 陈耀华; 王宏伟; 郑威强
Original assignee: Xinjiang University
Current assignee: Xinjiang University
Priority date: 2022-02-17
Filing date: 2022-02-17
Publication date: 2023-08-04
Anticipated expiration: 2042-02-17
Also published as: CN114474061A

Abstract

The invention discloses a cloud service-based multi-sensor fusion positioning navigation system and a cloud service-based multi-sensor fusion positioning navigation method for a robot, which comprise a robot and a navigation cloud service platform, wherein the robot comprises an indoor mobile robot chassis, a vehicle-mounted computer, a three-dimensional laser radar, a depth camera and a liquid crystal display, the indoor mobile robot chassis comprises an omnidirectional wheel, a shell, a shock absorber, a profile frame, a mounting plate, a driving motor, a driving control board and a vehicle-mounted battery, the navigation cloud service platform comprises a robot end, a cloud server, a client and network equipment, the robot autonomously completes real-time sensing and judging of the environmental characteristics of the robot on the basis of sensing and detecting the external environment to complete map construction, the current position is calibrated, a local map is updated, and the optimal path planning conforming to multiple evaluation indexes such as social constraint and task constraint is completed according to the environmental information, so that a robot executing mechanism is controlled to reach a target position.

Description

Cloud service-based multi-sensor fusion positioning navigation system and method for robot

Technical Field

The invention relates to the technical field of indoor service robots, in particular to a cloud service-based multi-sensor fusion positioning navigation system and method for a robot.

Background

In recent years, with the continuous development and application of artificial intelligence and computer technology, a great deal of advanced technology is promoted in the field of robot technology, and a great deal of algorithms are applied to the existing robot positioning navigation system. The robot positioning navigation system is a core module of the robot system, and is used for bearing the functions of sensing and path planning of the robot in a complex unknown environment, is a key technology for realizing autonomous operation of the robot, and is a basic technology for realizing higher-level functions. Most of the existing positioning navigation systems are deployed in portable computing devices of mobile robots to meet the operation requirements of the robots in the running process.

However, due to the limitation of computational power of the small computing devices of the current robot platform, the deployed positioning navigation system is often not capable of applying advanced positioning navigation algorithms, such as positioning navigation algorithms combined with deep learning and image processing algorithms, and is relatively time-consuming in deployment and operation, which is unfavorable for research of the algorithms by scientific researchers.

Disclosure of Invention

The invention aims to provide a cloud service-based multi-sensor fusion positioning navigation system and method for a robot, which are used for solving the problems in the background technology.

In order to achieve the above purpose, the present invention provides the following technical solutions:

the utility model provides a robot multisensor fuses location navigation based on cloud service, includes robot and navigation cloud service platform, the robot includes indoor mobile robot chassis, on-vehicle computer, three-dimensional laser radar, depth camera and LCD, indoor mobile robot chassis includes omnidirectional wheel, casing, bumper shock absorber, section bar frame, mounting panel, driving motor, drive control panel and on-vehicle battery, navigation cloud service platform includes robot end, cloud server, customer end and network equipment.

As a further scheme of the invention: the indoor mobile robot chassis is used for carrying sensors and driving equipment to realize stable operation of the robot, the vehicle-mounted computer is used for reading packed sensor data and sending the packed sensor data to the navigation cloud service platform through a wireless network, the depth camera is used for extracting color information and depth information in the environment, the three-dimensional laser radar is used for carrying out scanning matching on the environment information, the liquid crystal display is used for displaying real-time operation information of the robot, the multi-sensor fusion is that is the fusion of the depth camera and the three-dimensional laser radar on a data layer, the vehicle-mounted computer is used for packing collected sensor data and establishing communication with an upper cloud end server through an ROS node, transmitting the data to the cloud end server, receiving response information of the cloud end, and further controlling the behavior of the vehicle according to the response information of the cloud end.

As still further aspects of the invention: the robot end comprises a robot end ROS system and a wireless network card module, wherein the robot end ROS system is used for collecting image information and radar point cloud information generated in the operation of a sensor and packaging the information into ROS information for broadcasting, the wireless network card module is used for transmitting the information to a cloud server for processing after collecting the information through a wireless network card, the cloud server comprises a server end ROS system, a processor, a memory, a hard disk, a display card and an operating system, and the network equipment comprises a router and a switch.

As still further aspects of the invention: the cloud server is used for carrying a robot positioning navigation algorithm and processing the data in real time, the client is used for checking and controlling the running condition of the robot in real time, and the network equipment is used for providing the cloud server to access network services.

As still further aspects of the invention: the multi-sensor fusion positioning mapping method is used for sensing and positioning the robot in an unknown environment and constructing a global static map, accurately positioning the robot in the environment and the map and constructing a semantic grid map for navigation, the robot interacts with the environment in the map, the robot navigation method is used for robot path planning and map navigation, the robot can autonomously move from the current position to a certain target position in the environment after the autonomous positioning and planning control work are coordinated, the multi-sensor fusion positioning mapping method comprises a visual positioning mapping algorithm, a laser positioning mapping algorithm, a pose fusion algorithm, a back-end optimization algorithm and a map fusion algorithm, the visual positioning map building algorithm is used for robot positioning based on depth camera information, static semantic map building of the environment is achieved through a semantic segmentation algorithm and an optical flow method, the laser positioning map building algorithm is used for robot positioning based on three-dimensional laser radar information to build a 2D grid map, the pose fusion algorithm is used for fusing the pose of the visual positioning map building algorithm and the pose of the laser positioning map building algorithm, the rear-end optimization algorithm is used for optimizing the pose of the robot, the pose of a road mark point and optimizing and updating a map in real time in the robot navigation process, and the map fusion algorithm is used for fusing a semantic octree map and a grid map to build a static 2.5D map and updating the map according to real-time environment information when the robot is in operation.

As still further aspects of the invention: the map fusion algorithm comprises a local map fusion algorithm and a global map fusion algorithm.

As still further aspects of the invention: the visual positioning mapping algorithm comprises a tracking thread, a preprocessing thread and a semantic mapping thread.

A multi-sensor fusion positioning navigation method of a robot based on cloud service comprises the following steps:

step one: the robot completes repositioning and map updating on the basis of the prior map inspection, namely judging the environmental characteristics, calibrating the current position, updating the local map, completing optimal path planning conforming to multiple evaluation indexes such as social constraint, task constraint and the like according to the environmental information, and controlling the robot executing mechanism to reach the target position according to the planning result;

step two: extracting feature points of non-human part images in the images by adopting a semantic segmentation algorithm in a preprocessing thread, extracting pixels with inconsistent dynamic states in the images by adopting an optical flow algorithm, jointly removing dynamic feature points with high possibility in the images, and obtaining a transformation matrix to obtain stable camera pose by matching residual stable features;

step three: the three-dimensional laser radar adopts a particle filtering mode to realize the construction of a grid map, firstly randomly selecting particles according to prior probability, endowing weight values, carrying out state initialization, and then generating a particle set of the next generation from a current particle set according to proposal distribution and calculating the particle weight values; resampling particles, determining sampling times by weight of the particles, estimating pose through state parameters, obtaining a grid map based on laser data, semantic image information and point cloud information, generating an octree semantic map by utilizing the point cloud map, updating semantic information of the octree map by Bayesian fusion of semantic tags at different moments to form the octree semantic map, updating uncertainty of the grid map by using a Bayesian fusion algorithm according to laser radar information, fusing local grid maps until all the grid maps are fused, and thus completing the fusion of two sensor data to construct a map to obtain a 2.5D grid map containing semantic information;

Step four: after a 2.5D global semantic map is obtained, carrying out global matching on the map according to the current laser information and the visual word bag characteristics, completing a repositioning process, capturing an original key frame, and updating a laser grid map and a semantic map corresponding to the original key frame;

step five: the image construction precision is improved through fusion image construction of laser and visual sensors, then the advantages of parallel calculation and multi-dimensional search are fully exerted by using a double-layer ant colony algorithm, and finally redundant turning points are removed through smoothing treatment, paths are optimized, and global path planning and local obstacle avoidance are cooperated. The multi-mode pedestrian track prediction method is characterized in that the conventional uniform speed prediction method is improved into uniform speed and uniform acceleration motion prediction through a polynomial curve fitting algorithm to improve short-time prediction precision, a time space state diagram is established after a sufficient pedestrian position sequence is extracted to infer the interaction influence of each agent (pedestrian and robot), high-precision pedestrian position prediction is completed, and further, the pedestrian prediction positions of multiple time steps are clustered to judge whether each service object is in the same group or not, so that effective reasoning information is provided for the navigation process;

step six: in a dual-flow network with cross-modal feature fusion, RGB and depth features are extracted and fused, accurate prediction of pedestrians is achieved, based on pedestrian detection, pedestrian states including pedestrian position, moving direction and other information are obtained by combining laser radar point cloud data, and 3D pedestrian detection is achieved. And meanwhile, combining the depth image data to acquire pedestrian skeleton data, fusing RGB image information, depth image information and human skeleton information in a behavior recognition network to realize the social behavior recognition of pedestrians, and finally, combining the pedestrian state extraction and the social behavior recognition detection result to construct a social interaction space model on a 2.5D grid map.

Compared with the prior art, the invention has the beneficial effects that:

1. according to the invention, a cloud service-based robot platform is built, a multi-sensor fusion SLAM thread and a multi-sensor fusion navigation thread are processed on the cloud service platform through a network, deployment is built through remote control of a client, a 2.5D semantic grid map is built, and a positioning navigation task of the robot is completed.

2. According to the invention, data of the depth camera and the three-dimensional laser radar are transmitted into a cloud service platform through a wireless network form, the pose of the robot is fused and positioned, the transmitted image information is subjected to semantic segmentation by utilizing deep learning, an octree map with semantics is generated, global optimization is performed on the pose and the road mark point at the rear end, and a 2.5D semantic grid map for navigation is generated by combining with a two-dimensional grid map of the three-dimensional laser radar.

3. According to the invention, on the basis of the navigation module, the robot senses and detects the external environment to complete map construction, autonomously senses and judges the environmental characteristics, marks the current position, updates the local map, completes the optimal path planning meeting multiple evaluation indexes such as social constraint, task constraint and the like according to the environmental information, and controls the robot executing mechanism to reach the target position according to the planning result.

The invention is suitable for the technical field of indoor service robots and mainly comprises a robot navigation method based on cloud service, a navigation cloud service platform and a robot body. The robot body includes: vehicle-mounted computer, three-dimensional laser radar, liquid crystal display, depth camera, indoor mobile robot chassis. The three-dimensional laser radar and the depth camera are used for collecting environmental data; the vehicle-mounted computer is used for collecting various sensor data, communicating with the navigation cloud platform and generating corresponding control signals; the cloud service platform is used for receiving sensor data sent by the vehicle-mounted computer, completing map construction and navigation decision on the basis of the sensor data, and finally sending a result to the vehicle-mounted computer; the navigation method includes a multisensor fusion simultaneous localization and mapping (SLAM) method and a navigation method. By the mode, accurate navigation of the indoor mobile robot can be realized.

Drawings

Fig. 1 is a schematic diagram of a structure of a robot in a system and a method for positioning and navigation by fusion of multiple sensors of the robot based on cloud services.

Fig. 2 is a data flow diagram in a cloud service-based robotic multi-sensor fusion positioning navigation system and method.

Fig. 3 is a flowchart of a multi-sensor fusion positioning map building method in a multi-sensor fusion positioning navigation system and method of a robot based on cloud services.

Fig. 4 is a flow chart of a robot navigation method in a system and method for positioning and navigation by fusion of multiple sensors of a robot based on cloud services.

Fig. 5 is a control schematic diagram in a cloud service-based multi-sensor fusion positioning navigation system and method for a robot.

The figure shows: the vehicle-mounted computer 1, the section bar frame 2, the liquid crystal display 3, the three-dimensional laser radar 4, the three-dimensional laser radar module 5, the depth camera 6, the shock absorber 7, the omnidirectional wheel 8 and the shell 9.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Referring to fig. 1 to 5, in an embodiment of the present invention, a cloud service-based multi-sensor fusion positioning navigation system for a robot includes a robot and a navigation cloud service platform, wherein the robot includes an indoor mobile robot chassis, a vehicle-mounted computer 1, a three-dimensional laser radar 4, a depth camera 6 and a liquid crystal display 3, the indoor mobile robot chassis includes an omni-wheel 8, a housing 9, a shock absorber 7, a profile frame 2, a mounting plate, a driving motor, a driving control board and a vehicle-mounted battery, and the navigation cloud service platform includes a robot end, a cloud server, a client and a network device.

The indoor mobile robot chassis's connected mode is, casing 9 is the hexagon structure, casing 9 includes casing and lower casing, bumper shock absorber 7 is provided with three, and three bumper shock absorber 7 sets up respectively on casing 9 and is connected with the higher authority of lower casing, bumper shock absorber 7 comprises two spring construction and upper and lower connecting piece, driving motor is provided with three, and is three driving motor's front end is connected with the lower part connecting piece of bumper shock absorber 7 respectively, and is three driving motor rear end is connected with the lower casing, and is three driving motor with three omnidirectional wheel 8 one-to-one sets up, and is three omnidirectional wheel 8 is connected with the driving motor output shaft that corresponds, omnidirectional wheel 8 comprises 12 follow driving wheels, 2 piece wheel hubs and connecting piece, the drive control board is installed on the lower casing, on-vehicle battery is installed on the lower casing, section bar frame 2 is formed by 8 piece aluminium section bar connecting pieces connection and is connected with the higher authority of upper casing, the mounting panel divide into three.

The connection mode of robot is, be connected above vehicle-mounted computer 1 and the last casing, three-dimensional laser radar 4 is connected with upper mounting panel, and three-dimensional laser radar module is connected with middle level mounting panel, depth camera 6 is connected with one side of section bar frame 2, and the installation direction coincides with the positive direction of robot, liquid crystal display 3 is connected with one end of section bar frame 2, and the installation direction is 45 degrees with the vertical direction slope.

The indoor mobile robot chassis is used for carrying sensors and driving equipment to realize stable running of the robot, the vehicle-mounted computer 1 is used for reading and packaging sensor data and sending the sensor data to the navigation cloud service platform through a wireless network, the depth camera 6 is used for extracting color information and depth information in the environment, the three-dimensional laser radar 4 is used for scanning and matching the environment information, the liquid crystal display 3 is used for displaying real-time running information of the robot, multi-sensor fusion is that is fusion of the depth camera 6 and the three-dimensional laser radar 4 at a data layer, the vehicle-mounted computer 1 is used for packaging the collected sensor data, establishing communication with an upper cloud end server through an ROS node, transmitting the data to a cloud server, receiving response information of the cloud server, and further controlling the behavior of the vehicle according to the response information of the cloud server.

The invention uses the message transmission based on the ROS system for communication, and the transmission between the robot end and the cloud server end is based on the TCP/IP protocol.

The robot end comprises a robot end ROS system and a wireless network card module, wherein the robot end ROS system is used for collecting image information and radar point cloud information generated in the operation of a sensor and packaging the information into ROS information for broadcasting, the wireless network card module is used for transmitting the information to a cloud server for processing after collecting the information through a wireless network card, the cloud server comprises a server end ROS system, a processor, a memory, a hard disk, a display card and an operating system, and the network equipment comprises a router and a switch.

The cloud server is used for carrying a robot positioning navigation algorithm and processing the data in real time, the client is used for checking and controlling the running condition of the robot in real time, and the network equipment is used for providing the cloud server to access network services.

The host model of the cloud server is DELL T7920; the processor is Intel to strong gold medal 5118;12 core 24 threads, main frequency 2.3GHz; the memory is three-star DDR4 2666MHz 128GB,64GB*2; the hard disk is Intel solid state 2TB 2; the display card is Nvidia Quard RTX 5000, 16GB video memory and 3072 CUDA cores; the operating system is Ubuntu 18.04LTS, and the network card is Intel I219-LM 1000Mbps Ethernet card.

The model of the vehicle-mounted computer 1 is Intel NUC8i7HVK; the processor is Intel borui 7 8809G; the display card is AMD Radeon RX Vega M GH; the memory is Wei-just DDR4 2666MHz 16GB; the wireless network card is Intel 9560AC and supports 802.11AC wireless protocol.

The processor of the drive control board is STM32F103; ULN2003 motor drive module; and a CAN interface.

The router is TL-XDR6070, IEEE 802.11a/b/g/n/ac/ax wireless protocol, and 3 10/100/1000Mbps rate adaptive Ethernet interfaces with the highest wireless rate of 5952Mbps (2.4GHz 1148Mbps,5GHz 4804Mbps).

The switch is an Rg-S1824+ of the Rui Jie network, the transmission speed is 10/100Mbps, and the back plate bandwidth is 4.8Gbps.

The three-dimensional laser radar 4 is a three-dimensional multi-line three-dimensional laser radar 4VLP-16, and the laser lines are as follows: 16 lines, measuring range: up to 100m, measurement accuracy: 3cm.

The depth camera 6 is Kinectv1, the color resolution is 640×480, fps is 30fps, and the depth resolution is 320×240.

The method comprises a multi-sensor fusion positioning mapping method and a robot navigation method, wherein the multi-sensor fusion positioning mapping method is used for sensing and positioning of a robot in an unknown environment and constructing a global static map, accurate positions in the environment and the map and constructing a semantic grid map for navigation, the robot interacts with the environment in the map, the robot navigation method is used for robot path planning and map navigation, and the robot can autonomously move from a current position to a certain target position in the environment after the autonomous positioning and planning control work is coordinated.

The multi-sensor fusion positioning mapping method comprises a visual positioning mapping algorithm, a laser positioning mapping algorithm, a pose fusion algorithm, a back-end optimization algorithm and a map fusion algorithm.

The visual positioning map building algorithm is used for robot positioning based on depth camera 6 information, static semantic map building of the environment is achieved through a semantic segmentation algorithm and an optical flow method, the laser positioning map building algorithm is used for robot positioning based on three-dimensional laser radar 4 information to build a 2D grid map, the pose fusion algorithm is used for fusing the pose of the visual positioning map building algorithm and the pose of the laser positioning map building algorithm, the rear-end optimization algorithm is used for optimizing the pose of the robot, the pose of a road mark point and optimizing and updating a map in real time in the robot navigation process, and the map fusion algorithm is used for fusing a semantic octree map and a grid map to build a static 2.5D map and updating the map according to real-time environment information when the robot is in operation.

The multi-sensor fusion positioning map building method comprises the steps that according to sensor information of an ROS system, a visual positioning map building algorithm subscribes to depth image information and color image information of a depth camera 6, visual positioning pose and a semantic octree map are output, according to sensor information of the ROS system, a laser positioning algorithm subscribes to a three-dimensional laser radar 4 sensor information, laser positioning pose and a grid map are output, according to visual positioning pose and laser positioning pose information, a pose fusion algorithm fuses pose information to output estimated robot pose states, according to visual landmark points, laser path landmark points and estimated robot pose states, a rear-end optimization algorithm optimizes the robot pose and landmark points, outputs optimized landmark points and robot pose, and in the robot map building process, according to the robot pose, the semantic octree map and the grid map, a map fusion algorithm fuses the map to construct a static 2.5D map, in the robot navigation process, according to a real-time environment map fusion algorithm updates the map, and outputs real-time local map.

The laser positioning mapping algorithm is used for selecting indexes of key frames, splicing sub-key frames into a local map, establishing a pose solving equation through matching of a current frame and the local map so as to obtain the pose of a laser thread, and outputting a grid map at the same time, wherein the pose fusion algorithm flow is that the laser SLAM thread issues a topic with a message type of/phase_laser; the visual SLAM thread issues topics with message type of/phase_kinect; the method comprises the steps of subscribing the two topics by using a robot_phase_ ekf and outputting a topic with a message type of/phase, and completing fusion of two sensor odometers through a ExtendedKalmanFilter (EKF) filter, wherein the algorithm flow of the back-end optimization module is that the back-end optimization of a visual positioning module is realized, a similar transformation matrix sim3 and a relative posture relation are calculated, the position of a road mark point is adjusted, the pose diagram is optimized, the position of a map point is correspondingly adjusted according to the optimized pose, and global Bundle Adjustment (BA) optimization is carried out.

The map fusion algorithm comprises a local map fusion algorithm and a global map fusion algorithm.

The global map fusion algorithm is used for fusing the static obstacle part of the semantic map with the grid map to generate a 2.5D grid map according to the grid map output by the laser positioning map-in algorithm and the semantic octree map output by the visual positioning module through the map projection algorithm and the map fusion algorithm, and the local map fusion algorithm is used for updating objects different from the prior map in the environment according to real-time environment information, marking pedestrians on the local map and fusing the pedestrians with the global map.

The visual positioning mapping algorithm comprises a tracking thread, a preprocessing thread and a semantic mapping thread.

The tracking thread is used for estimating the pose of the camera, selecting proper frames as key frames through the co-view relation among frames, updating key frames and local map points, deleting mismatching according to the pose, storing the key frames and map points as the basis for executing repositioning or selecting the key frames, writing the key frames into a key frame list, the preprocessing thread is used for carrying out semantic segmentation and optical flow estimation on RGB images, marking a dynamic target area on the original images according to the consistency of the optical flow, marking an area belonging to people on the original images according to the semantic segmentation images, the semantic map building thread is used for constructing a 3D point cloud, generating a point cloud map with semantic labels by combining the semantic segmentation images, and constructing an incremental semantic octree map through an octree generation algorithm and a semantic fusion algorithm.

The robot navigation method comprises a global path planning algorithm, a dynamic social interaction space algorithm, a multi-mode pedestrian track prediction algorithm and a local path planning algorithm, wherein the global path planning algorithm is used for carrying out global path planning according to global map information, the dynamic social interaction space algorithm is used for establishing two states of static and moving of individuals and crowds respectively based on task constraint and social constraint conditions, describing respective dynamic social interaction space and navigation obstacle avoidance, the multi-mode pedestrian track prediction algorithm is used for predicting pedestrian positions with high precision, adding information of the individuals and the obstacles to a local map and assisting pedestrians to provide basis, and the local path planning algorithm is used for carrying out local path planning according to the local map information.

Specifically, the robot navigation process is that a local map recognition process is carried out according to prior 2.5D map information and a multi-sensor fusion positioning algorithm, visual and laser map features of a transmitted local map are compared with an original map to complete a robot repositioning task and a local map updating task, global path planning is carried out according to a 2.5D grid map, global path planning and local obstacle avoidance are cooperated, two states of static and moving of a person and a crowd are respectively established according to task constraint and social constraint conditions through a dynamic social interaction space algorithm, respective dynamic social interaction space is described, real-time updated sensor collected by a robot is introduced to detect data of the person and the crowd, a multi-mode pedestrian track prediction method is adopted, interaction influences of a time space state diagram for reasoning pedestrians, the robot and the like are established after a sufficient pedestrian position sequence is extracted, high-precision moving target position prediction is completed, and local dynamic path planning is carried out according to behavior and moving target position information, and moving targets are avoided.

The robot runs under the environment covered by a network by virtue of a motion chassis and carries a three-dimensional laser radar 4 sensor and a module thereof, a depth camera 6, a display and a vehicle-mounted computer 1 (NUC), wherein the depth camera 6 and the three-dimensional laser radar 4 acquire original image information and radar point cloud information generated by the sensor in the environment.

The NUC is based on the ROS open source system to transmit information, package the original image information into color image information/camera_rgb and depth image information/camera_depth to be transmitted, and transmit the color image information/camera_depth to the cloud SLAM server for processing through a wireless network port.

The RGB image information is respectively fed into a tracking thread and a preprocessing thread of the visual SLAM, in the tracking thread, an image pyramid of each frame of RGB image is calculated, ORB characteristics are extracted, descriptors are calculated, meanwhile, the preprocessing thread can carry out image semantic segmentation on the RGB image by using a PSPnet network, masks are added to an original image, characteristic points belonging to a human or potential dynamic target area in the image are removed, an LB method is used for carrying out optical flow estimation, and the characteristic points belonging to a dynamic target are removed through optical flow consistency.

The tracking thread calculates Bagofwords (BOW) feature vectors of the current frame, sets a matching threshold, performs feature matching by using continuous static feature points, performs pose estimation on the camera by using a Perspotive-n-Point (PNP) method according to whether the motion model is met, selects proper frames as key frames by using a common view relation among frames, updates key frames and local map points, performs projection matching on the local map points, optimizes the current frame by using a pose map, deletes mismatching according to pose, saves the key frames and map points to serve as a basis for executing repositioning or selecting the key frames, and writes the key frames into a key frame list.

The map line Cheng Charu key frame of the visual SLAM is constructed, local Bundle Adjustment (BA) optimization is performed by removing redundant map points, the redundant key frame is removed, a 3D point cloud is constructed by matching between a depth image and a reference frame and the position of a pixel in the image and the internal reference of a camera while the sparse feature point cloud is reserved for later loop detection service, and a point cloud map with semantic tags is generated by combining semantic segmentation images. The point cloud is subjected to downsampling at a given certain resolution through a point cloud filter, then is inserted into nodes of the octree, the occupancy rate of voxels with different resolutions is updated, and semantic information of the multi-view octree is fused in a Bayes fusion mode to construct an incremental semantic octree map.

In the laser SLAM thread, according to the curvature of points serving as an index for extracting characteristic information of laser frames, the point with lower curvature serving as a plane point and the point with higher curvature serving as an edge point, matching of two laser frames is completed through the corresponding plane point and the edge point between two frames, a rotation angle exceeding 5 degrees or translation exceeding 10cm is defined as a selection index of a key frame, 10 laser frames near the key frame are selected as sub-key frames, the sub-key frames are spliced into a local map, a pose solving equation is established through matching of the current frame and the local map, so that the pose of the laser thread is obtained, and meanwhile, a grid map is output.

Ros communication pose fusion: the laser SLAM thread issues topics with message type of/phase_laser; the visual SLAM thread issues topics with message type of/phase_kinect; the method comprises the steps of subscribing the two topics by adopting the robot_post_ ekf and outputting a topic with the message type of/post, so that fusion of two sensor odometers is completed.

And loop detection: the visual SLAM and the laser SLAM respectively perform loop detection, the loop detection of the visual SLAM detects continuous candidate frames in the candidate frames by calculating closed-loop candidate frames, calculates a similar transformation matrix sim3 and a relative posture relation, adjusts the positions of key frame poses connected with the current frame and map points observed by the key frames, matches the road mark points of the closed-loop frames and the key frames connected with the closed-loop frames with the points of the key frames connected with the current frame, updates a common view through the matching relation among the frames, optimizes the pose map, correspondingly adjusts the positions of the map points according to the optimized poses, and performs global BA optimization.

After the robot works in a building map, according to the current environment information, the sensors acquire data, the current position is correspondingly matched in a 2.5D grid map, the autonomous positioning work is completed, the robot comprehensively considers the safety of the robot and the surrounding environment according to the target position to plan a moving path meeting multiple evaluation indexes, and the indoor mobile robot chassis executing mechanism completes the planning control work.

The overall path planning, firstly, the image construction precision is improved through fusion image construction of laser and visual sensors, then, the advantages of parallel calculation and multi-dimensional search are fully exerted by using a double-layer ant colony algorithm, finally, redundant turning points are removed through smoothing processing, the path is optimized, the actual running requirement of a robot is met, and the overall path planning and the local obstacle avoidance are cooperated.

The multi-mode pedestrian track prediction method improves the existing uniform speed prediction method into uniform speed and uniform acceleration motion prediction to improve short-time prediction precision through a polynomial curve fitting algorithm, establishes a time space state diagram to infer interaction influence of each agent (pedestrians and robots) after a sufficient pedestrian position sequence is extracted, completes high-precision pedestrian position prediction, further judges whether each service object is in the same group through clustering of pedestrian prediction positions of multiple time steps, and provides effective reasoning information for navigation process.

The method comprises the steps of local path planning, firstly obtaining the current pose of a robot in a local map, and then establishing a dynamic social interaction space by combining pedestrian detection information, pedestrian state extraction information and social behavior information based on task constraint and social constraint conditions; and then, introducing real-time updated pedestrian track prediction and crowd grouping information collected by the robot, and finally, adjusting a dynamic window weight value and weight combination mode in real time so as to improve a dynamic window evaluation function, complete safety obstacle avoidance and path planning, and meet the comfort and safety of human bodies in a social interaction environment.

Firstly, NUC converts a path execution control instruction in a move_base into a specific robot chassis motion control instruction: the method comprises the steps of selecting a mode by a driver, configuring driving enabling and controlling speed of a chassis motor, then sending a control command in an event triggering mode through can port communication, taking 50ms for two frames of control command time of the same type without a fixed period, feeding back the execution condition of the robot chassis, feeding back real-time driving state and motor rotating speed information to an upper computer through can ports, and controlling and adjusting.

A multi-sensor fusion positioning navigation method of a robot based on cloud service; the method comprises the following steps:

step one: and the robot completes the relocation and map updating on the basis of the prior map inspection, namely, the environmental characteristics are judged, the current position is calibrated, the local map is updated, the robot completes the optimal path planning meeting the multiple evaluation indexes such as social constraint, task constraint and the like according to the environmental information, and the robot executing mechanism is controlled to reach the target position according to the planning result.

Step three: the three-dimensional laser radar 4 adopts a particle filtering mode to realize the construction of a grid map, firstly randomly selecting particles according to prior probability, endowing weight values, carrying out state initialization, and then generating a particle set of the next generation from the current particle set according to proposal distribution and calculating the particle weight values; resampling particles, determining sampling times by weight of the particles, estimating pose through state parameters, obtaining a grid map based on laser data, semantic image information and point cloud information, generating an octree semantic map by utilizing the point cloud map, updating semantic information of the octree map by Bayesian fusion of semantic tags at different moments to form the octree semantic map, updating uncertainty of the grid map by using a Bayesian fusion algorithm according to laser radar information, fusing local grid maps until all the grid maps are fused, and thus completing the fusion of two sensor data to construct a map to obtain a 2.5D grid map containing semantic information;

The invention provides an indoor service robot multi-sensor fusion positioning navigation system and method based on cloud service, wherein the robot CAN realize efficient and accurate positioning and mapping under the environment covered by a network to realize the navigation purpose, the robot senses the external environment through a loaded three-dimensional laser radar 4 and a depth camera 6 under the position environment, the data information is transmitted into a vehicle-mounted computer 1, a vehicle-mounted computing unit encapsulates sensor information through a Robot Operating System (ROS) operating system, the sensor information is transmitted to a cloud server through a wireless network, the cloud server subscribes the sensor information through the ROS system and operates a robot positioning navigation algorithm, the navigation information is transmitted to the vehicle-mounted computer 1 through the network, the vehicle-mounted computer 1 transmits the processed navigation information to a driving control board through a CAN (controller area network) port to drive a motor to rotate, so as to realize the navigation target, and meanwhile, a client CAN establish communication, and CAN realize real-time checking and controlling of the running condition of the robot.

The invention provides a cloud service-based indoor service robot multi-sensor fusion positioning navigation system and a cloud service-based indoor service robot multi-sensor fusion positioning navigation method. Under the position environment of the robot, the robot senses the external environment through the loaded three-dimensional laser radar and the depth camera, transmits data information into the vehicle-mounted computer, the vehicle-mounted computing unit encapsulates sensor information through a Robot Operating System (ROS) operating system, sends the sensor information to the cloud server through a wireless network, the cloud server subscribes the sensor information through the ROS system, operates a robot positioning navigation algorithm, sends navigation information to the vehicle-mounted computer through the network, and the vehicle-mounted computer sends the processed navigation information to the driving control board through the CAN port to drive the motor to rotate so as to realize navigation targets.

According to one aspect of the invention, a system and a method for positioning and navigation of a robot based on multi-sensor data fusion in a cloud mode are provided, wherein the system and the method comprise the following steps:

a robot navigation method based on cloud service, a navigation cloud service platform and a robot body, wherein:

the robot body is used for carrying a sensor and a vehicle-mounted computer, and is communicated with the navigation cloud service platform to realize stable movement;

the navigation cloud service platform is used for sending and receiving data of the client and the robot, completing a multi-sensor fusion positioning navigation algorithm in the cloud server, establishing communication with the client, and sending a navigation message to the robot;

the robot navigation method based on the cloud service is used for processing laser and vision sensor information, completing robot positioning, constructing a 2.5D grid map and realizing robot navigation.

Further, in the above system and method, the robot body includes: indoor mobile robot chassis, on-vehicle computer, three-dimensional laser radar, depth camera, LCD, wherein:

the indoor mobile robot chassis is used for carrying sensors and driving equipment to realize stable running of the robot;

The vehicle-mounted computer is used for reading the packed sensor data and sending the packed sensor data to the navigation cloud service platform through the wireless network;

the depth camera is used for extracting color information and depth information in the environment;

the three-dimensional laser radar is used for carrying out scanning matching on the environmental information;

the liquid crystal display is used for displaying real-time running information of the robot.

Further, the indoor mobile robot chassis includes: an omnidirectional wheel, a shell, a shock absorber, a section bar frame, a mounting plate, a driving motor, a driving control board and a vehicle-mounted battery,

specifically, the indoor mobile robot chassis is connected in a manner that the shell is divided into an upper shell and a lower shell by a hexagonal structure, three shock absorbers are respectively arranged on the shell and connected with the upper surface of the lower shell, each shock absorber consists of two spring structures and an upper connecting piece and a lower connecting piece, the front ends of three driving motors are respectively connected with the lower connecting pieces of the shock absorbers, the rear ends of the three driving motors are connected with the lower shell, three omni wheels are connected with a driving motor shaft, each omni wheel consists of 12 driven wheels, 2 hubs and connecting pieces, a driving control board is arranged on the lower shell, a vehicle-mounted battery is arranged on the lower shell, a profile frame is formed by connecting 8 aluminum profiles and connected with the upper surface of the upper shell, and the profile frame is divided into three layers by the mounting plate;

Specifically, the connected mode of robot body is, on-vehicle computer is connected with the top of last casing, three-dimensional laser radar is connected with upper mounting panel, and three-dimensional laser radar module is connected with middle level mounting panel, the depth camera is connected with one side of section bar frame, and the installation direction coincides with the positive direction of robot, liquid crystal display is connected with one end of section bar frame, and the installation direction is 45 degrees with the vertical direction slope.

Further, in the above system and method, the navigation cloud service platform includes: robot end, cloud server, customer end, network equipment, wherein:

the robot end is used for sending sensor data in real time;

the cloud server is used for carrying a robot positioning navigation algorithm and processing data in real time;

the client is used for checking and controlling the running condition of the robot in real time;

the network equipment is used for providing a cloud server to access network services;

preferably, the present invention uses message transmission based on ROS systems for communication.

Preferably, the transmission between the robot end and the cloud server end is based on a TCP/IP protocol.

The robot terminal comprises: a robot end ROS system and a wireless network card module,

The robot end ROS system is used for collecting image information and radar point cloud information generated in the operation of the sensor, and packaging the information into ROS information for broadcasting;

and the wireless network card module is used for transmitting the information to the cloud server for processing through the wireless network card after collecting the information.

The cloud server includes: server side ROS system, processor, memory, hard disk, display card and operating system.

The network device includes: routers, switches.

Specifically, the cloud server has a host model of DELL T7920; the processor is Intel to strong gold medal 5118;12 core 24 threads, main frequency 2.3GHz; the memory is three-star DDR4 2666MHz 128GB,64GB*2; the hard disk is Intel solid state 2TB 2; the display card is Nvidia Quard RTX 5000, 16GB video memory and 3072 CUDA cores; the operating system is Ubuntu 18.04LTS, and the network card is Intel I219-LM 1000Mbps Ethernet card.

Specifically, the vehicle-mounted computer is of the model number of Intel NUC8i7HVK; the processor is Intel borui 7 8809G; the display card is AMD Radeon RX Vega M GH; the memory is Wei-just DDR4 2666MHz 16GB; the wireless network card is Intel 9560AC and supports 802.11AC wireless protocol.

Specifically, the driving control board is a processor STM32F103; ULN2003 motor drive module; and a CAN interface.

Specifically, the router is a TL-XDR6070, IEEE 802.11a/b/g/n/ac/ax wireless protocol, and a highest wireless rate 5952Mbps (2.4GHz 1148Mbps,5GHz 4804Mbps) 3 10/100/1000Mbps rate adaptive Ethernet interface.

Specifically, the switch is an RG-S1824+ of the acute network, the transmission speed is 10/100Mbps, and the back plate bandwidth is 4.8Gbps.

Specifically, the three-dimensional laser radar is a three-dimensional multi-line laser radar VLP-16, and the laser lines are as follows: 16 lines, measuring range: up to 100m, measurement accuracy: 3cm.

Specifically, the depth camera is Kinectv1, the color resolution is 640×480, the fps is 30fps, and the depth resolution is 320×240.

Further, in the above system and method, the method for navigating a robot based on a cloud service includes: a multi-sensor fusion positioning mapping method, a robot navigation method, wherein:

the multi-sensor fusion positioning mapping method is used for sensing and positioning the robot in an unknown environment and constructing a global static map;

the robot navigation method is used for robot path planning and map navigation.

The multi-sensor fusion positioning mapping method comprises the following steps: a visual positioning mapping algorithm, a laser positioning mapping algorithm, a pose fusion algorithm, a back-end optimization algorithm and a map fusion algorithm, wherein:

the visual positioning mapping algorithm is used for robot positioning based on depth camera information, and static semantic map construction of the environment is realized through a semantic segmentation algorithm and an optical flow method;

the laser positioning mapping algorithm is used for robot positioning based on three-dimensional laser radar information to construct a 2D grid map;

the pose fusion algorithm is used for fusing the pose of the visual positioning mapping algorithm and the pose of the laser positioning mapping algorithm;

the back-end optimization method is used for optimizing the pose of the robot, marking the pose of the point and optimizing and updating the map in real time in the navigation process of the robot;

the map fusion algorithm is used for fusing semantic octree maps and grid maps to establish static 2.5D maps, and updating the maps according to real-time environment information when the robot runs.

Specifically, the flow of the multi-sensor fusion positioning mapping method is that according to ROS system sensor information, a visual positioning mapping algorithm subscribes to depth image information and color image information of a depth camera, and outputs visual positioning pose and a semantic octree map; according to the sensor information of the ROS system, the laser positioning algorithm subscribes to the three-dimensional laser radar sensor information and outputs the laser positioning pose and the grid map; according to the visual positioning pose and the laser positioning pose information, a pose fusion algorithm fuses the pose information and outputs the estimated robot pose state; according to the visual road marking points, the laser road marking points and the estimated pose state of the robot, a back-end optimization algorithm optimizes the pose of the robot and the road marking points, and outputs the optimized road marking points and the pose of the robot; and in the robot map building process, according to the pose of the robot, the semantic octree map and the grid map, the map is fused by a map fusion algorithm to construct a static 2.5D map, and in the robot navigation process, the map is updated according to a real-time environment information map fusion algorithm, and a real-time local map is output.

The visual positioning mapping algorithm comprises three threads, including: tracking a thread, preprocessing the thread and establishing a semantic graph thread;

the tracking thread is used for estimating the pose of the camera, selecting a proper frame as a key frame through the co-view relation among frames, updating the key frame and a local map point, deleting mismatching according to the pose, storing the key frame and the map point as the basis for executing repositioning or selecting the key frame, and writing the key frame into a key frame list;

the preprocessing thread is used for carrying out semantic segmentation and optical flow estimation on the RGB image, marking a dynamic target area on the original image according to the consistency of the optical flow, and marking an area belonging to a person on the original image according to the semantic segmentation image;

the semantic mapping thread is used for constructing a 3D point cloud, generating a point cloud map with semantic tags by combining semantic segmentation images, and constructing an incremental semantic octree map through an octree generation algorithm and a semantic fusion algorithm.

The laser positioning mapping algorithm is used for splicing sub-key frames into a local map for the selection index of the key frames, and establishing a pose solving equation through matching of the current frame and the local map so as to obtain the pose of the laser thread and output a grid map. .

The pose fusion algorithm flow is that a laser SLAM thread issues topics with message type of/phase_laser; the visual SLAM thread issues topics with message type of/phase_kinect; adopting a robot_post_ ekf to subscribe the two topics and outputting a topic with the message type of/post, and finishing the fusion of the two sensor odometers through a ExtendedKalmanFilter (EKF) filter;

the back-end optimization module is used for optimizing the back-end of the visual positioning module, calculating a similarity transformation matrix sim3 and a relative posture relation, adjusting the position of a road mark point, optimizing a pose chart, correspondingly adjusting the position of the map point according to the optimized pose, and performing global Bundle Adjustment (BA) optimization.

The map fusion algorithm comprises the following steps: a local map fusion algorithm and a global map fusion algorithm;

the global map fusion algorithm fuses the static obstacle part of the semantic map and the grid map through the map projection algorithm and the map fusion algorithm according to the grid map output by the laser positioning map-entering algorithm and the semantic octree map output by the visual positioning module to generate a 2.5D grid map;

the local map fusion algorithm is used for updating objects different from the prior map in the environment according to real-time environment information, marking pedestrians on the local map and fusing the pedestrians with the global map.

The robot navigation method comprises the following steps: a global path planning algorithm, a dynamic social interaction space algorithm, a multi-mode pedestrian track prediction algorithm and a local path planning algorithm, wherein:

the global path planning algorithm is used for carrying out global path planning according to global map information;

the dynamic social interaction space algorithm is used for establishing two states of a person and a crowd in static and moving respectively based on task constraint and social constraint conditions, describing respective dynamic social interaction spaces and navigating obstacle avoidance;

the multi-mode pedestrian track prediction algorithm is used for predicting the positions of pedestrians with high precision, adding information structures of people and obstacles to a local map, and providing basis for assisting in avoiding pedestrians;

the local path planning algorithm is used for carrying out local path planning according to the local map information.

Specifically, the robot navigation side process is that a local map identification process is carried out according to prior 2.5D map information and a multi-sensor fusion positioning algorithm, visual and laser map features of a transmitted local map are compared with an original map to complete a robot repositioning task and a local map updating task, global path planning is carried out according to a 2.5D grid map, global path planning and local obstacle avoidance are cooperated, two states of static and moving of a person and a crowd are respectively established according to task constraint and social constraint conditions through a dynamic social interaction space algorithm, respective dynamic social interaction space is described, real-time updated sensor collected by a robot is introduced to detect data of the person and the crowd, interaction influence of a pedestrian, the robot and the like is inferred according to a time space state diagram after a sufficient pedestrian position sequence is extracted through a multi-mode pedestrian track prediction method, high-precision moving target position prediction is completed, and local dynamic path planning is carried out according to behavior and moving target position information, and moving targets are avoided.

Portions of the present application may be implemented as a computer program product, such as computer program instructions, which when executed by a computer, may invoke or provide methods and/or techniques in accordance with the present application by way of operation of the computer. Program instructions for invoking the methods of the present application may be stored in fixed or removable recording media and/or transmitted via a data stream in a broadcast or other signal bearing medium and/or stored within a working memory of a computer device operating according to the program instructions. An embodiment according to the present application comprises an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein the computer program instructions, when executed by the processor, trigger the apparatus to operate a method and/or a solution according to the embodiments of the present application as described above.

In order to achieve the above-mentioned purpose, the embodiment of the present invention provides a positioning navigation system for a multi-sensor data fusion robot based on a cloud computing mode, which comprises the following specific steps:

in the embodiment of the invention, as shown in the first figure, the multi-sensor data fusion robot positioning navigation system based on the cloud computing mode comprises the following steps:

the robot runs under the environment covered by the network by virtue of the motion chassis and is loaded with a three-dimensional laser radar sensor and a module, a depth camera, a display and a vehicle-mounted computer (NUC) thereof;

the depth camera and the laser radar acquire original image information and radar point cloud information generated by the sensor in the environment.

The NUC is based on an ROS open source system to carry out information transmission, original image information is packed into color image information/carrier_rgb and depth image information/carrier_depth to be transmitted, and the color image information/carrier_depth is transmitted to a cloud SLAM server through a wireless network port to be processed;

The RGB image information respectively enters a tracking thread and a preprocessing thread of the visual SLAM, in the tracking thread, an image pyramid of each frame of RGB image is calculated, ORB characteristics are extracted, descriptors are calculated, meanwhile, the preprocessing thread performs image semantic segmentation on the RGB image by using a PSPnet network, masks are added to an original image, and characteristic points belonging to a human or potential dynamic target area in the image are removed. And performing optical flow estimation by using an LB method, and removing characteristic points belonging to the dynamic target through optical flow consistency.

The tracking thread calculates a bag of words (BOW) feature vector of a current frame, sets a matching threshold, performs feature matching by using continuous static feature points, performs pose estimation on a camera by using a Perspotive-n-Point (PNP) method according to whether the motion model is met, selects a proper frame as a key frame through a common view relation among frames, updates the key frame and a local map Point, performs projection matching on the local map Point, optimizes the current frame by using a pose map, deletes mismatching according to pose, saves the key frame and the map Point to be used as a basis for executing repositioning or selecting the key frame, and writes the key frame into a key frame list.

Ros communication pose fusion: the laser SLAM thread issues topics with message type of/phase_laser; the visual SLAM thread issues topics with message type of/phase_kinect; the robot_post_ ekf is adopted to subscribe to the two topics and output the topic with the message type of/post, so that the fusion of the two sensor odometers is completed

And loop detection is carried out, and the vision SLAM and the laser SLAM respectively carry out loop detection. The loop detection of the visual SLAM is carried out by calculating closed-loop candidate frames, detecting continuous candidate frames in the candidate frames, calculating a similar transformation matrix sim3 and a relative posture relation, adjusting the positions of key frames connected with the current frame and map points observed by the key frames, matching the closed-loop frames, road mark points of the key frames connected with the closed-loop frames and the points of the key frames connected with the current frame, updating a common view through the matching relation among frames, optimizing the pose map, correspondingly adjusting the positions of the map points according to the optimized pose, and carrying out global BA optimization.

After the robot works to build the map, according to the current environmental information, the sensor collects data, and correspondingly matches the data in the 2.5D grid map, and the current position is calibrated to complete the autonomous positioning work; and the robot comprehensively considers the safety of the robot and the surrounding environment according to the target position to plan a moving path meeting multiple evaluation indexes, controls a robot chassis executing mechanism and completes planning control work.

The method comprises the steps of local path planning, firstly, obtaining the current pose of a robot in a local map, and then, establishing a dynamic social interaction space by combining pedestrian detection information, pedestrian state extraction information and social behavior information based on task constraint and social constraint conditions; then, introducing real-time updated pedestrian track prediction and crowd grouping information collected by the robot; finally, the dynamic window weight value and the weight combination mode are adjusted in real time, so that the dynamic window evaluation function is improved; and (5) completing safe obstacle avoidance and path planning. The comfort and safety of human bodies in a social interaction environment are met.

Specifically, in the robot planning control method, first, NUC converts a path execution control instruction in move_base into a specific robot chassis motion control instruction: the method comprises the steps of selecting a mode by a driver, configuring driving enabling and controlling speed of a chassis motor, then sending a control command in an event triggering mode through can port communication, taking 50ms for two frames of control command time of the same type without a fixed period, feeding back the execution condition of the robot chassis, feeding back real-time driving state and motor rotating speed information to an upper computer through can ports, and controlling and adjusting.

The system comprises a vehicle-mounted computer, a three-dimensional laser radar, a liquid crystal display, a depth camera, an indoor mobile robot chassis and a network construction method, wherein the vehicle-mounted computer, the three-dimensional laser radar, the liquid crystal display, the depth camera, the indoor mobile robot chassis and the network construction method are based on multi-sensor data fusion robot positioning navigation system in a cloud computing mode, the robot positioning mapping method in the cloud mode, a cloud computing platform of cloud SLAM and the navigation method by using a priori semantic map;

the laser radar is used for carrying out scanning matching on the environmental information;

the multi-sensor fusion is that the depth camera and the three-dimensional laser radar are fused on the data layer;

The vehicle-mounted computer is used for packaging the collected sensor data, establishing communication with the upper cloud end server through the ROS node, transmitting the data to the cloud end server, receiving response information of the server end, and further controlling the behavior of the vehicle according to the response information of the cloud end server.

The positioning and mapping method is used for positioning the accurate position of the robot in the environment and the map and constructing a semantic grid map which can be used for navigation, and is used for the robot to interact with the environment in the map;

the autonomous navigation method is used for enabling the robot to autonomously move from the current position to a certain target position in the environment after the autonomous positioning and planning control work are coordinated.

The positioning and mapping method comprises the following steps:

a preprocessing thread, which is used for processing the color image and comprises two parts of semantic segmentation and optical flow estimation for the image: the semantic segmentation thread is used for acquiring semantic segmentation images extracted by the deep learning network;

the tracking thread is used for simultaneously estimating the pose by adopting a visual algorithm and a laser algorithm, wherein the visual algorithm utilizes information provided by the depth camera to estimate the pose, and the laser algorithm is used for estimating the pose by a method for scanning matching features;

And the pose fusion thread is used for fusing the laser pose and the visual pose through a fusion algorithm.

The rear-end optimization thread is used for simultaneously introducing pose information and road sign information of vision and laser into the factor graph for optimization;

and the map construction thread is used for processing the dense map points of the depth image and the laser data and constructing a map for navigation of the robot.

The autonomous navigation method comprises the following steps:

the robot finishes repositioning and map updating on the basis of the prior map inspection, namely judging the environmental characteristics, calibrating the current position and updating the local map;

and the robot completes the optimal path planning meeting the social constraint, task constraint and other multiple evaluation indexes according to the environmental information, and the planning result controls the robot executing mechanism to reach the target position.

The visual algorithm performs pose estimation: the method comprises the steps of extracting feature points of a non-human part image in an image by adopting a semantic segmentation algorithm in a preprocessing thread, extracting pixels with inconsistent dynamic states in the image by adopting an optical flow algorithm, jointly removing dynamic feature points with high possibility in the image, and obtaining a transformation matrix to obtain a stable camera pose by matching residual stable features.

The map construction method comprises the following steps: the laser radar adopts a particle filtering mode to realize the construction of a grid map, firstly randomly selecting particles according to prior probability, giving weight values, and carrying out state initialization; generating a next generation particle set from the current particle set according to the proposal distribution and calculating a particle weight; and resampling the particles, wherein the sampling times are determined by the weight of the particles, and finally, estimating the pose through the state parameters, and obtaining the grid map based on the laser data.

The semantic image information and the point cloud information: firstly, generating an octree semantic map by utilizing a point cloud map, and updating semantic information of the octree map by Bayesian fusion of semantic tags at different moments to form the octree semantic map;

the laser radar information: and updating the uncertainty of the grid map by using a Bayesian fusion algorithm, and fusing the local grid map until all the grid maps are fused, so that two sensor data fusion construction maps are completed, and the 2.5D grid map containing semantic information is obtained.

The repositioning and map updating: after the 2.5D global semantic map is obtained, the map is subjected to global matching according to the current laser information and the visual word bag characteristics, a repositioning process is completed, an original key frame is grabbed, and a laser grid map and a semantic map corresponding to the original key frame are updated.

The optimal path planning: the image construction precision is improved through fusion image construction of laser and visual sensors, then the advantages of parallel calculation and multi-dimensional search are fully exerted by using a double-layer ant colony algorithm, and finally redundant turning points are removed through smoothing treatment, paths are optimized, and global path planning and local obstacle avoidance are cooperated. The multi-mode pedestrian track prediction method improves the existing uniform speed prediction method into uniform speed and uniform acceleration motion prediction to improve short-time prediction precision through a polynomial curve fitting algorithm, establishes a time space state diagram to infer interaction influence of each agent (pedestrians and robots) after a sufficient pedestrian position sequence is extracted, completes high-precision pedestrian position prediction, further judges whether each service object is in the same group through clustering of pedestrian prediction positions of multiple time steps, and provides effective reasoning information for navigation process.

The social constraint: in a dual-flow network with cross-modal feature fusion, RGB and depth features are extracted and fused, so that accurate prediction of pedestrians is realized. Based on pedestrian detection, combining laser radar point cloud data to acquire pedestrian states including pedestrian position, moving direction and other information, so as to realize 3D pedestrian detection. And meanwhile, combining the depth image data to acquire pedestrian skeleton data, and fusing RGB image information, depth image information and human skeleton information in a behavior recognition network to realize the social behavior recognition of pedestrians. And finally, combining pedestrian state extraction and social behavior recognition detection results, and constructing a social interaction space model on the 2.5D grid map.

Although the present invention has been described with reference to the foregoing embodiments, it will be apparent to those skilled in the art that modifications may be made to the embodiments described, or equivalents may be substituted for elements thereof, and any modifications, equivalents, improvements and changes may be made without departing from the spirit and principles of the present invention.

Claims

1. The cloud service-based multi-sensor fusion positioning navigation system of the robot comprises a robot and a navigation cloud service platform, wherein the robot comprises an indoor mobile robot chassis, a vehicle-mounted computer (1), a three-dimensional laser radar (4), a depth camera (6) and a liquid crystal display (3), the indoor mobile robot chassis comprises an omnidirectional wheel (8), a shell (9), a shock absorber (7), a profile frame (2), a mounting plate, a driving motor, a driving control board and a vehicle-mounted battery, and the navigation cloud service platform comprises a robot end, a cloud server, a client and network equipment; the method is characterized in that:

step one: the robot completes repositioning and map updating on the basis of the prior map inspection, namely judging the environmental characteristics, calibrating the current position, updating the local map, completing optimal path planning conforming to social constraint and task constraint according to the environmental information, and controlling the robot executing mechanism to reach the target position according to the planning result;

step three: the three-dimensional laser radar (4) adopts a particle filtering mode to realize the construction of a grid map, firstly randomly selecting particles according to prior probability, endowing weight values, carrying out state initialization, and then generating a particle set of the next generation from the current particle set according to proposal distribution and calculating the particle weight values; resampling particles, determining sampling times by weight of the particles, estimating pose through state parameters, obtaining a grid map based on laser data, semantic image information and point cloud information, generating an octree semantic map by utilizing the point cloud map, updating semantic information of the octree map by Bayesian fusion of semantic tags at different moments to form the octree semantic map, updating uncertainty of the grid map by using a Bayesian fusion algorithm according to laser radar information, fusing local grid maps until all the grid maps are fused, and thus completing the fusion of two sensor data to construct a map to obtain a 2.5D grid map containing semantic information;

Step four: after a 2.5D global semantic map is obtained, carrying out global matching on the map according to the current laser information and the visual word bag characteristics, completing a repositioning process, capturing an original key frame, and updating a laser grid map and a semantic map corresponding to the original key frame; step five: the image construction precision is improved through fusion image construction of laser and a visual sensor, then a double-layer ant colony algorithm is utilized to fully exert the advantages of parallel calculation and multi-dimensional search, and finally redundant turning points are removed through smoothing treatment, a path is optimized, and global path planning and local obstacle avoidance are cooperated; the multi-mode pedestrian track prediction method is characterized in that the conventional uniform speed prediction method is improved into uniform speed and uniform acceleration motion prediction through a polynomial curve fitting algorithm to improve short-time prediction precision, a time space state diagram is established after a sufficient pedestrian position sequence is extracted to infer the interaction effect of each agent, high-precision pedestrian position prediction is completed, and then whether each service object is in the same group or not is judged through clustering pedestrian prediction positions of multiple time steps, so that effective reasoning information is provided for the navigation process; step six: in a dual-flow network with cross-modal feature fusion, RGB and depth features are extracted and fused, accurate prediction of pedestrians is achieved, on the basis of pedestrian detection, pedestrian states including pedestrian positions and moving directions are obtained by combining laser radar point cloud data, and 3D pedestrian detection is achieved; and meanwhile, combining the depth image data to acquire pedestrian skeleton data, fusing RGB image information, depth image information and human skeleton information in a behavior recognition network to realize the social behavior recognition of pedestrians, and finally, combining the pedestrian state extraction and the social behavior recognition detection result to construct a social interaction space model on a 2.5D grid map.

2. The cloud service-based multi-sensor fusion positioning navigation method for the robot, which is characterized in that: the indoor mobile robot chassis is used for carrying sensors and driving equipment to realize stable operation of the robot, the vehicle-mounted computer (1) is used for reading and packing sensor data and sending the sensor data to the navigation cloud service platform through a wireless network, the depth camera (6) is used for extracting color information and depth information in the environment, the three-dimensional laser radar (4) is used for scanning and matching the environment information, the liquid crystal display (3) is used for displaying real-time operation information of the robot, multi-sensor fusion is that is fusion of the depth camera (6) and the three-dimensional laser radar (4) on a data layer, the vehicle-mounted computer (1) is used for packing collected sensor data and establishing communication with an upper cloud end server through a ROS node, transmitting the data to the cloud end server, receiving response information of the cloud end, and further controlling the behavior of the vehicle according to the response information of the cloud end.

3. The cloud service-based multi-sensor fusion positioning navigation method for the robot, which is characterized in that: the robot end comprises a robot end ROS system and a wireless network card module, wherein the robot end ROS system is used for collecting image information and radar point cloud information generated in the operation of a sensor and packaging the information into ROS information for broadcasting, the wireless network card module is used for transmitting the information to a cloud server for processing after collecting the information through a wireless network card, the cloud server comprises a server end ROS system, a processor, a memory, a hard disk, a display card and an operating system, and the network equipment comprises a router and a switch.

4. The cloud service-based multi-sensor fusion positioning navigation method for the robot, which is characterized in that: the cloud server is used for carrying a robot positioning navigation algorithm and processing the data in real time, the client is used for checking and controlling the running condition of the robot in real time, and the network equipment is used for providing the cloud server to access network services.

5. The cloud service-based multi-sensor fusion positioning navigation method for the robot, which is characterized in that: the multi-sensor fusion positioning map building method is used for sensing and positioning of a robot in an unknown environment and building a global static map, accurate positions in the environment and the map are used for building a semantic grid map which can be used for navigation, the robot interacts with the environment in the map, the robot navigation method is used for robot path planning and map navigation, the robot can autonomously move to a certain target position in the environment from the current position after autonomous positioning and planning control work are coordinated, the multi-sensor fusion positioning map building method comprises a visual positioning map building algorithm, a laser positioning map building algorithm, a pose fusion algorithm, a rear end optimization algorithm and a map fusion algorithm, the visual positioning map building algorithm is used for robot positioning based on depth camera (6) information, the laser positioning map building algorithm is used for building a 2D grid map based on the robot positioning of three-dimensional laser radar (4) information, the pose fusion is used for fusing the pose of the visual positioning algorithm and the static map of the map, the pose fusion algorithm is used for optimizing the robot map at the end of the map building map in real time according to the map updating algorithm of the map of the robot map, and the map is used for optimizing the map building of the map in real time when the robot map is used for updating the map in the navigation map in the position of the robot map in the navigation system.

6. The cloud service-based multi-sensor fusion positioning navigation method for the robot, which is characterized in that: the map fusion algorithm comprises a local map fusion algorithm and a global map fusion algorithm.

7. The cloud service-based multi-sensor fusion positioning navigation method for the robot, which is characterized in that: the visual positioning mapping algorithm comprises a tracking thread, a preprocessing thread and a semantic mapping thread.