CN115546829A - Pedestrian spatial information sensing method and device based on ZED (zero-energy-dimension) stereo camera - Google Patents

Abstract

The invention discloses a pedestrian spatial information sensing method and device based on a ZED stereo camera, which are mainly used for intelligently sensing the spatial position and the moving speed of a pedestrian by a guiding robot in public scenes such as an exhibition hall and the like. Real-time data in a scene are collected by a ZED binocular vision camera and uploaded to a cloud server; inputting the preprocessed RGB data into a deployed human key point detection network to obtain human key point two-dimensional information, and generating a pedestrian enclosure frame according to the human two-dimensional key point information of the upper half body trunk region of the pedestrian; continuously tracking multi-target pedestrians under continuous multiple frames; acquiring three-dimensional space coordinates of key points of the human body in a corresponding area by combining point cloud data, and calculating the spatial position and the moving speed of the pedestrian; and finally, the navigation robot performs body movement control according to the acquired pedestrian space information, and intelligent navigation tasks such as autonomous following and obstacle avoidance are completed, so that the flexibility of the navigation robot is increased, and the interaction experience of visitors is improved.

Description

Pedestrian Spatial Information Perception Method and Device Based on ZED Stereo Camera

technical field

The invention relates to the field of machine vision, in particular to a method and device for pedestrian spatial information perception based on a ZED stereo camera.

Background technique

In exhibition halls, museums and other public places, the use of intelligent guide robots instead of human guides can effectively save manpower. The navigation robot needs to have intelligent perception of its environment, and pedestrians are uncertain as a dynamic target in the scene. It is of great significance to intelligently perceive spatial information such as the spatial position and moving speed of pedestrians.

Vision is an important way for robots to obtain external information. First obtain the pedestrian detection frame based on the 2D image, and then obtain the spatial position information of the pedestrian according to the coordinate system conversion relationship and depth information or point cloud information. This method relies on the accuracy of the pedestrian detection frame, and pedestrians are flexible and their body postures are constantly changing. The traditional pedestrian detection bounding box will have a lot of background noise when the pedestrian's body changes, resulting in a large error in the subsequent 2D to 3D space conversion based on this bounding box, which will further affect the estimation of the pedestrian's spatial position and moving speed .

Contents of the invention

In order to solve the deficiencies of the prior art and achieve the purpose of improving the accuracy of identifying the spatial position of pedestrians and the moving speed of pedestrians, the present invention adopts the following technical solutions:

A pedestrian spatial information perception method based on a stereo camera, comprising the steps of:

Step S1: Obtain real-time image data of the stereo camera, including RGB image data and point cloud data;

Step S2: Detect the key points of the human body through the RGB image to obtain the two-dimensional key point information of the pedestrian, determine the trunk area of the upper body of the pedestrian according to the dynamic characteristics of the pedestrian, and combine the two-dimensional key point information of the trunk area of the upper body of the pedestrian to generate a pedestrian enclosing frame to surround the pedestrian frame as a pedestrian detection frame;

The generated pedestrian bounding box is expanded proportionally, and the expanded pedestrian bounding box is used as the pedestrian detection box. Since the generated pedestrian bounding box is the smallest pedestrian bounding box, considering that the minimum pedestrian bounding box is only the minimum pedestrian bounding box, the minimum pedestrian bounding box needs to be expanded; the area ratio of the expanded pedestrian bounding box to the minimum pedestrian bounding box is 1.2;

Step S3: According to the two-dimensional key point similarity feature and the pedestrian detection frame, perform multi-target tracking on the pedestrians under the continuous multi-frame images;

Step S4: For the continuously tracked pedestrians, according to their two-dimensional key point information and point cloud data, obtain pedestrian three-dimensional key point information, calculate the spatial position coordinates of the pedestrian relative to the stereo camera coordinate system in the current frame, and combine the frame interval Calculate the pedestrian's moving speed and generate real-time spatial information of the pedestrian relative to the stereo camera.

Further, in the step S2, the RGB image data is obtained, and the human body key point detection network is used to perform forward reasoning, and the key point heat map and part of the correlation domain are output, and the two-dimensional key points are extracted according to the key point heat map and part of the correlation domain. Carry out grouping, match the 2D key points belonging to the same pedestrian to the current pedestrian, and obtain the 2D key point coordinates of each pedestrian in the current image.

Further, the step S3 includes the following steps:

Step S3.1: According to the pedestrian detection frame, obtain pedestrian motion features; according to the two-dimensional key point similarity feature, obtain pedestrian appearance features;

Step S3.2: Obtain the measured status information of pedestrians at the current time t according to the pedestrian movement characteristics and pedestrian appearance characteristics;

Step S3.3: Data-associate historical trajectories with the measured status information of pedestrians at time t to obtain the ID of each pedestrian at time t; the purpose of data association is to combine the detection results at the current moment with historical trajectories through appearance, geometry, etc. Features are matched to determine the ID of each person detected in the current frame;

Step S3.4: Update the historical trajectory through the ID of each pedestrian at time t, so as to continuously track pedestrians.

Further, for the two-dimensional key point similarity feature in the step S3.1, the target key point similarity evaluation index OKS is used for similarity calculation, and a preset threshold is used to judge whether the two-dimensional key points are related.

Further, the data association in the step S3.3 is to associate two kinds of motion features and appearance features for each frame, and perform linear weighting to obtain the final association matrix. According to this association matrix, the Hungarian matching algorithm is used to obtain the inter-frame Pedestrian matching results.

Further, the step S4 includes the following steps:

Step S4.1: Screen according to the trunk area of the upper body of the pedestrian, combined with the confidence of the two-dimensional key points, obtain point cloud data according to the screened two-dimensional key points, and obtain the three-dimensional coordinate set of the key points of the pedestrian;

Step S4.2: For each pedestrian target, according to the 3D coordinate set of its key points, calculate the mean value of the 3D coordinates as the spatial position coordinates of the pedestrian target; and calculate the actual distance of the pedestrian relative to the stereo camera according to the Euclidean distance, according to the current moment The spatial position coordinates of pedestrians and the spatial position coordinates of pedestrians at the previous moment are used to calculate the moving distance of pedestrians relative to the stereo camera at the time interval between the current moment and the previous moment, and combined with the time used, the moving speed of pedestrians at the current moment is obtained.

Further, the pedestrian moving speed formula in the step S4.2 is as follows:

、

、

表示行人在当前帧下相对于立体相机的空间位置坐标，

、

、

分别表示在

时刻下相对立体相机的空间位置坐标，

， m表示帧间隔数，f为立体相机帧率。Among them, X, Y, and Z are the three-dimensional spatial position coordinates of pedestrians, i represents the currently tracked pedestrian ID, and t represents the time of the current frame,

,

,

Indicates the spatial position coordinates of the pedestrian relative to the stereo camera in the current frame,

,

,

respectively expressed in

The spatial position coordinates of the relative stereo camera at the moment,

, m represents the number of frame intervals, and f is the frame rate of the stereo camera.

A pedestrian spatial information perception device based on a stereo camera, comprising a real-time image data acquisition module, a pedestrian detection frame acquisition module, a multi-target tracking module, and a real-time spatial information generation module;

The real-time image data acquisition module acquires RGB image data and point cloud data through a stereo camera;

The pedestrian detection frame acquisition module detects the key points of the human body through the RGB image, obtains the two-dimensional key point information of the pedestrian, determines the trunk area of the upper body of the pedestrian according to the dynamic characteristics of the pedestrian, and combines the two-dimensional key point information of the trunk area of the upper body of the pedestrian to generate pedestrian surround Frame, the pedestrian bounding box is used as the pedestrian detection frame;

The multi-target tracking module, according to the two-dimensional key point similarity feature and the pedestrian detection frame, performs multi-target tracking to pedestrians under continuous multi-frame images;

The real-time spatial information generation module obtains the pedestrian's three-dimensional key point information for the continuously tracked pedestrian according to its two-dimensional key point information combined with point cloud data, and calculates the spatial position coordinates of the pedestrian relative to the stereo camera coordinate system under the current frame , and combine the frame interval to calculate the pedestrian moving speed, and generate the real-time spatial information of the pedestrian relative to the stereo camera.

A pedestrian spatial information perception method based on a ZED camera. According to the pedestrian spatial information perception method based on a stereo camera, a ZED binocular vision camera installed on a tour robot is used to collect real-time image data and transmit it to a cloud server for Pedestrian detection frame acquisition, multi-target tracking and real-time spatial information generation, including preprocessing of RGB images, including adjusting image size resize and encoding, to improve subsequent data transmission efficiency, preprocessed data is transmitted to the cloud through message middleware The server, the ZED binocular vision camera’s field of view range is set in conjunction with the trunk area of the upper body of the pedestrian. According to the dynamic characteristics of the pedestrian and the field of view of the ZED binocular vision camera on the guide robot, the trunk area of the upper body of the pedestrian is determined, and the cloud server compares the pedestrian’s The real-time spatial information from the ZED binocular vision camera is transmitted to the guide robot, and the guide robot controls the movement of the main body according to the real-time spatial information to complete the guide task.

A pedestrian spatial information perception device based on a ZED camera, including a cloud server and a ZED binocular vision camera arranged on a navigation robot, the cloud server includes a pedestrian detection frame acquisition module, a multi-target tracking module, and a real-time spatial information generation module ;

The ZED binocular vision camera acquires RGB image data and point cloud data in real time, and transmits them to the cloud server;

The pedestrian detection frame acquisition module detects the key points of the human body through the RGB image, obtains the two-dimensional key point information of the pedestrian, determines the trunk area of the upper body of the pedestrian according to the dynamic characteristics of the pedestrian, and combines the two-dimensional key point information of the trunk area of the upper body of the pedestrian to generate pedestrian surround Frame, the pedestrian bounding box is used as the pedestrian detection frame; the field of view of the ZED binocular vision camera is set in conjunction with the trunk area of the upper body of the pedestrian, and is determined according to the dynamic characteristics of pedestrians and the field of view of the ZED binocular vision camera on the navigation robot. Pedestrian upper body trunk area;

The multi-target tracking module, according to the two-dimensional key point similarity feature and the pedestrian detection frame, performs multi-target tracking to pedestrians under continuous multi-frame images;

The real-time spatial information generation module obtains the pedestrian's three-dimensional key point information for the continuously tracked pedestrian according to its two-dimensional key point information combined with point cloud data, and calculates the spatial position coordinates of the pedestrian relative to the stereo camera coordinate system under the current frame , and combine the frame interval to calculate the pedestrian's moving speed, and generate the real-time spatial information of the pedestrian relative to the stereo camera;

The navigation robot obtains the real-time spatial information of pedestrians relative to the ZED binocular vision camera from the cloud server, performs movement control of the main body, and completes the navigation task.

Advantage and beneficial effect of the present invention are:

The present invention uses a non-contact ZED binocular vision camera installed on the head of the navigation robot to intelligently perceive pedestrians in the scene; a lightweight human body key point detection network deployed in the cloud is used to obtain two-dimensional key point information, which can be fully utilized The powerful computing resources and storage resources of the cloud server can effectively solve the problem of insufficient computing power of the robot body; considering the movement characteristics of the human body and the range of vision of the navigation robot, the key points of the trunk area of the upper body of the human body are determined as the target area, which does not include the arm For the key points of the human body with large changes, the pedestrian detection frame generated based on this area is more stable, and the accuracy of the pedestrian spatial position and pedestrian moving speed calculated with the 3D coordinates of the pedestrian key points in this area is also higher.

Description of drawings

Fig. 1 is a flow chart of the pedestrian spatial information perception method based on stereo camera vision of the present invention.

Fig. 2 is a schematic diagram of a scenario-based application of pedestrian spatial information perception based on ZED vision for a navigation robot in an embodiment of the present invention.

Fig. 3a is a parameter diagram of key points of a human body in an embodiment of the present invention.

Fig. 3b is a schematic diagram of generating a bounding box of a pedestrian based on two-dimensional key points of the upper body of the pedestrian in an embodiment of the present invention.

Fig. 3c is a schematic diagram of three-dimensional key points in the embodiment of the present invention.

Fig. 4 is a visual flowchart of pedestrian spatial information perception based on ZED vision in an actual measurement scene in an embodiment of the present invention.

Fig. 5 is a comparison diagram of pedestrian distance measurement errors based on human body key points and traditional human body detection frames in the embodiment of the present invention.

Fig. 6 is a flow chart of the method for sensing spatial information of pedestrians based on a ZED camera of the present invention.

Fig. 7 is a schematic structural diagram of a pedestrian spatial information perception device based on a stereo camera in an embodiment of the present invention.

detailed description

Specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to illustrate and explain the present invention, and are not intended to limit the present invention.

As shown in Figure 1, the pedestrian spatial information perception method based on stereo camera vision includes the following steps:

Step S1: Obtain real-time image data of the stereo camera, including RGB image data and point cloud data;

In the embodiment of the present invention, as shown in Figure 2, the ZED binocular vision camera is installed on the head of the wheeled robot, about 1.2 meters above the ground level. Obtain the RGB data and point cloud data of the ZED camera (left eye), the frame rate is 30Hz, and the resolution of the RGB color image is 1280×720. Preprocess the RGB image, including resize and encoding, to improve the efficiency of subsequent data transmission. The size of resize is 456×256, which is consistent with the input of the subsequent human key point detection network. The preprocessed data is transmitted through the message middleware to the cloud server.

Step S2: Detect the key points of the human body through the RGB image to obtain the two-dimensional key point information of the pedestrian, determine the trunk area of the upper body of the pedestrian according to the dynamic characteristics of the pedestrian, and combine the two-dimensional key point information of the trunk area of the upper body of the pedestrian to generate a pedestrian enclosing frame to surround the pedestrian frame as a pedestrian detection frame;

Obtain RGB image data, use human body key point detection network to perform forward reasoning, output key point heat map and partial correlation domain, extract two-dimensional key points and group them according to key point heat map and partial correlation domain, and group two persons belonging to the same pedestrian The two-dimensional key points are matched to the current pedestrian, and the two-dimensional key point coordinates of each pedestrian in the current image are obtained.

In the embodiment of the present invention, the cloud server receives real-time data from the ZED camera collected by the robot, and inputs it to the deployed key point detection network of the human body for forward reasoning. The key point detection network of the human body adopts LightWeightOpenPose, and its backbone network adopts the improved The later MoblieNet can meet the real-time requirements, and obtain the two-dimensional key point information of pedestrians in the image through this framework. The input of the network is the decoded RGB data, the size is [1,3,256,456], and the output of the forward reasoning network is the key point heatmap (Heatmaps) and the partial association field (Part AffinityField, PAFs), the size is [1, 19, 32, 57], [1, 38, 32, 57], extract all key points according to Heatmaps and PAFs and group them, match the key points belonging to the same pedestrian to the current pedestrian, and get each pedestrian in the current image Assuming that the number of pedestrians detected in the current image is N, the output is N×[18,3], where 18 means that the number of key points of each pedestrian is 18, and 3 means the number of key points in the image coordinate system Horizontal axis x, vertical axis y and confidence level, the confidence range is 0-1;

Since pedestrians have the characteristics of posture changes, especially the limbs of the human body change greatly with the movement of the human body, and when the human body is close to the robot, the robot's field of vision can only see the upper part of the human body. Considering these two Influenced by factors, according to the dynamic characteristics of pedestrians and the range of vision of the navigation robot, the trunk area of the upper body of the pedestrian is determined as our human body target area; the key points of the human body contained in this area are: {0:NOSE, 1:NECK, 2:RIGHT_SHOULDER , 5:LEFT_SHOULDER, 8:RTGHT_HIP, 11:LEFT_HIP, 14:RIGHT_EYE, 15:LEFT_EYE, 16:RIGHT_EAR, 17:LEFT_EAR}, generate pedestrian bounding boxes according to the above two-dimensional key points, as shown in Figure 3a "Human Key Points" As shown; according to the two-dimensional key points in this area, the boundingRect function of OpenCV is used to generate the minimum bounding box, as shown in P1 in the 2D key point schematic diagram in Figure 3b, which is represented by a dotted line; considering that this bounding box is only the minimum of the pedestrian skeleton The bounding box needs to be appropriately expanded. The expanded bounding box is shown in P2, which is represented by a solid line. The area ratio of P2 to P1 is 1.2. This bounding box is used as the pedestrian detection box in the subsequent pedestrian tracking algorithm.

Step S3: According to the two-dimensional key point similarity feature and the pedestrian detection frame, perform multi-target tracking on the pedestrians under the continuous multi-frame images, including the following steps:

Step S3.1: According to the pedestrian detection frame, obtain pedestrian motion features; according to the two-dimensional key point similarity feature, obtain the pedestrian appearance feature; two-dimensional key point similarity feature, use the target key point similarity evaluation index OKS to perform similarity Calculate, and judge whether the two-dimensional key points are related through the preset threshold;

Step S3.2: Obtain the measured status information of pedestrians at the current time t according to the pedestrian movement characteristics and pedestrian appearance characteristics;

Step S3.3: Data-associate historical trajectories with the measured status information of pedestrians at time t to obtain the ID of each pedestrian at time t; data association is to associate motion features and appearance features of each frame, and carry out linear weighting , get the final correlation matrix, and use the Hungarian matching algorithm to get the pedestrian matching results between frames according to the correlation matrix;

Step S3.4: Update the historical trajectory through the ID of each pedestrian at time t.

In the embodiment of the present invention, for the pedestrian targets in each frame of RGB images transmitted to the cloud, we assign unique identity information (ID) to them through a multi-target tracking method for inter-frame continuous tracking. The multi-target pedestrian tracking method adopts the principle of DeepSort algorithm. Obtain pedestrian motion features based on the pedestrian detection frame generated in step S2, obtain pedestrian appearance features based on two-dimensional key point similarity features, use OKS (object keypoint similarity) for similarity calculation, and use the set threshold to determine whether the association is successful or not The judgment of the two kinds of characteristic information can be combined to obtain the measured state information of pedestrians at the current time t; then the historical trajectory and the state results at the time t are associated with data, and the two associations of motion characteristics and appearance characteristics are combined to carry out linear weighting as the final The correlation matrix, according to the correlation matrix, the Hungarian matching algorithm is used to obtain the matching result, and the ID of each pedestrian target at time t can be obtained, and finally the historical trajectory is updated with the result at time t, so as to obtain the ID of the pedestrian.

The purpose of data association is to match the detection results at the current moment with the historical trajectory through appearance, geometry and other features, so as to determine the ID of each person detected in the current frame, and linearly weight the two associations of motion features and appearance features as the final According to the incidence matrix, the Hungarian matching algorithm is used to obtain the matching result.

Step S4: For the continuously tracked pedestrians, according to their two-dimensional key point information and point cloud data, obtain pedestrian three-dimensional key point information, calculate the spatial position coordinates of the pedestrian relative to the stereo camera coordinate system in the current frame, and combine the frame interval Calculating the moving speed of pedestrians and generating real-time spatial information of pedestrians relative to the stereo camera, including the following steps:

Step S4.1: Screen according to the trunk area of the upper body of the pedestrian, combined with the confidence of the two-dimensional key points, obtain point cloud data according to the screened two-dimensional key points, and obtain the three-dimensional coordinate set of the key points of the pedestrian;

In the embodiment of the present invention, according to the trunk area of the upper body of the pedestrian determined in step 2, the key points are screened based on the confidence of the key points, and the key points with a confidence greater than 0.6 participate in the next calculation, and then retrieve the corresponding points according to the obtained ZED point cloud data. 3D key points, to obtain the 3D coordinate set of the corresponding pedestrian key points;

)},其中k为关键点个数，取值为k=0,1,2,5,8,11,14,15,16,17，

为相应的置信度，如图3b中“2D关键点示意图”实心原点所示。结合ZED相机采集的点云信息进行检索，我们根据行人主干区域并结合二维关键点置信度可以获取行人相应的三维坐标，行人所有二维关键点对应的三维坐标如图3c中“3D关键点示意图”所示，包含主干区域的人体关键点的三维坐标集合为

)}，如图3c中三角形图标所示，该坐标系是以ZED相机左目摄像头为基准，通过相机参数COORDINATE_SYSTEM.LEFT_HANDED_Y_UP 进行设置，坐标系及XYZ轴的方向如图2所示；The human body key point detection network outputs the 2D key point coordinate set of pedestrians in the image coordinate system as

)}, where k is the number of key points, the value is k=0,1,2,5,8,11,14,15,16,17,

is the corresponding confidence, as shown by the solid origin in the "2D key point schematic diagram" in Figure 3b. Combined with the point cloud information collected by the ZED camera for retrieval, we can obtain the corresponding 3D coordinates of pedestrians according to the pedestrian backbone area and the confidence of 2D key points. The 3D coordinates corresponding to all 2D key points of pedestrians are shown in "3D key points As shown in the schematic diagram, the three-dimensional coordinate set of the key points of the human body including the backbone area is

)}, as shown by the triangle icon in Figure 3c, the coordinate system is based on the left-eye camera of the ZED camera, and is set through the camera parameter COORDINATE_SYSTEM.LEFT_HANDED_Y_UP, and the coordinate system and the direction of the XYZ axis are shown in Figure 2;

Step S4.2: For each pedestrian target, according to the 3D coordinate set of its key points, calculate the mean value of the 3D coordinates as the spatial position coordinates of the pedestrian target; and calculate the actual distance of the pedestrian relative to the stereo camera according to the Euclidean distance, according to the current moment The spatial position coordinates of pedestrians and the spatial position coordinates of pedestrians at the previous moment are used to calculate the moving distance of pedestrians relative to the stereo camera under the time interval between the current moment and the previous moment, and combined with the time used, the pedestrian moving speed at the current moment is obtained;

，其中N=10，表示每一个行人上半身区域的关键点的数量； i表示当前跟踪到的行人ID，设当前帧的时间为t，则行人在当前帧下相对于机器人的空间位置为

，在

时刻相对机器人的空间位置为

，则行人的移动速度为

，其中，

， f为ZED相机帧率，m表示帧间隔数，综合考虑算法总耗时，m的取值为

；In the embodiment of the present invention, according to the acquired three-dimensional coordinate set of key points of the pedestrian's human body, the average value is calculated as the spatial three-dimensional coordinate position of the target pedestrian, namely

, where N=10, represents the number of key points in the upper body area of each pedestrian; i represents the currently tracked pedestrian ID, if the time of the current frame is t, then the spatial position of the pedestrian relative to the robot in the current frame is

,exist

The spatial position relative to the robot at all times is

, then the moving speed of the pedestrian is

,in,

, f is the frame rate of the ZED camera, m is the number of frame intervals, considering the total time consumption of the algorithm, the value of m is

;

，其中（X,Y,Z）为行人的三维空间坐标，如图5所示，深色线表示本发明的方法，浅色线表示对比方法，可以看出，在行人运动过程中，基于人体检测框的方法会随着人体姿态的变化引入较大噪声，而我们的方法抗干扰性更好，能够获取更精准的行人定位信息。As shown in Figure 4, the ZED camera obtains RBG data and point cloud data, and then obtains the two-dimensional coordinates of the key points of the human body according to the key point detection algorithm of the human body, and finally combines the two-dimensional key point information and the corresponding point cloud information to obtain the key points of the human body three-dimensional coordinates. In order to verify the effectiveness of the method of the present invention, about 1000 frames of images were collected, and the distance from the pedestrian to the camera was calculated using the human body detection frame based on YOLOV3 combined with the point cloud information and the distance from the pedestrian to the camera was calculated based on the key points of the main body area of the present invention , comparative tests and statistics were carried out respectively, and the distance from the pedestrian to the camera was calculated using the Euclidean distance formula:

, where (X, Y, Z) are the three-dimensional space coordinates of pedestrians, as shown in Figure 5, the dark line represents the method of the present invention, and the light color line represents the comparison method, it can be seen that during the movement of pedestrians, based on the human body The detection frame method will introduce larger noise with the change of human body posture, while our method has better anti-interference ability and can obtain more accurate pedestrian positioning information.

A pedestrian spatial information perception device based on a stereo camera, used to realize the pedestrian spatial information perception method based on a stereo camera, including a real-time image data acquisition module, a pedestrian detection frame acquisition module, a multi-target tracking module, and real-time spatial information generation module;

The real-time image data acquisition module acquires RGB image data and point cloud data through a stereo camera;

The pedestrian detection frame acquisition module detects the key points of the human body through the RGB image, obtains the two-dimensional key point information of the pedestrian, determines the trunk area of the upper body of the pedestrian according to the dynamic characteristics of the pedestrian, and combines the two-dimensional key point information of the trunk area of the upper body of the pedestrian to generate pedestrian surround Frame, the pedestrian bounding box is used as the pedestrian detection frame;

The multi-target tracking module, according to the two-dimensional key point similarity feature and the pedestrian detection frame, performs multi-target tracking to pedestrians under continuous multi-frame images;

The real-time spatial information generation module obtains the pedestrian's three-dimensional key point information for the continuously tracked pedestrian according to its two-dimensional key point information combined with point cloud data, and calculates the spatial position coordinates of the pedestrian relative to the stereo camera coordinate system under the current frame , and combine the frame interval to calculate the pedestrian moving speed, and generate the real-time spatial information of the pedestrian relative to the stereo camera.

The implementation manners of this part of the content are similar to the implementation manners of the foregoing method embodiments, and will not be repeated here.

As shown in Figure 6, a pedestrian spatial information perception method based on a ZED camera, according to the pedestrian spatial information perception method based on a stereo camera, adopts a ZED binocular vision camera installed on a tour robot to collect real-time image data, And transmit it to the cloud server for pedestrian detection frame acquisition, multi-target tracking and real-time spatial information generation. Among them, the field of view of the ZED binocular vision camera is set in conjunction with the trunk area of the upper body of the pedestrian. The field of view of the binocular vision camera determines the trunk area of the upper body of the pedestrian, and the cloud server transmits the real-time spatial information of the pedestrian relative to the ZED binocular vision camera to the guide robot, and the guide robot performs the movement control of the main body according to the real-time spatial information , complete navigation tasks such as autonomous follow and obstacle avoidance.

Specifically, include the following steps:

Step S101: Obtain the real-time image data of the ZED binocular vision camera on the tour robot, including RGB image data and point cloud data, and transmit them to the cloud server;

Step S102: The cloud server detects the key points of the human body through the RGB image, obtains the two-dimensional key point information of the pedestrian, determines the trunk area of the upper body of the pedestrian according to the dynamic characteristics of the pedestrian, and generates the bounding box of the pedestrian by combining the two-dimensional key point information of the trunk area of the upper body of the pedestrian. The pedestrian bounding box is used as the pedestrian detection frame; among them, the field of view of the ZED binocular vision camera is set in conjunction with the trunk area of the upper body of the pedestrian, and the upper body of the pedestrian is determined according to the dynamic characteristics of the pedestrian and the field of view of the ZED binocular vision camera on the navigation robot backbone area;

Step S103: The cloud server performs multi-target tracking on the pedestrians under the continuous multi-frame images according to the two-dimensional key point similarity feature and the pedestrian detection frame;

Step S104: For the continuously tracked pedestrian, the cloud server obtains the 3D key point information of the pedestrian according to the 2D key point information combined with the point cloud data, and calculates the spatial position coordinates of the pedestrian relative to the ZED binocular vision camera coordinate system in the current frame , and combine the frame interval to calculate the pedestrian's moving speed, and generate the real-time spatial information of the pedestrian relative to the ZED binocular vision camera;

Step S105: The cloud server transmits the real-time spatial information of pedestrians relative to the ZED binocular vision camera to the guide robot, and the guide robot controls the movement of the main body according to the real-time spatial information to complete the guide task.

In the embodiment of the present invention, the cloud stores the pedestrian ID, the spatial position of the pedestrian, and the moving speed of the pedestrian at the current moment in the database of the cloud server, and stores them in the form of a message queue. The basic format is data={'key1':'value1 ', 'key2': 'value2', 'key3': 'value3'}, where key1, key2, and key3 are 'p_ID', 'p_Pos3D', and 'p_Speed' respectively, indicating pedestrian ID, pedestrian three-dimensional space coordinates, pedestrian movement Velocity, the corresponding value is the calculated identity ID of the specific pedestrian i, the three-dimensional coordinates of the pedestrian and the moving speed of the pedestrian. According to the request instructions from the robot side, the cloud sends the data to the robot side in real time through the RocketMQ message middleware. When the distance is less than the safety distance, the robot stops moving to avoid collisions, and adjusts its own moving speed according to the real-time moving speed of pedestrians, so as to realize intelligent navigation tasks such as autonomous following and obstacle avoidance.

A pedestrian spatial information perception device based on a ZED camera, used to realize a pedestrian spatial information perception method based on a ZED camera, including a cloud server and a ZED binocular vision camera arranged on a navigation robot, the cloud server includes a pedestrian detection frame Acquisition module, multi-target tracking module, real-time spatial information generation module;

The ZED binocular vision camera acquires RGB image data and point cloud data in real time, and transmits them to the cloud server;

The pedestrian detection frame acquisition module detects the key points of the human body through the RGB image, obtains the two-dimensional key point information of the pedestrian, determines the trunk area of the upper body of the pedestrian according to the dynamic characteristics of the pedestrian, and combines the two-dimensional key point information of the trunk area of the upper body of the pedestrian to generate pedestrian surround Frame, the pedestrian bounding box is used as the pedestrian detection frame; the field of view of the ZED binocular vision camera is set in conjunction with the trunk area of the upper body of the pedestrian, and is determined according to the dynamic characteristics of pedestrians and the field of view of the ZED binocular vision camera on the navigation robot. Pedestrian upper body trunk area;

The multi-target tracking module, according to the two-dimensional key point similarity feature and the pedestrian detection frame, performs multi-target tracking to pedestrians under continuous multi-frame images;

The real-time spatial information generation module obtains the pedestrian's three-dimensional key point information for the continuously tracked pedestrian according to its two-dimensional key point information combined with point cloud data, and calculates the spatial position coordinates of the pedestrian relative to the stereo camera coordinate system under the current frame , and combine the frame interval to calculate the pedestrian's moving speed, and generate the real-time spatial information of the pedestrian relative to the stereo camera;

The navigation robot obtains the real-time spatial information of pedestrians relative to the ZED binocular vision camera from the cloud server, performs movement control of the main body, and completes the navigation task.

The implementation manners of this part of the content are similar to the implementation manners of the foregoing method embodiments, and will not be repeated here.

Corresponding to the foregoing embodiment of the pedestrian spatial information perception method based on stereo camera vision, the present invention also provides an embodiment of a pedestrian spatial information perception device based on stereo camera vision.

Referring to FIG. 7 , the pedestrian spatial information perception device based on stereo camera vision provided by an embodiment of the present invention includes a memory and one or more processors, executable codes are stored in the memory, and the one or more processors execute the described When the code is executable, it is used to implement the pedestrian spatial information perception method based on stereo camera vision in the above embodiments.

The embodiment of the pedestrian spatial information perception device based on stereo camera vision of the present invention can be applied to any device with data processing capability, and any device with data processing capability can be a device or device such as a computer. The device embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory for operation by the processor of any device capable of data processing. From the hardware level, as shown in Figure 7, it is a hardware structure diagram of any device with data processing capabilities where the pedestrian spatial information perception device based on stereo camera vision is located, except for the processor and memory shown in Figure 7 In addition to , network interface, and non-volatile memory, any device with data processing capability where the device in the embodiment is located usually can also include other hardware according to the actual function of any device with data processing capability. repeat.

For the implementation process of the functions and effects of each unit in the above device, please refer to the implementation process of the corresponding steps in the above method for details, and will not be repeated here.

As for the device embodiment, since it basically corresponds to the method embodiment, for related parts, please refer to the part description of the method embodiment. The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. It can be understood and implemented by those skilled in the art without creative effort.

An embodiment of the present invention also provides a computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, the spatial information perception method for pedestrians based on stereo camera vision in the above-mentioned embodiments is implemented.

The computer-readable storage medium may be an internal storage unit of any device capable of data processing described in any of the foregoing embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device capable of data processing, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, or a flash memory card equipped on the device. (Flash Card), etc. Further, the computer-readable storage medium may also include both an internal storage unit of any device capable of data processing and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device capable of data processing, and may also be used to temporarily store data that has been output or will be output.

The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be described in the foregoing embodiments Modifications to the technical solutions, or equivalent replacement of some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. a pedestrian spatial information perception method based on stereo camera, it is characterized in that comprising the steps: Step S1: Obtain real-time image data of the stereo camera, including RGB image data and point cloud data; Step S2: Detect the key points of the human body through the RGB image to obtain the two-dimensional key point information of the pedestrian, determine the trunk area of the upper body of the pedestrian according to the dynamic characteristics of the pedestrian, and combine the two-dimensional key point information of the trunk area of the upper body of the pedestrian to generate a pedestrian enclosing frame to surround the pedestrian frame as a pedestrian detection frame; Step S3: According to the two-dimensional key point similarity feature and the pedestrian detection frame, perform multi-target tracking on the pedestrians under the continuous multi-frame images; Step S4: For the continuously tracked pedestrians, according to their two-dimensional key point information and point cloud data, obtain pedestrian three-dimensional key point information, calculate the spatial position coordinates of the pedestrian relative to the stereo camera coordinate system in the current frame, and combine the frame interval Calculate the pedestrian's moving speed and generate real-time spatial information of the pedestrian relative to the stereo camera. 2. The pedestrian spatial information perception method based on a stereo camera according to claim 1, characterized in that: in the step S2, RGB image data is obtained, and a human body key point detection network is used to perform forward reasoning, and a key point heat map is output and part of the correlation domain, extract and group two-dimensional key points according to the key point heat map and part of the correlation domain, match the two-dimensional key points belonging to the same pedestrian to the current pedestrian, and obtain the two-dimensional key points of each pedestrian in the current image coordinate. 3. The pedestrian spatial information perception method based on stereo camera according to claim 1, characterized in that: said step S3 comprises the following steps: Step S3.1: According to the pedestrian detection frame, obtain pedestrian motion features; according to the two-dimensional key point similarity feature, obtain pedestrian appearance features; Step S3.2: Obtain the measured status information of pedestrians at the current time t according to the pedestrian movement characteristics and pedestrian appearance characteristics; Step S3.3: Data association of the historical trajectory with the measured status information of pedestrians at time t, to obtain the ID of each pedestrian at time t; Step S3.4: Update the historical trajectory through the ID of each pedestrian at time t. 4. The pedestrian spatial information perception method based on a stereo camera according to claim 3, characterized in that: the two-dimensional key point similarity feature in the step S3.1 uses the target key point similarity evaluation index OKS to perform similar degree calculation, and judge whether the two-dimensional key points are related through the preset threshold. 5. The pedestrian spatial information perception method based on a stereo camera according to claim 3, characterized in that: the data association in the step S3.3 is to carry out two associations of motion features and appearance features for each frame, and perform linear Weighted to get the final correlation matrix, and according to the correlation matrix, the Hungarian matching algorithm is used to obtain the matching results of pedestrians between frames. 6. The pedestrian spatial information perception method based on stereo camera according to claim 1, characterized in that: said step S4 comprises the following steps: Step S4.1: Screen according to the trunk area of the upper body of the pedestrian, combined with the confidence of the two-dimensional key points, obtain point cloud data according to the screened two-dimensional key points, and obtain the three-dimensional coordinate set of the key points of the pedestrian; Step S4.2: For each pedestrian target, according to the 3D coordinate set of its key points, calculate the mean value of the 3D coordinates as the spatial position coordinates of the pedestrian target; and calculate the actual distance of the pedestrian relative to the stereo camera according to the Euclidean distance, according to the current moment The spatial position coordinates of pedestrians and the spatial position coordinates of pedestrians at the previous moment are used to calculate the moving distance of pedestrians relative to the stereo camera at the time interval between the current moment and the previous moment, and combined with the time used, the moving speed of pedestrians at the current moment is obtained. 7. The pedestrian spatial information perception method based on a stereo camera according to claim 6, characterized in that: the pedestrian moving speed formula in the step S4.2 is as follows:

Among them, X, Y, and Z are the three-dimensional spatial position coordinates of pedestrians, i represents the currently tracked pedestrian ID, and t represents the time of the current frame,

,

,

Indicates the spatial position coordinates of the pedestrian relative to the stereo camera in the current frame,

,

,

respectively expressed in

The spatial position coordinates of the relative stereo camera at the moment,

, m represents the number of frame intervals, and f is the frame rate of the stereo camera. 8. A pedestrian space information perception device based on a stereo camera, used to realize the pedestrian space information perception method based on a stereo camera according to any one of claims 1-7, comprising a real-time image data acquisition module, a pedestrian detection frame acquisition Module, multi-target tracking module, real-time spatial information generation module, is characterized in that: The real-time image data acquisition module acquires RGB image data and point cloud data through a stereo camera; The pedestrian detection frame acquisition module detects the key points of the human body through the RGB image, obtains the two-dimensional key point information of the pedestrian, determines the trunk area of the upper body of the pedestrian according to the dynamic characteristics of the pedestrian, and combines the two-dimensional key point information of the trunk area of the upper body of the pedestrian to generate pedestrian surround Frame, the pedestrian bounding box is used as the pedestrian detection frame; The multi-target tracking module, according to the two-dimensional key point similarity feature and the pedestrian detection frame, performs multi-target tracking to pedestrians under continuous multi-frame images; The real-time spatial information generation module obtains the pedestrian's three-dimensional key point information for the continuously tracked pedestrian according to its two-dimensional key point information combined with point cloud data, and calculates the spatial position coordinates of the pedestrian relative to the stereo camera coordinate system under the current frame , and combine the frame interval to calculate the pedestrian moving speed, and generate the real-time spatial information of the pedestrian relative to the stereo camera. 9. A pedestrian spatial information perception method based on a ZED camera, characterized in that according to the pedestrian spatial information perception method based on a stereo camera according to any one of claims 1-7, the ZED dual-axis sensor installed on the navigation robot is adopted. The eye vision camera collects real-time image data and transmits it to the cloud server for pedestrian detection frame acquisition, multi-target tracking, and real-time spatial information generation. Among them, the field of view of the ZED binocular vision camera is set in conjunction with the trunk area of the upper body of the pedestrian. Features and the field of view of the ZED binocular vision camera on the guide robot, determine the trunk area of the upper body of the pedestrian, and the cloud server transmits the real-time spatial information of the pedestrian relative to the ZED binocular vision camera to the guide robot. Real-time spatial information is used to control the movement of the main body and complete the navigation task. 10. A pedestrian spatial information perception device based on a ZED camera, used to realize the pedestrian spatial information perception method based on a ZED camera according to claim 9, comprising a cloud server and a ZED binocular vision camera arranged on a navigation robot, It is characterized in that: the cloud server includes a pedestrian detection frame acquisition module, a multi-target tracking module, and a real-time spatial information generation module; The ZED binocular vision camera acquires RGB image data and point cloud data in real time, and transmits them to the cloud server; The pedestrian detection frame acquisition module detects the key points of the human body through the RGB image, obtains the two-dimensional key point information of the pedestrian, determines the trunk area of the upper body of the pedestrian according to the dynamic characteristics of the pedestrian, and combines the two-dimensional key point information of the trunk area of the upper body of the pedestrian to generate pedestrian surround Frame, the pedestrian bounding box is used as the pedestrian detection frame; the field of view of the ZED binocular vision camera is set in conjunction with the trunk area of the upper body of the pedestrian, and is determined according to the dynamic characteristics of pedestrians and the field of view of the ZED binocular vision camera on the navigation robot. Pedestrian upper body trunk area; The multi-target tracking module, according to the two-dimensional key point similarity feature and the pedestrian detection frame, performs multi-target tracking to pedestrians under continuous multi-frame images; The real-time spatial information generation module obtains the pedestrian's three-dimensional key point information for the continuously tracked pedestrian according to its two-dimensional key point information combined with point cloud data, and calculates the spatial position coordinates of the pedestrian relative to the stereo camera coordinate system under the current frame , and combine the frame interval to calculate the pedestrian's moving speed, and generate the real-time spatial information of the pedestrian relative to the stereo camera; The navigation robot obtains the real-time spatial information of pedestrians relative to the ZED binocular vision camera from the cloud server, performs movement control of the main body, and completes the navigation task.

Images