WO2019196313A1 - Robot walking obstacle detection method and apparatus, computer device, and storage medium - Google Patents

Abstract

A robot walking obstacle detection method and apparatus. The method comprises: acquiring an image by means of a camera, and using an image human body detection algorithm to obtain human body image coordinates (u, v) of a human body joint point under an image coordinate system (S10); converting the human body image coordinates (u, v) under the image coordinate system into image physical coordinates (x, y) under an image physical coordinate system (S20); determining a vertical deflection angle α and a horizontal deflection angle β of a laser emitter irradiating the key human body joint point according to x and y, and starting the laser emitter to emit laser to measure a distance Z' from the human body to the robot (S30); calculating coordinates (X, Y, Z) of the human body joint point in the coordinate system of the camera according to the x, y and Z' (S40); calculating, according to the coordinates (X₁, Y₁, Z₁) and (X₂, Y₂, Z₂) of the corresponding human body joint points of any two persons in the coordinate system of the camera, a distance L between the corresponding human body joint points of the any two persons (S50); and determining a walking direction according to the distance L between the corresponding human body joint points of the any two persons (S60). Accurate pedestrian obstacle avoidance can be achieved. Also disclosed are a computer device and a storage medium.

Description

Robot walking obstacle detection method, device, computer equipment and storage medium

This application claims the priority of the Chinese Patent Application entitled "Robot Walking Obstacle Detection Method, Apparatus, Computer Equipment and Storage Medium" submitted by the Chinese Patent Office on April 10, 2018, application number 201810314149.5, the entire contents of which are The citations are incorporated herein by reference.

Technical field

The present application relates to the field of robot obstacle avoidance technology, and in particular, to a robot walking obstacle detection method and apparatus, and a computer device and a storage medium storing computer readable instructions.

Background technique

In the navigation process of the robot, the obstacle detection of the robot is the key factor for the successful navigation movement of the robot. Compared with the stationary object, the pedestrian in the movement adds many difficulties to the obstacle avoidance of the robot. The inventor realized that in the current robot obstacle detection method, there are many problems, such as limited range of ultrasonic detection, and it is difficult to comprehensively cover the three-dimensional space even if a large number is loaded; although the laser detection has high precision, the same problem exists. The coverage is detected by the depth camera, but due to the huge data processing and visual depth of field limitation, there is also a problem that the obstacle detection accuracy is not high. And these methods are mainly for stationary objects, but the pedestrians are equated with other obstacle detection, which is more passive when formulating strategies to avoid pedestrians, reducing the efficiency of effective obstacle avoidance.

Summary of the invention

The purpose of the present application is to solve at least one of the above technical drawbacks, in particular, technical defects that are difficult to avoid obstacles.

The present application provides a robot walking obstacle detecting method, the robot having a camera, a laser emitter, and a laser receiver, the method comprising the following steps: Step S10: acquiring an image through a camera, applying an image human body detection algorithm to obtain a human joint point Human body image coordinates (u, v) in the image coordinate system; step S20: converting human body image coordinates (u, v) in the image coordinate system into image physical coordinates (x, y) in the image physical coordinate system; step S30 : determining, according to x, y, the vertical deflection angle α and the horizontal deflection angle β of the laser emitter irradiating the key joint points of the human body, starting the laser emitter to emit laser light to measure the distance Z′ of the human body to the robot; step S40: according to x, y, Z 'calculate the coordinates (X, Y, Z) of the human joint point in the camera coordinate system; step S50: according to the coordinates (X ₁ , Y ₁ , Z ₁ ) of the camera coordinate system of the corresponding human joint points of any two people and ( X ₂ , Y ₂ , Z ₂ ) Calculate the distance L between the respective human joint points of any two persons; Step S60: Determine the walking direction according to the distance L between the respective human joint points of any two persons.

The present application also provides a robot walking obstacle detecting device, the robot having a camera, a laser emitter, and a laser receiver, the device comprising: an image coordinate acquiring module, configured to acquire an image through a camera, and apply an image human body detecting algorithm to obtain a human body The human body image coordinates (u, v) of the off node in the image coordinate system; the physical coordinate acquisition module is used to convert the human body image coordinates (u, v) in the image coordinate system into the image physical coordinates in the image physical coordinate system ( x, y); human body distance acquisition module for determining the vertical deflection angle α and the horizontal deflection angle β of the laser emitter to illuminate the key joint points of the human body according to x, y, and starting the laser emitter to emit laser light to measure the distance from the human body to the robot Z a camera coordinate acquisition module for calculating coordinates (X, Y, Z) of the human joint point in the camera coordinate system according to x, y, Z'; a crowd spacing acquisition module for corresponding human joints according to any two people coordinates in the camera coordinate system _{(X 1, Y 1, Z} 1) and _{(X 2, Y 2, Z} 2) calculate the distance L between any two respective joints of the human body; row Direction determining means for determining the traveling direction L the distance between the respective points of any two human joints.

The application also provides a computer device comprising a memory and a processor, the memory storing computer readable instructions, the computer readable instructions being executed by the processor, causing the processor to perform a robot walking The obstacle detecting method includes the following steps: Step S10: acquiring an image by a camera, and applying an image human body detecting algorithm to obtain human body image coordinates (u, v) of the human joint point in the image coordinate system; Step S20: Converting the human body image coordinates (u, v) in the image coordinate system to the image physical coordinates (x, y) in the image physical coordinate system; Step S30: determining the vertical deflection of the laser emitter illuminating the key human joint points according to x, y The angle α and the horizontal deflection angle β start the laser emitter to emit laser light to measure the distance Z′ of the human body to the robot; step S40: calculate the coordinates of the human joint point in the camera coordinate system according to x, y, Z′ (X, Y, Z); Step S50: Calculate any two coordinates (X ₁ , Y ₁ , Z ₁ ) and (X ₂ , Y ₂ , Z ₂ ) of the camera coordinate system according to the corresponding human joint points of any two persons The distance L between the corresponding human joint points of the person; Step S60: determining the walking direction according to the distance L between the respective human joint points of any two persons.

The present application also provides a non-volatile storage medium storing computer readable instructions that, when executed by one or more processors, cause one or more processors to perform a robotic walking obstacle detection The method for detecting a walking obstacle of a robot includes the following steps: Step S10: acquiring an image by a camera, applying an image human body detection algorithm to obtain body image coordinates (u, v) of a human joint point in an image coordinate system; and step S20: placing the image The human body image coordinates (u, v) in the coordinate system are converted into image physical coordinates (x, y) in the image physical coordinate system; step S30: determining the vertical deflection angle α of the laser emitter irradiating the key human joint points according to x, y And the horizontal deflection angle β, the laser emitter is activated to emit laser light to measure the distance Z' of the human body to the robot; step S40: calculating the coordinates (X, Y, Z) of the human joint point in the camera coordinate system according to x, y, Z' Step S50: Calculate the corresponding human body of any two persons according to the coordinates (X ₁ , Y ₁ , Z ₁ ) and (X ₂ , Y ₂ , Z ₂ ) of the camera coordinate system of the corresponding human joint points of any two persons. The distance L between the nodes; step S60: determining the walking direction according to the distance L between the respective human joint points of any two persons.

The above-mentioned robot walking obstacle detecting method, device, computer equipment and storage medium, by combining image recognition and laser ranging to determine the actual coordinates of the human body (in the coordinate of the camera coordinate system), the real-time position of the human body can be accurately obtained, thereby determining any two people. The distance between the two is used to judge the walking path in real time, and accurate pedestrian obstacle detection is realized.

DRAWINGS

1 is a schematic diagram showing the internal structure of a computer device in an embodiment;

2 is a schematic flow chart of a method for detecting a walking obstacle of a robot according to an embodiment;

3 is a schematic diagram of a human body detection process of an embodiment;

4 is a diagram showing an example of human body detection of an embodiment;

FIG. 5 is a schematic diagram of a module of a robot walking obstacle detecting device according to an embodiment.

detailed description

FIG. 1 is a schematic diagram showing the internal structure of a computer device in an embodiment. As shown in FIG. 1, the computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected by a system bus. The non-volatile storage medium of the computer device stores an operating system, a database, and computer readable instructions. The database may store a sequence of control information. When the computer readable instructions are executed by the processor, the processor may implement a processor. A robot walking obstacle detection method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. Computer readable instructions may be stored in the memory of the computer device, the computer readable instructions being executable by the processor to cause the processor to perform a robotic walking obstacle detection method. The network interface of the computer device is used to communicate with the terminal connection. It will be understood by those skilled in the art that the structure shown in FIG. 1 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation of the computer device to which the solution of the present application is applied. The specific computer device may It includes more or fewer components than those shown in the figures, or some components are combined, or have different component arrangements.

The robot walking obstacle detecting method described below can be applied to an intelligent robot such as a customer service robot, a sweeping robot, and the like.

FIG. 2 is a schematic flow chart of a method for detecting a walking obstacle of a robot according to an embodiment.

A robot walking obstacle detecting method, the robot having a camera, a laser emitter, and a laser receiver, the method comprising the following steps:

Step S10: acquiring an image by using a camera, and applying an image human body detection algorithm to obtain human body image coordinates (u, v) of the human joint point in the image coordinate system.

The image human body detection algorithm refers to a technique for recognizing a human body through image recognition technology. There are three methods for image human body detection, which are global feature based methods, human body part based methods, and stereo vision based methods.

Global feature based approach

This type of method is currently the more mainstream human detection method. It mainly uses various static features of images such as edge features, shape features, statistical features or transform features to describe the human body. Representative features include Haar wavelet features, HOG features, and Edgelet. Features, Shapelet features, and outline template features.

(1) Method based on Haar wavelet feature

Papageorgiou and Poggio first proposed the concept of Harr wavelet, Viola et al. introduced the concept of integral graph, accelerated the extraction speed of Harr features, and applied the method to human detection, combined with human motion and appearance mode to construct human detection system. The better detection effect lays the foundation for the development of human detection technology.

(2) HOG-based method

Applying the concept of Histogram of Oriented Gradients (HOG) to human detection, achieving nearly 100% detection success rate on the MIT human body database; on the INRIA human body database containing changes in perspective, illumination and background , also achieved a test success rate of about 90%.

(3) Method based on edgelet feature

The "Edgelet" feature, which is a short line or curve segment, is applied to human detection of a single image of a complex scene, yielding a detection rate of approximately 92% on the CAVIAR database.

(4) Method based on Shapelet feature

Features can be automatically obtained using machine learning methods, namely Shapelet features. The algorithm first extracts the gradient information of different directions from the training samples, and then uses the AdaBoost algorithm to train, thus obtaining the Shapelet feature.

(5) Method based on contour template

The method refers to constructing a template by using information such as edge contour, texture and gray scale of the target object in the image, and detecting the target by template matching.

Method based on human body parts

The basic idea of this kind of method is to divide the human body into several components, and then separately test each part of the image, and finally integrate the detection results according to a certain constraint relationship, and finally determine whether there is a human body.

Stereo vision based method

This type of method refers to image acquisition by two or more cameras, and then analyzing the three-dimensional information of the target in the image to identify the human body.

In this embodiment, the image human body detection algorithm is not specifically limited as long as the human body and its joint points can be identified. Human joint points refer to key nodes of the human body such as the head, neck, shoulders, upper arms, elbows, wrists, waist, chest, thighs, knees, calves, ankles, etc. There are multiple.

In the present embodiment, human body detection can be performed using a method based on a human body part. OpenPose is an open source library written in C++ using OpenCV and Caffe for real-time detection of multi-threaded multi-person key points. The main detection of human body detection consists of two parts: human joint point detection and joint point correlation detection. Among them, the joint point detection uses a Convolutional Pose Machine. The attitude machine can infer the position of the key points of the human body by adding the convolutional neural network, using the texture information of the image, the spatial structure information and the center map. Simultaneously, a convolutional neural network is used in parallel to detect the correlation between joint points, ie which joint points should be connected together.

As shown in Figure 3, when an image containing a human body is acquired, we use the entire image as an input to the detection network, and simultaneously input the image into two parallel binary networks, while detecting the confidence of the joint point and the joint point. relativity. At the same time, the joint point confidence map and the joint point correlation map and the input image are combined together as the input of the next stage. In the actual detection process, six such detection networks are cascaded.

As shown in Fig. 4, Fig. a is a complete input image, Fig. b is a confidence feature map of joint points, Fig. c is a joint point correlation feature map, and Fig. d is a series of candidate joint point matching after sparse method The figure, figure e is the figure after finally completing the combined connection of the joint points of the human body.

A plurality of people may be recognized in the image. Therefore, in the embodiment, an image is acquired by the camera, and an image human body detection algorithm is applied to obtain a series of human body image coordinates {(u _k1 , v _k1 ), ..., (u _ki , v _ki ),...,(u _kn ,v _kn )}, where k represents the kth person and i represents the i-th body joint point. For ease of understanding, some of the following calculations take only one joint point as an example.

The image coordinate system is a two-dimensional coordinate system in the imaging plane of the camera. The origin is usually located at the upper left corner of the screen. The width (transverse) of the screen is the x-axis, and the height (longitudinal) of the image is the y-axis, in units of pixels. By recognizing the position of the human body in the picture, the human body image coordinates (u, v) of the human body nodes in the image coordinate system are obtained, u represents the u-th pixel point of the x-axis, and v represents the v-th pixel point of the y-axis.

Step S20: Convert the human body image coordinates (u, v) in the image coordinate system to the image physical coordinates (x, y) in the image physical coordinate system. Since the image coordinate system is based on the number of pixels, which is not conducive to calculation, it is necessary to convert the human body image coordinates (u, v) in the image coordinate system into the physical coordinates (x, y) of the image in the image physical coordinate system. Easy to calculate. The image physical coordinate system is an image coordinate system expressed in physical units (for example, meters), which can be understood as a translation coordinate system of an image coordinate system, and is also a two-dimensional coordinate system.

In one embodiment, the specific process of step S20 includes:

The human body image coordinates in the image coordinate system are converted into image physical coordinates in the image physical coordinate system, and the conversion relationship is:

The above formula is a matrix operation formula, (u, v) is the human body image coordinate in the image coordinate system, (x, y) is the human body image coordinate in the image physical coordinate system, and u ₀ and v ₀ are the camera optical axis. The coordinate values in the image coordinate system, dx and dy are the physical size values of the x-axis and y-axis of each pixel.

Step S30: Determine, according to x, y, the vertical deflection angle α and the horizontal deflection angle β of the laser emitter to illuminate the key joint points of the human body, and activate the laser emitter to emit laser light to measure the distance Z' of the human body to the robot.

The key human joint point here is at least one of the above-mentioned human joint points, for example, all joint points in the above-mentioned human joint points, or several important joint points, such as a head, a shoulder, an elbow, a wrist, Knees and ankles. In some embodiments, for convenience, only one of the important joint points may be selected as a key human joint point, such as a head. The distance from the key human joint point to the robot is obtained, which is equivalent to the distance from the human body to the robot.

In one embodiment, the specific process of step S30 includes:

Determining the vertical deflection angle α and the horizontal deflection angle β of the laser emitter to illuminate the key joint points of the human body, and starting the laser emitter to emit laser light to measure the distance Z' of the human body to the robot, wherein:

d _u is the offset of the camera coordinate system relative to the image physical coordinate system, d _v is the offset of the laser coordinate system relative to the image physical coordinate system, and d is the horizontal distance of the laser emitter to the imaging plane of the camera. The camera coordinate system is a three-dimensional coordinate system whose origin is the optical center of the camera. The X and Y axes are parallel to the x and y axes of the image coordinate system. The z axis is the optical axis of the camera, and the z axis is perpendicular to the imaging plane of the camera.

The vertical deflection angle α and the horizontal deflection angle β are relative to the camera coordinate system. After calculating the vertical deflection angle α and the horizontal deflection angle β, the laser emitter adjusts to the corresponding position according to the current posture and α, β to align the key human body. The node is turned off and then a laser is emitted to measure the distance Z' of the human body to the robot.

Through the above steps, x, y, and Z' of each human joint point can be obtained, so that the specific position (X, Y, Z) of the joint point of the human body can be determined through subsequent steps, and the spacing between any two persons can be determined.

Step S40: Calculate the coordinates (X, Y, Z) of the human joint point in the camera coordinate system according to x, y, and Z'.

The position of the imaging plane of the camera coordinate system and the coordinate origin of the laser coordinate system differ only by one d in the z-axis direction, ignoring the slight deviation of the position between the laser emitter and the camera (usually the laser emitter and the camera are not far apart in space) The laser measurement value is the true value of the corresponding world coordinate system in the camera coordinate system. We can find the distance between two people in the real world coordinate system according to the proportional relationship between the camera coordinate value and the real world coordinate value.

In one embodiment, the specific process of step S40 includes:

According to x, y, d, Z', the coordinates (X, Y, Z) of each human joint point in the camera coordinate system are calculated, and the corresponding human joint points between any two people are calculated according to (X, Y, Z). The distance L, where the coordinates of the camera coordinate system (X, Y, Z) and the coordinates (x, y) of the image physical coordinate system are:

f is the focal length of the camera, Z=Z'+d.

Step S50: Calculate between the corresponding human joint points of any two persons according to the coordinates (X ₁ , Y ₁ , Z ₁ ) and (X ₂ , Y ₂ , Z ₂ ) of the camera coordinate system of the corresponding human joint points of any two persons. The distance L. The distance L between two people in the real world coordinate system can be obtained from the proportional relationship between the camera coordinate value and the real world coordinate value.

Step S60: determining the walking direction according to the distance L between the respective human joint points of any two persons. When the position of each person is determined, thereby determining the distance L between any two people, the robot can determine the walking strategy at this time. When determining the walking strategy, at least one factor analysis such as the distance between the human body and the robot, the walking speed of the human body, and the walking speed of the robot may be combined.

Suppose the body image coordinates of the human joint point are {(x _k1 , y _k1 ),...,(x _ki ,y _ki ),...,(x _kn ,y _kn )}, and calculate the spacing of any two of them. Finally, select the best separation distance to pass.

Suppose we calculate the interval between two people, P ₁ P ₂ , where:

P ₁ ={(x ₁₁ ,y ₁₁ ),...,(x _1i ,y _1i ),...,(x _1n ,y _1n )}

P ₂ ={(x ₂₁ ,y ₂₁ ),...,(x _2i ,y _2i ),...,(x _2n ,y _2n )}

Since the distance between the two is proportional to the distance in the X-axis direction, you can focus on only the X-axis coordinates, so you can group all the detected points of the two into one queue:

P={x ₁₁ ,x ₂₁ ,...,x _1i ,x _2i ,...,x _1n ,x _2n }

Using the quick sort algorithm, you can quickly find the nearest or intersecting points between two discrete point groups, so that the distance between these points can be calculated according to the proportional relationship between the camera coordinate values and the real world coordinate values described above, and Stop detecting the distance between each two people and find the best passing distance in real time.

FIG. 5 is a schematic diagram of a module of a robot walking obstacle detecting device according to an embodiment. Corresponding to the above-mentioned robot walking obstacle detecting method, the present application further provides a robot walking obstacle detecting device, the robot having a camera, a laser emitter, and a laser receiver, the device comprising: an image coordinate acquiring module 100 and a physical coordinate acquiring module 200. The human body distance acquiring module 300, the camera coordinate acquiring module 400, the crowd spacing acquiring module 500, and the walking direction determining module 600.

The image coordinate acquiring module 100 is configured to acquire an image by using a camera, and apply an image human body detection algorithm to obtain human body image coordinates (u, v) of the human joint point in the image coordinate system; the physical coordinate acquiring module is configured to use the human body image in the image coordinate system. The coordinates (u, v) are converted into image physical coordinates (x, y) in the physical coordinate system of the image; the human body distance acquisition module is configured to determine the vertical deflection angle α and the horizontal deflection of the laser emitter to illuminate the key joint points of the human body according to x and y. Angle β, start the laser emitter to emit laser to measure the distance Z' of the human body to the robot; the camera coordinate acquisition module is used to calculate the coordinates (X, Y, Z) of the human joint point in the camera coordinate system according to x, y, Z' The crowd spacing acquisition module is configured to calculate the corresponding human body of any two persons according to the coordinates (X ₁ , Y ₁ , Z ₁ ) and (X ₂ , Y ₂ , Z ₂ ) of the camera coordinate system of the corresponding human joint points of any two persons. The distance L between the nodes is closed; the walking direction determining module is configured to determine the walking direction according to the distance L between the respective human joint points of any two persons.

The image coordinate acquisition module 100 acquires an image through a camera, and applies an image human body detection algorithm to obtain human body image coordinates (u, v) of the human joint point in the image coordinate system.

The image human body detection algorithm refers to a technique for recognizing a human body through image recognition technology. There are three methods for image human body detection, which are global feature based methods, human body part based methods, and stereo vision based methods.

Global feature based approach

This type of method is currently the more mainstream human detection method. It mainly uses various static features of images such as edge features, shape features, statistical features or transform features to describe the human body. Representative features include Haar wavelet features, HOG features, and Edgelet. Features, Shapelet features, and outline template features.

(1) Method based on Haar wavelet feature

Papageorgiou and Poggio first proposed the concept of Harr wavelet, Viola et al. introduced the concept of integral graph, accelerated the extraction speed of Harr features, and applied the method to human detection, combined with human motion and appearance mode to construct human detection system. The better detection effect lays the foundation for the development of human detection technology.

(2) HOG-based method

Applying the concept of Histogram of Oriented Gradients (HOG) to human detection, achieving nearly 100% detection success rate on the MIT human body database; on the INRIA human body database containing changes in perspective, illumination and background , also achieved a test success rate of about 90%.

(3) Method based on edgelet feature

The "Edgelet" feature, which is a short line or curve segment, is applied to human detection of a single image of a complex scene, yielding a detection rate of approximately 92% on the CAVIAR database.

(4) Method based on Shapelet feature

Features can be automatically obtained using machine learning methods, namely Shapelet features. The algorithm first extracts the gradient information of different directions from the training samples, and then uses the AdaBoost algorithm to train, thus obtaining the Shapelet feature.

(5) Method based on contour template

The method refers to constructing a template by using information such as edge contour, texture and gray level of the target object in the image, and detecting the target by template matching method.

Method based on human body parts

The basic idea of this kind of method is to divide the human body into several components, and then separately test each part of the image, and finally integrate the detection results according to a certain constraint relationship, and finally determine whether there is a human body.

Stereo vision based method

This type of method refers to image acquisition by two or more cameras, and then analyzing the three-dimensional information of the target in the image to identify the human body.

In this embodiment, the image human body detection algorithm is not specifically limited as long as the human body and its joint points can be identified. Human joint points refer to key nodes of the human body such as the head, neck, shoulders, upper arms, elbows, wrists, waist, chest, thighs, knees, calves, ankles, etc. There are multiple.

In the present embodiment, human body detection can be performed using a method based on a human body part. OpenPose is an open source library written in C++ using OpenCV and Caffe for real-time detection of multi-threaded multi-person key points. The main detection of human body detection consists of two parts: human joint point detection and joint point correlation detection. Among them, the joint point detection uses a Convolutional Pose Machine. The attitude machine can infer the position of the key points of the human body by adding the convolutional neural network, using the texture information of the image, the spatial structure information and the center map. Simultaneously, a convolutional neural network is used in parallel to detect the correlation between joint points, ie which joint points should be connected together.

As shown in Figure 3, when an image containing a human body is acquired, we use the entire image as an input to the detection network, and simultaneously input the image into two parallel binary networks, while detecting the confidence of the joint point and the joint point. relativity. At the same time, the joint point confidence map and the joint point correlation map and the input image are combined together as the input of the next stage. In the actual detection process, six such detection networks are cascaded.

As shown in Fig. 4, Fig. a is a complete input image, Fig. b is a confidence feature map of joint points, Fig. c is a joint point correlation feature map, and Fig. d is a series of candidate joint point matching after sparse method The figure, figure e is the figure after finally completing the combined connection of the joint points of the human body.

A plurality of people may be recognized in the image. Therefore, in the embodiment, an image is acquired by the camera, and an image human body detection algorithm is applied to obtain a series of human body image coordinates {(u _k1 , v _k1 ), ..., (u _ki , v _ki ),...,(u _kn ,v _kn )}, where k represents the kth person and i represents the i-th body joint point. For ease of understanding, some of the following calculations take only one joint point as an example.

The image coordinate system is a two-dimensional coordinate system in the imaging plane of the camera. The origin is usually located at the upper left corner of the screen. The width (transverse) of the screen is the x-axis, and the height (longitudinal) of the image is the y-axis, in units of pixels. By recognizing the position of the human body in the picture, the human body image coordinates (u, v) of the human body nodes in the image coordinate system are obtained, u represents the u-th pixel point of the x-axis, and v represents the v-th pixel point of the y-axis.

The physical coordinate acquisition module 200 converts the human body image coordinates (u, v) in the image coordinate system into image physical coordinates (x, y) in the image physical coordinate system. Since the image coordinate system is based on the number of pixels, which is not conducive to calculation, it is necessary to convert the human body image coordinates (u, v) in the image coordinate system into the physical coordinates (x, y) of the image in the image physical coordinate system. Easy to calculate. The image physical coordinate system is an image coordinate system expressed in physical units (for example, meters), which can be understood as a translation coordinate system of an image coordinate system, and is also a two-dimensional coordinate system.

In one embodiment, the physical coordinate acquisition module 200 is specifically configured to:

The human body image coordinates in the image coordinate system are converted into image physical coordinates in the image physical coordinate system, and the conversion relationship is:

The above formula is a matrix operation formula, (u, v) is the human body image coordinate in the image coordinate system, (x, y) is the human body image coordinate in the image physical coordinate system, and u ₀ and v ₀ are the camera optical axis. The coordinate values in the image coordinate system, dx and dy are the physical size values of the x-axis and y-axis of each pixel.

The human body distance acquisition module 300 determines the vertical deflection angle α and the horizontal deflection angle β of the laser emitter to illuminate the key human joint points according to x, y, and activates the laser emitter to emit laser light to measure the distance Z' of the human body to the robot.

The key human joint point here is at least one of the above-mentioned human joint points, for example, all joint points in the above-mentioned human joint points, or several important joint points, such as a head, a shoulder, an elbow, a wrist, Knees and ankles. In some embodiments, for convenience, only one of the important joint points may be selected as a key human joint point, such as a head. The distance from the key human joint point to the robot is obtained, which is equivalent to the distance from the human body to the robot.

In one embodiment, the human body distance acquisition module 300 is specifically configured to:

Determining the vertical deflection angle α and the horizontal deflection angle β of the laser emitter to illuminate the key joint points of the human body, and starting the laser emitter to emit laser light to measure the distance Z' of the human body to the robot, wherein:

d _u is the offset of the camera coordinate system relative to the image physical coordinate system, d _v is the offset of the laser coordinate system relative to the image physical coordinate system, and d is the horizontal distance of the laser emitter to the imaging plane of the camera. The camera coordinate system is a three-dimensional coordinate system whose origin is the optical center of the camera. The X and Y axes are parallel to the x and y axes of the image coordinate system. The z axis is the optical axis of the camera, and the z axis is perpendicular to the imaging plane of the camera.

The vertical deflection angle α and the horizontal deflection angle β are relative to the camera coordinate system. After calculating the vertical deflection angle α and the horizontal deflection angle β, the laser emitter adjusts to the corresponding position according to the current posture and α, β to align the key human body. The node is turned off and then a laser is emitted to measure the distance Z' of the human body to the robot.

The camera coordinate acquisition module 400 calculates the coordinates (X, Y, Z) of the human joint point in the camera coordinate system according to x, y, and Z'.

The position of the imaging plane of the camera coordinate system and the coordinate origin of the laser coordinate system differ only by one d in the z-axis direction, ignoring the slight deviation of the position between the laser emitter and the camera (usually the laser emitter and the camera are not far apart in space) The laser measurement value is the true value of the corresponding world coordinate system in the camera coordinate system. We can find the distance between two people in the real world coordinate system according to the proportional relationship between the camera coordinate value and the real world coordinate value.

In one embodiment, the camera coordinate acquisition module 400 is specifically configured to:

According to x, y, d, Z', the coordinates (X, Y, Z) of each human joint point in the camera coordinate system are calculated, and the corresponding human joint points between any two people are calculated according to (X, Y, Z). The distance L, where the coordinates of the camera coordinate system (X, Y, Z) and the coordinates (x, y) of the image physical coordinate system are:

f is the focal length of the camera, Z=Z'+d.

The crowd spacing acquisition module 500 calculates the corresponding human joint points of any two persons according to the coordinates (X ₁ , Y ₁ , Z ₁ ) and (X ₂ , Y ₂ , Z ₂ ) of the camera coordinate system of the corresponding human joint points of any two persons. The distance between L. The distance L between two people in the real world coordinate system can be obtained from the proportional relationship between the camera coordinate value and the real world coordinate value.

The walking direction determining module 600 determines the walking direction based on the distance L between the respective human joint points of any two persons. When the position of each person is determined, thereby determining the distance L between any two people, the robot can determine the walking strategy at this time. When determining the walking strategy, at least one factor analysis such as the distance between the human body and the robot, the walking speed of the human body, and the walking speed of the robot may be combined.

Suppose the body image coordinates of the human joint point are {(x _k1 , y _k1 ),...,(x _ki ,y _ki ),...,(x _kn ,y _kn )}, and calculate the spacing of any two of them. Finally, select the best separation distance to pass.

Suppose we calculate the interval between two people, P ₁ P ₂ , where:

P ₁ ={(x ₁₁ ,y ₁₁ ),...,(x _1i ,y _1i ),...,(x _1n ,y _1n )}

P ₂ ={(x ₂₁ ,y ₂₁ ),...,(x _2i ,y _2i ),...,(x _2n ,y _2n )}

Since the distance between the two is proportional to the distance in the X-axis direction, you can focus on only the X-axis coordinates, so you can group all the detected points of the two into one queue:

P={x ₁₁ ,x ₂₁ ,...,x _1i ,x _2i ,...,x _1n ,x _2n }

The walking direction determining module 600 uses the quick sorting algorithm to quickly find the nearest or intersecting points between the two discrete point groups, so that the distance between the camera coordinates and the real world coordinate values can be calculated between the points. The distance and the distance between each two are constantly detected, and the best passing distance can be found in real time.

The application also provides a computer device comprising a memory and a processor, the memory storing computer readable instructions, the computer readable instructions being executed by the processor, causing the processor to perform any of the above The steps of the robot walking obstacle detecting method according to the embodiment.

The present application also provides a storage medium storing computer readable instructions that, when executed by one or more processors, cause one or more processors to perform the robotic walking described in any of the above embodiments The steps of the obstacle detection method.

The above-mentioned robot walking obstacle detection method, device, computer equipment and storage medium acquire images through a camera, and apply image human body detection algorithm to obtain human body image coordinates (u, v) of human joint points in an image coordinate system; The human body image coordinates (u, v) are converted into image physical coordinates (x, y) in the image physical coordinate system; the vertical deflection angle α and the horizontal deflection angle β of the laser emitter irradiating the key human joint points are determined according to x, y, Starting the laser emitter to emit laser light to measure the distance Z' of the human body to the robot; calculating the coordinates (X, Y, Z) of the human joint point in the camera coordinate system according to x, y, Z'; according to the corresponding human joint of any two people Point the coordinates (X ₁ , Y ₁ , Z ₁ ) and (X ₂ , Y ₂ , Z ₂ ) of the camera coordinate system to calculate the distance L between the corresponding human joint points of any two persons; according to the corresponding human body of any two persons The distance L between the off nodes determines the direction of travel. By combining image recognition and laser ranging to determine the actual coordinates of the human body (in the coordinates of the camera coordinate system), the real-time position of the human body can be accurately obtained, thereby determining the spacing L between any two people, and then determining the walking path in real time through L, realizing Accurate pedestrian obstacle detection.

A person skilled in the art can understand that all or part of the process of implementing the above embodiment method can be completed by a computer program to instruct related hardware, and the computer program can be stored in a computer readable storage medium. When executed, the flow of an embodiment of the methods as described above may be included. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

Claims (16)

1. A robot walking obstacle detecting method, the robot having a camera, a laser emitter, and a laser receiver, the method comprising the following steps:

   Step S10: acquiring an image by using a camera, and applying an image human body detection algorithm to obtain human body image coordinates (u, v) of the human joint point in the image coordinate system;

   Step S20: converting human body image coordinates (u, v) in the image coordinate system into image physical coordinates (x, y) in the image physical coordinate system;

   Step S30: determining, according to x, y, the vertical deflection angle α and the horizontal deflection angle β of the laser emitter irradiating the key joint points of the human body, and starting the laser emitter to emit the laser to measure the distance Z' of the human body to the robot;

   Step S40: calculating coordinates (X, Y, Z) of the joint point of the human body in the camera coordinate system according to x, y, and Z';

   Step S50: Calculate between the corresponding human joint points of any two persons according to the coordinates (X ₁ , Y ₁ , Z ₁ ) and (X ₂ , Y ₂ , Z ₂ ) of the camera coordinate system of the corresponding human joint points of any two persons. Distance L;

   Step S60: determining the walking direction according to the distance L between the respective human joint points of any two persons.
2. The method for detecting a walking obstacle of a robot according to claim 1, wherein the step S20 comprises:

   The human body image coordinates in the image coordinate system are converted into image physical coordinates in the image physical coordinate system, and the conversion relationship is:

   

   Where (u, v) is the human body image coordinate in the image coordinate system, (x, y) is the human body image coordinate in the image physical coordinate system, and u ₀ and v ₀ are the coordinates of the camera optical axis in the image coordinate system. The values, dx and dy are the physical size values for each pixel on the x and y axes.
3. The method for detecting a walking obstacle of a robot according to claim 2, wherein the step S30 comprises:

   Determining the vertical deflection angle α and the horizontal deflection angle β of the laser emitter to illuminate the key joint points of the human body, and starting the laser emitter to emit laser light to measure the distance Z' of the human body to the robot, wherein:

   

   

   d _u is the offset of the camera coordinate system relative to the image physical coordinate system, d _v is the offset of the laser coordinate system relative to the image physical coordinate system, and d is the horizontal distance of the laser emitter to the imaging plane of the camera.
4. The method for detecting a walking obstacle of a robot according to claim 3, wherein the step S40 comprises:

   According to x, y, d, Z', the coordinates (X, Y, Z) of each human joint point in the camera coordinate system are calculated, and the corresponding human joint points between any two people are calculated according to (X, Y, Z). The distance L, where the coordinates of the camera coordinate system (X, Y, Z) and the coordinates (x, y) of the image physical coordinate system are:

   

   f is the focal length of the camera, Z=Z'+d.
5. A robot walking obstacle detecting device, the robot having a camera, a laser emitter, and a laser receiver, the device comprising:

   An image coordinate acquisition module, configured to acquire an image by using a camera, and apply an image human body detection algorithm to obtain human body image coordinates (u, v) of a human joint point in an image coordinate system;

   a physical coordinate acquisition module, configured to convert human body image coordinates (u, v) in an image coordinate system into image physical coordinates (x, y) in an image physical coordinate system;

   a human body distance acquisition module, configured to determine, according to x, y, a vertical deflection angle α and a horizontal deflection angle β of the laser emitter to illuminate the key joint points of the human body, and activate the laser emitter to emit laser light to measure the distance Z' of the human body to the robot;

   a camera coordinate acquisition module, configured to calculate coordinates (X, Y, Z) of the human joint point in the camera coordinate system according to x, y, and Z';

   The crowd spacing acquisition module is configured to calculate the corresponding human body of any two persons according to the coordinates (X ₁ , Y ₁ , Z ₁ ) and (X ₂ , Y ₂ , Z ₂ ) of the camera coordinate system of the corresponding human joint points of any two persons The distance L between the nodes;

   The walking direction determining module is configured to determine a walking direction according to a distance L between respective human joint points of any two persons.
6. The robot walking obstacle detecting device according to claim 5, wherein the physical coordinate acquiring module is configured to:

   The human body image coordinates in the image coordinate system are converted into image physical coordinates in the image physical coordinate system, and the conversion relationship is:

   

   Where (u, v) is the human body image coordinate in the image coordinate system, (x, y) is the human body image coordinate in the image physical coordinate system, and u ₀ and v ₀ are the coordinates of the camera optical axis in the image coordinate system. The values, dx and dy are the physical size values for each pixel on the x and y axes.
7. The robot walking obstacle detecting device according to claim 6, wherein the human body distance acquiring module is configured to:

   Determining the vertical deflection angle α and the horizontal deflection angle β of the laser emitter to illuminate the key joint points of the human body, and starting the laser emitter to emit laser light to measure the distance Z' of the human body to the robot, wherein:

   

   

   d _u is the offset of the camera coordinate system relative to the image physical coordinate system, d _v is the offset of the laser coordinate system relative to the image physical coordinate system, and d is the horizontal distance of the laser emitter to the imaging plane of the camera.
8. The robot walking obstacle detecting device according to claim 7, wherein the camera coordinate acquiring module is configured to:

   According to x, y, d, Z', the coordinates (X, Y, Z) of each human joint point in the camera coordinate system are calculated, and the corresponding human joint points between any two people are calculated according to (X, Y, Z). The distance L, where the coordinates of the camera coordinate system (X, Y, Z) and the coordinates (x, y) of the image physical coordinate system are:

   

   f is the focal length of the camera, Z=Z'+d.
9. A computer device comprising a memory and a processor, the memory storing computer readable instructions, the computer readable instructions being executed by the processor, causing the processor to perform a robot walking obstacle detection method, The robot walking obstacle detection method includes the following steps:

   Step S10: acquiring an image by using a camera, and applying an image human body detection algorithm to obtain human body image coordinates (u, v) of the human joint point in the image coordinate system;

   Step S20: converting human body image coordinates (u, v) in the image coordinate system into image physical coordinates (x, y) in the image physical coordinate system;

   Step S30: determining, according to x, y, the vertical deflection angle α and the horizontal deflection angle β of the laser emitter irradiating the key joint points of the human body, and starting the laser emitter to emit the laser to measure the distance Z' of the human body to the robot;

   Step S40: calculating coordinates (X, Y, Z) of the joint point of the human body in the camera coordinate system according to x, y, and Z';

   Step S50: Calculate between the corresponding human joint points of any two persons according to the coordinates (X ₁ , Y ₁ , Z ₁ ) and (X ₂ , Y ₂ , Z ₂ ) of the camera coordinate system of the corresponding human joint points of any two persons. Distance L;

   Step S60: determining the walking direction according to the distance L between the respective human joint points of any two persons.
10. The computer device of claim 9, the step S20 comprising:

    The human body image coordinates in the image coordinate system are converted into image physical coordinates in the image physical coordinate system, and the conversion relationship is:

    

    Where (u, v) is the human body image coordinate in the image coordinate system, (x, y) is the human body image coordinate in the image physical coordinate system, and u ₀ and v ₀ are the coordinates of the camera optical axis in the image coordinate system. The values, dx and dy are the physical size values for each pixel on the x and y axes.
11. The computer device of claim 10, the step S30 comprising:

    Determining the vertical deflection angle α and the horizontal deflection angle β of the laser emitter to illuminate the key joint points of the human body, and starting the laser emitter to emit laser light to measure the distance Z' of the human body to the robot, wherein:

    

    

    d _u is the offset of the camera coordinate system relative to the image physical coordinate system, d _v is the offset of the laser coordinate system relative to the image physical coordinate system, and d is the horizontal distance of the laser emitter to the imaging plane of the camera.
12. The computer device of claim 11, the step S40 comprising:

    According to x, y, d, Z', the coordinates (X, Y, Z) of each human joint point in the camera coordinate system are calculated, and the corresponding human joint points between any two people are calculated according to (X, Y, Z). The distance L, where the coordinates of the camera coordinate system (X, Y, Z) and the coordinates (x, y) of the image physical coordinate system are:

    

    f is the focal length of the camera, Z=Z'+d.
13. A non-volatile storage medium storing computer readable instructions, when executed by one or more processors, causing one or more processors to perform a robotic walking obstacle detection method, The robot walking obstacle detection method includes the following steps:

    Step S10: acquiring an image by using a camera, and applying an image human body detection algorithm to obtain human body image coordinates (u, v) of the human joint point in the image coordinate system;

    Step S20: converting human body image coordinates (u, v) in the image coordinate system into image physical coordinates (x, y) in the image physical coordinate system;

    Step S30: determining, according to x, y, the vertical deflection angle α and the horizontal deflection angle β of the laser emitter irradiating the key joint points of the human body, and starting the laser emitter to emit the laser to measure the distance Z' of the human body to the robot;

    Step S40: calculating coordinates (X, Y, Z) of the joint point of the human body in the camera coordinate system according to x, y, and Z';

    Step S50: Calculate between the corresponding human joint points of any two persons according to the coordinates (X ₁ , Y ₁ , Z ₁ ) and (X ₂ , Y ₂ , Z ₂ ) of the camera coordinate system of the corresponding human joint points of any two persons. Distance L;

    Step S60: determining the walking direction according to the distance L between the respective human joint points of any two persons.
14. The nonvolatile storage medium of claim 13, wherein the step S20 comprises:

    The human body image coordinates in the image coordinate system are converted into image physical coordinates in the image physical coordinate system, and the conversion relationship is:

    

    Where (u, v) is the human body image coordinate in the image coordinate system, (x, y) is the human body image coordinate in the image physical coordinate system, and u ₀ and v ₀ are the coordinates of the camera optical axis in the image coordinate system. The values, dx and dy are the physical size values for each pixel on the x and y axes.
15. The nonvolatile storage medium of claim 14, the step S30 comprising:

    Determining the vertical deflection angle α and the horizontal deflection angle β of the laser emitter to illuminate the key joint points of the human body, and starting the laser emitter to emit laser light to measure the distance Z' of the human body to the robot, wherein:

    

    

    d _u is the offset of the camera coordinate system relative to the image physical coordinate system, d _v is the offset of the laser coordinate system relative to the image physical coordinate system, and d is the horizontal distance of the laser emitter to the imaging plane of the camera.
16. The nonvolatile storage medium according to claim 15, wherein the step S40 comprises:

    According to x, y, d, Z', the coordinates (X, Y, Z) of each human joint point in the camera coordinate system are calculated, and the corresponding human joint points between any two people are calculated according to (X, Y, Z). The distance L, where the coordinates of the camera coordinate system (X, Y, Z) and the coordinates (x, y) of the image physical coordinate system are:

    

    f is the focal length of the camera, Z=Z'+d.

Images