WO2022227664A1

WO2022227664A1 - Robot posture control method, robot, storage medium and computer program

Info

Publication number: WO2022227664A1
Application number: PCT/CN2021/142242
Authority: WO
Inventors: 彭飞
Original assignee: 达闼机器人股份有限公司
Priority date: 2021-04-25
Filing date: 2021-12-28
Publication date: 2022-11-03
Also published as: CN113146634A

Abstract

A robot posture control method. The method comprises: according to a target posture of a target object in image data, acquiring a three-dimensional skeleton model corresponding to the target posture; mapping the three-dimensional skeleton model into a joint space of a robot, so as to acquire a posture feature of the robot and a skeleton posture feature of the three-dimensional skeleton model; adjusting the posture feature of the robot to a position that matches the skeleton posture feature, so as to acquire rotation angle information of each joint of the robot; and according to the rotation angle information of each joint, controlling the motion of the corresponding joint of the robot, so as to form the target posture. By means of the method, an additional motion capture sensor is not needed to upload a motion of a target object in real time, thereby reducing the cost of controlling a robot to move according to a target posture. The present application further relates to a robot posture control apparatus, a robot, a computer-readable storage medium and a computer program.

Description

Robot attitude control method, robot, storage medium and computer program

cross reference

This application claims the priority of the Chinese patent application filed on April 25, 2021 with the application number 202110450270.2 and the title of the invention is "Control Method for Robot Pose, Robot and Storage Medium", the entire contents of which are incorporated by reference in in this application.

technical field

Embodiments of the present invention relate to the field of robots, and in particular, to a method for controlling robot posture, a robot, a storage medium, and a computer program.

Background technique

With the development of science and technology, a large number of motion control systems of intelligent robots have been designed and manufactured and applied in social production and life to improve social productivity and improve people's quality of life. Robot actions are generated by a sequence of sequences, such as shaking hands, raising hands or shaking heads, etc. At present, the action sequence generation of the robot can be manually adjusted according to the target position of the robot motion; or the human body posture perception can be realized through motion capture equipment, for example, The sensor data or 2D video processing technology can be used to detect and track the human skeleton to realize human posture perception; set the robot's action sequence according to the perceived human posture.

However, the manual debugging method needs to manually design the motion of each joint one by one, generate the action sequence of the robot, and form the posture of the robot. Since it needs to be debugged one by one, and the movement of each joint will affect the movement of other joints, the debugging time The process is long and the process is complicated; while capturing human movements through motion capture equipment requires additional equipment to capture human movements, resulting in inflexible generation of machine action sequences and high costs.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present invention is to provide a robot posture control method, robot, storage medium and computer program, which can quickly generate a target posture that matches the target object, enrich the robot's actions, and simplify the cost of the robot's learning action posture.

In order to solve the above technical problems, in the first aspect, the embodiments of the present disclosure provide a method for controlling the posture of a robot, including: obtaining a three-dimensional skeleton model corresponding to the target posture according to the target posture of the target object in the image data; The three-dimensional skeleton model is mapped into the joint space of the robot, and the posture features of the robot and the skeleton posture features of the three-dimensional skeleton model are obtained; the posture features of the robot are adjusted to match the skeleton posture features. The target position is obtained, and the rotation angle information of each joint of the robot is obtained; the movement of the corresponding joint of the robot is controlled according to the rotation angle information of each joint to form the target posture.

Further, obtaining the three-dimensional skeleton model corresponding to the target pose according to the target pose of the target object in the image data includes: inputting the image data into a preset first neural network model, and obtaining the target pose the two-dimensional skeleton data, the first neural network model is used to identify the target posture of the target object in the image data, and generate corresponding two-dimensional skeleton data based on the target posture; Inputting the preset second neural network model to obtain a three-dimensional skeleton model corresponding to the target pose.

Further, adjusting the posture feature of the robot to a target position matching the skeleton posture feature, and acquiring the rotation angle information of each joint of the robot, includes: dividing the posture feature of the robot into a plurality of mapping parts; The following processing is performed for each mapping part: the position of the vector of each joint in the mapping part is transformed into the position of the vector of the corresponding key point in the skeleton pose feature, and the rotation angle of the joint is obtained as the joint corner information.

Further, the plurality of mapping parts include: a trunk part, a limb part, a head part and a waist part.

Further, the method further includes: taking the corner information of each joint as the motion data of the current frame; performing collision detection on the motion data of the current frame; if no collision is detected, determining to execute the motion data according to each joint. The rotation angle information of the control corresponds to the movement of the joint to form the step of forming the target posture.

Further, the controlling the movement of the robot corresponding to the joints according to the rotation angle information of the joints to form the target posture includes: filtering the rotation angle information of the joints; The rotation angle information of the joint controls the rotation of the corresponding joint to form the target posture.

Further, before the three-dimensional skeleton model is mapped into the joint space of the robot, and the posture features of the robot and the skeleton posture features of the three-dimensional skeleton model are acquired, the method further includes: mapping the three-dimensional skeleton. The model is normalized.

Further, before the two-dimensional skeleton data of the target pose is obtained by inputting the image data into the preset first neural network model, the method further includes: collecting video data of the target object; The image data is obtained from video data.

On the other hand, an embodiment of the present disclosure also provides a robot posture control device, including: a model acquisition module, configured to acquire a three-dimensional skeleton model corresponding to the target posture according to the target posture of the target object in the image data; an attitude acquisition module, used to map the three-dimensional skeleton model into the joint space of the robot, and acquire the attitude features of the robot and the skeleton attitude characteristics of the three-dimensional skeleton model; the corner acquisition module is used to map the robot The posture feature of the robot is adjusted to the target position matching the skeleton posture feature, and the rotation angle information of each joint of the robot is obtained; the control motion module is used to control the corresponding joint of the robot according to the rotation angle information of each joint. movement to form the target pose.

Further, the model acquisition module is specifically used for:

Inputting the image data into a preset first neural network model to obtain two-dimensional skeleton data of the target posture, and the first neural network model is used to identify the target posture of the target object in the image data, and generate corresponding two-dimensional skeleton data based on the target posture; input the two-dimensional skeleton data into a preset second neural network model to obtain a three-dimensional skeleton model corresponding to the target posture.

Further, the angle acquisition module is specifically used to: divide the posture feature of the robot into a plurality of mapping parts; perform the following processing for each mapping part: transform the position of the vector where each joint in the mapping part is located For the position of the vector where the corresponding key point in the skeleton pose feature is located, the rotation angle of the joint is obtained as the rotation angle information of the joint.

Further, the device further includes: a motion data module, used for taking the rotation angle information of each joint as the motion data of the current frame; a collision detection module, used for collision detection on the motion data of the current frame; control motion The sub-module is configured to determine and execute the step of controlling the movement of the corresponding joint according to the rotation angle information of each joint to form the target posture if no collision is detected.

Further, the motion control module is specifically configured to: filter the rotation angle information of each joint; and control the rotation of the corresponding joint according to the processed rotation angle information of each joint to form the target posture.

Further, the apparatus further includes: a normalization module, configured to perform normalization processing on the three-dimensional skeleton model.

Further, the device further comprises: a video acquisition module for acquiring video data of the target object; an image acquisition module for acquiring the image data from the video data.

In another aspect, an embodiment of the present disclosure also provides a robot, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores information that can be used by the at least one processor Instructions executed by the processor, where the instructions are executed by the at least one processor, so that the at least one processor can execute the above-mentioned method for controlling the posture of the robot.

On the other hand, an embodiment of the present disclosure also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned method for controlling the attitude of the robot is implemented.

On the other hand, an embodiment of the present disclosure also provides a computer program, including instructions, which, when executed on a computer, cause the computer to execute the above-mentioned method for controlling the posture of a robot.

In the embodiment of the present disclosure, the limbs of the robot move in the three-dimensional space, and the three-dimensional skeleton model corresponding to the target pose is obtained through the target pose of the target object in the image data, and the three-dimensional skeleton model is mapped into the joint space of the robot, so that The three-dimensional skeleton model and the joint space are in the same coordinate system, and the rotation angle information of each joint in the robot is obtained, so that the posture formed by controlling the motion of the robot based on the rotation angle information of each joint corresponds to the target posture, and the target object in the image data guides the robot's movement. Motion, without the need for additional motion capture sensors to upload the motion of the target object in real time, reducing the cost of controlling the robot to move according to the target posture; and no need to manually debug each joint, simplifying the complex steps to generate the target posture, making the robot faster The action to learn the target pose.

Description of drawings

One or more embodiments are exemplified by the pictures in the corresponding drawings, and these exemplifications do not constitute limitations of the embodiments, and elements with the same reference numerals in the drawings are denoted as similar elements, Unless otherwise stated, the figures in the accompanying drawings do not constitute a scale limitation.

FIG. 1 is a flowchart of a method for controlling a robot posture in an embodiment of the present disclosure;

2 is a schematic diagram of a two-dimensional human skeleton provided in an embodiment of the present disclosure;

FIG. 3 is a flow chart of acquiring rotation angle information of each joint provided in an embodiment of the present disclosure;

Fig. 4 is a flow chart that provides collision detection on the rotation angle information of each joint in one embodiment of the present disclosure;

FIG. 5 is a flowchart of filtering processing of the rotation angle information of each joint provided in an embodiment of the present disclosure;

FIG. 6 is a flowchart of normalizing a three-dimensional skeleton model provided in an embodiment of the present disclosure;

FIG. 7 is a flowchart of obtaining a three-dimensional skeleton model provided in an embodiment of the present disclosure;

FIG. 8 is a flowchart of acquiring image data provided in an embodiment of the present disclosure;

FIG. 9 is a flowchart provided in an embodiment of the present disclosure for obtaining the rotation angle information of the joint and performing collision detection on the rotation angle information of the joint;

Fig. 10 is the flow chart of the control method of the robot posture after adding filtering processing in Fig. 9;

Fig. 11 is the flow chart of the control method of the robot posture that adds the normalization processing to the three-dimensional skeleton model in Fig. 10;

Fig. 12 is the flow chart of adding the control method of acquiring the robot posture of the three-dimensional skeleton model in Fig. 11;

Fig. 13 is the flow chart of the control method of adding the robot posture of acquiring image data in Fig. 11;

14 is a schematic diagram of a control device for a robot posture provided in another embodiment of the present disclosure;

FIG. 15 is a schematic structural diagram of a robot in another embodiment of the present disclosure.

Detailed ways

In order to make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those of ordinary skill in the art will appreciate that, in the various embodiments of the present invention, numerous technical details are set forth in order to provide the reader with a better understanding of the present disclosure. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present disclosure can be realized.

The following divisions of the various embodiments are for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention, and the various embodiments may be combined with each other and referred to each other on the premise of not contradicting each other.

The flow of the control method for the robot posture in the embodiment of the present disclosure is shown in FIG. 1 :

Step 101: According to the target posture of the target object in the image data, obtain a three-dimensional skeleton model corresponding to the target posture.

Specifically, the image data may be an image captured by a robot, or image data extracted from video data, for example, a frame of image in the video data is used as the image data. The target object can be a human body, an animal, etc. The robot in this embodiment is a multi-joint robot, such as a human-type robot, an animal-type robot, and the like.

The action gesture of the target object can be extracted as the target gesture by identifying the target object in the image data. Two-dimensional skeleton data corresponding to the target pose is acquired, and a three-dimensional skeleton model of the target object can be constructed based on the two-dimensional skeleton data.

In this example, the multi-joint robot takes a humanoid robot as an example, and the target object is a human body. Usually the skeleton of the human body is composed of 17 three-dimensional joint points, such as the two-dimensional human skeleton shown in Figure 2. The serial numbers 0 to 16 in Figure 2 represent: 0 pelvic center, 1 right hip joint, 2 right knee joint, 3 Right Ankle, 4 Left Hip, 5 Left Knee, 6 Left Ankle, 7 Midpoint of Spine, 8 Midpoint of Cervical Vertebra, 9 Head, 10 Linggai, 11 Left Shoulder, 12 Left Elbow, 13 Left Wrist, 14 Right Shoulder joint, 15 right elbow joint and 16 right wrist joint.

Step 102 : Map the three-dimensional skeleton model into the joint space of the robot, and obtain the pose feature of the robot and the skeleton pose feature of the three-dimensional skeleton model.

Specifically, for an operating arm with n degrees of freedom, all its link positions can be determined by a set of n joint variables. The set of joint variables is called the n x 1 joint vector, and the space composed of all joint vectors is called the joint space. The joint space of the robot can be determined according to the robotic arm of the robot. For example, the robotic arm of the robot can have 7 degrees of freedom, then based on the robotic arm with 7 degrees of freedom, a 7 x 1 joint vector can be constructed, and the space composed of all joint vectors is The joint space corresponding to the current robot arm. The joint space corresponding to the torso of the robot may also be used as the joint space of the robot.

The space angle between adjacent parts in the 3D skeleton model is the same as the space angle between adjacent parts corresponding to the robot, and the robot can present the same target pose as the 3D skeleton model. The 3D skeleton model and the robot are not in the same coordinate system. In this example, the 3D skeleton model is mapped to the joint space of the robot, and the pose feature of the robot is obtained. The pose feature includes the position of the vector of each joint of the robot and the 3D skeleton model. The skeleton pose feature includes the vector formed by each key point in the skeleton. For example, the magnitude of the vector where the joint 11 is located as shown in Fig. 2 may be the coordinate difference between the joint point 11 and the joint point 12, and the direction may be the execution of the joint point 12 from the joint point.

Step 103: Adjust the posture feature of the robot to a target position matching the posture feature of the skeleton, and obtain the rotation angle information of each joint of the robot.

Specifically, the posture feature of the robot includes the vectors where each joint of the robot is located, and the vector where each joint is located has a corresponding target vector, and the target vector is the vector where the corresponding key points in the skeleton posture feature are located. Transform the vector of the joint so that the vector of the joint coincides with the target vector, and obtain the rotation angle of the joint during the transformation process as the rotation angle information of the joint.

Step 104: Control the motion of the corresponding joint of the robot according to the rotation angle information of each joint to form a target posture.

The target pose can be formed by controlling each joint of the robot to move according to the corresponding joint angle.

In one embodiment, step 103 may perform sub-steps, the flow of which is shown in FIG. 3 .

Sub-step 1031: Divide the pose feature of the robot into multiple mapping parts.

Specifically, the robot pose feature can be divided into several mapping parts according to the position of the joint points. Due to the need to obtain the angles of the joints, the RPY system can be constructed based on the positions of the joint points, and the divided mapping parts include: trunk, limbs, head and waist. The trunk consists of: 0 center of pelvis, 11 left shoulder joint and 14 right shoulder joint. The limbs include: left upper limb, left lower limb, right upper limb and right lower limb. The left upper limb is: 11 left shoulder joint, 12 left elbow joint, 13 left wrist joint. The left lower limb is: 4 left hip joint, 5 left knee joint, 6 left ankle joint. The right upper limb is: 14 right shoulder joint, 15 right elbow joint and 16 right wrist joint. The right lower limb is: 1 right hip joint, 2 right knee joint, 3 right ankle joint. The head includes: 8 midpoints of cervical vertebrae, 9 heads, and 10 days Linggai. The waist includes: 0 pelvic center, 1 right hip joint, 4 left hip joint.

By dividing the mapping parts, it is convenient to obtain the Euler angles of the joints in each mapping part, thereby facilitating the calculation of the rotation angles of the joints, reducing the time for calculating the rotation angle information of the joints, and improving the calculation speed.

Sub-step 1032: Perform the following processing for each mapping part: transform the position of the vector of each joint in the mapping part into the position of the vector of the corresponding key point in the skeleton pose feature, and obtain the rotation angle of the joint as the rotation angle information of the joint.

By subtracting the adjacent key points, the line segments of the three-dimensional skeleton model can be converted into vectors, and the vector of the key points in the skeleton pose feature can be obtained. The rotation angle of each joint is obtained by means of Euclidean geometric settlement. Taking the left upper limb as an example and in conjunction with Figure 2, the detailed calculation process is as follows:

Take the normal vector of the plane formed by the three

joints

0, 11, and 14 as the z-axis, and the vector formed by the joint 0 to the joint 8 as the x-axis to construct a coordinate system. The positive direction of the x-axis of the coordinate system is the vertical when the person is upright. The positive direction of the y-axis is the right side when the person is upright, and the positive direction of the z-axis is the front side when the person is upright. Converting all other joints to this coordinate system, the movement of human limbs is based on the torso, and the movement of the robot that imitates the movement of the human body is also based on the torso, so the torso of the robot can be used as the basic coordinate system for the entire calculation.

Calculate the joint rotation angle of the keypoint "11→12" vector. In this example, the space vector v corresponding to the key point in the skeleton pose feature is known, and the respective rotation angles of the two joints a and b of the left shoulder are obtained. The shoulder of the robot (the position of the key point 11 in Figure 2) corresponds to the two joints , respectively control the front and rear swing of the boom and the left and right swing. After the x-axis is first rotated around the vector (cos20°, sin20°, 0) by a radian, and then rotated around the z-axis by b radian to coincide with v. In this example, the shoulder joint of the robot does not swing back and forth along the horizontal axis, but swings in the direction of an inclination of 20° with the horizontal axis, so the x-axis rotates around the vector (cos20°, sin20°, 0). In the process of algorithm calculation, there are usually multiple groups of solutions, and the group within the joint limit is selected from the multiple groups of solutions as the final output solution.

After obtaining the rotation angle of joint a and the rotation angle of joint b corresponding to the key point "11→12" vector, calculate the respective rotation angle information of joint c and d corresponding to the key point "12→13" vector. The rotation angle information of joint d is the angle between the key point "11→12" direction and the "12→13" vector. The rotation angle information of joint c can be obtained by reverse calculation, that is, the two vectors are rotated to negative b radians around the z-axis, and then rotated to negative a radians around the vector (cos20°, sin20°, 0), so that the v vector is equal to - The x vector coincides, and the key point "12→13" vector reaches a new position, and the vector of the new position is projected to the y-z plane, and the angle between y and z is the corner information of the joint c.

The rotation angles of the three joints of the head are obtained by means of Euclidean geometric settlement.

The joint structure of the head is a typical roll pitch yall system with three axes perpendicular to each other. The Euler angles can be restored by constructing a rotation matrix from

key points

8, 9, and 10. The "9→10" vector is used as the z-axis vector to form the third column of the rotation, and the "8→9" vector is cross-multiplied by the "9→10" vector to obtain the result as the y vector, which forms the second column of the rotation matrix, and then the y cross-multiplies The z vector gets the x vector, as the first column of the rotation matrix. The method of solving the Euler angles by the rotation matrix will not be repeated here.

The structure of the waist is similar to the structure of the head, and it is also a typical three-axis mutually perpendicular rpy system. The calculation of the rotation angle information of the waist joints is similar to the calculation of the rotation angle information of each joint in the head, and will not be repeated here.

The data structure of the skeleton of the human body in this example adopts the skeleton model of Human 3.6M.

In this embodiment, the three-dimensional skeleton model is divided into a plurality of mapping parts, and each mapping part is mapped separately. Compared with the entire three-dimensional skeleton model, the mapping part has a simpler structure and is easier to map to the joint space, reducing the difficulty of mapping At the same time, the limbs and joint spaces in the three-dimensional skeleton model are in the same space, which is more conducive to calculating the rotation angle information of the joints. At the same time, the Euler angle calculation method is adopted, and the calculation is simple and fast.

In one embodiment, the control method of the robot posture can also perform the following steps, and the process is shown in FIG. 4 :

Step 104-1: Use the rotation angle information of each joint as the motion data of the current frame.

Step 104-2: Perform collision detection on the motion data of the current frame.

Due to the various poses of the recognized 3D skeleton model, in order to avoid the problem of self-interference between the robot limbs when the joint information is applied to the robot, the collision detection can be performed on the rotation angle information of each joint. The collision detection model can be preset, and the collision detection model can be used to simulate the operation of each limb of the robot. For example, the moveit program can be used for collision detection, and the URDF file of the robot can be imported into the moveit, where the URDF file is the robot model description format; By inputting the rotation angle information of each joint, you can simulate the movement of each limb of the robot according to the rotation angle information of each joint, and judge whether there is a collision between the limbs. If no collision is detected, it means that the current information on the rotation angles of each joint is legal and can be applied to the robot.

Step 104-3: If no collision is detected, determine to execute the step of controlling the motion of the corresponding joint according to the rotation angle information of each joint to form the target posture.

The joint rotation angle is sent to the robot for execution, so as to achieve the effect of the actual robot tracking the posture of the target object.

In this embodiment, collision detection is performed on the rotation angle information of each joint, and if the probability of collision is detected to be less than a preset threshold, the rotation angle information of each shutdown is applied to the robot to ensure the accuracy of the robot posture, It also ensures the safety of the robot.

In one embodiment, step 104 can also perform the following sub-steps, and the flow is shown in FIG. 5 :

Step 1041: Perform filtering processing on the rotation angle information of each joint.

The rotation angle information of each joint is obtained based on the identified skeleton, and the identified skeleton inevitably has noise and beating. The corner information of the joints based on the noisy skeleton will also have noise. To filter the rotation angle information of the joints, a sliding window filtering method can be used to perform the filtering processing to remove the burr noise in the joint motion.

Step 1042: Control the rotation of the corresponding joint according to the processed rotation angle information of each joint to form a target pose.

In this embodiment, filtering processing is performed on the rotation angle information of each joint, noise and beating are eliminated, and the accuracy of the rotation angle information is improved.

In one embodiment, before step 102, how sub-steps may be performed, and the process is shown in FIG. 6 :

Sub-step 102-1: Normalize the three-dimensional skeleton model.

Specifically, after the three-dimensional skeleton model of the target pose is obtained, the length of the skeleton is not suitable for the limbs of the robot because the length of the skeleton is uneven. In order to facilitate the mapping of the 3D skeleton model into the joint space, the 3D skeleton model can be normalized. For example, the skeleton is converted from a line segment to a vector by subtracting adjacent keypoints. The vector is then unitized, and the resulting skeleton consists of unit vectors.

In this embodiment, by normalizing the three-dimensional skeleton model, the problem that the size of the limbs in different target objects is different from that of the robot is eliminated, which facilitates the subsequent mapping of the three-dimensional skeleton model to the joint space.

In one embodiment, step 101 may perform sub-steps, and the flow is shown in FIG. 7 :

Step 1011: Input the image data into the preset first neural network model to obtain the two-dimensional skeleton data of the target posture. The first neural network model is used to identify the target posture of the target object in the image data, and generate a corresponding target posture based on the target posture. 2D skeleton data.

Skeleton extraction is mainly completed by two neural networks. The first neural network accepts image input to complete the recognition of target objects in image data and the extraction of two-dimensional skeleton data. The two-dimensional skeleton data includes each joint point of the target object and the position of each joint point.

Step 1012: Input the two-dimensional skeleton data into a preset second neural network model to obtain a three-dimensional skeleton model corresponding to the target pose.

The input of the second neural network is the 2D skeleton node output by the first neural network, which outputs a 3D skeleton model.

In this embodiment, since the neural network model has strong learning energy, two neural network models are used to obtain the three-dimensional skeleton model, which can improve the accuracy and applicability of the three-dimensional skeleton model.

In one embodiment, before step 1011 is performed, the following steps may also be performed, and the flow is shown in FIG. 8 :

Step 1011-1: Collect video data of the target object.

Specifically, the video data of the target object can be collected in real time. The video data of the target object collected by other devices may also be acquired in real time.

Step 1011-2: Obtain image data from video data.

Image data can be obtained from the video data, and steps 101 to 104 are performed on the image data, so that the robot can present the target pose of the target object. For continuous image data, the robot can track and learn the pose of the target object.

The above embodiments can be combined with each other and referenced to each other. For example, the following is an example of the combination of the various embodiments, but it is not limited thereto; the various embodiments can be arbitrarily combined into a new embodiment under the premise of not contradicting each other.

In one embodiment, as shown in FIG. 9 , it is a flow chart of acquiring the rotation angle information of the joint and performing collision detection on the rotation angle information of the joint.

Step 101: Acquire a three-dimensional skeleton model corresponding to the target posture according to the target posture of the target object in the image data.

In one embodiment, FIG. 10 is a flowchart of adding filtering processing in FIG. 9 .

In one embodiment, as shown in FIG. 11 , the flowchart of adding the normalization process to the three-dimensional skeleton model in FIG. 10 is shown.

Sub-step 102-1: Normalize the three-dimensional skeleton model.

In one embodiment, as shown in FIG. 12 , the flowchart of acquiring a three-dimensional skeleton model in FIG. 11 is added.

Sub-step 102-1: Normalize the three-dimensional skeleton model.

Sub-step 1041: Perform filtering processing on the rotation angle information of each joint.

Sub-step 1042: Control the rotation of the corresponding joint according to the processed rotation angle information of each joint to form a target pose.

In one embodiment, as shown in FIG. 13 , the flowchart of acquiring image data in FIG. 12 is added.

Step 1011-1: Collect video data of the target object.

Step 1011-2: Obtain image data from video data.

Sub-step 102-1: Normalize the three-dimensional skeleton model.

FIG. 14 is a schematic diagram of a control device for a robot posture provided in another embodiment of the present disclosure. The robot posture control device includes: a model acquisition module 201 , a posture acquisition module 202 , a rotation angle acquisition module 203 and a motion control module 204 . in:

The model obtaining module 201 is configured to obtain a three-dimensional skeleton model corresponding to the target pose according to the target pose of the target object in the image data.

Specifically, the image data may be an image captured by a robot, or image data extracted from video data, for example, a frame of image in the video data is used as the image data. The target object can be a human body, an animal, etc. The robot in this embodiment is a multi-joint robot, such as a human-type robot, an animal-type robot, and the like. The action gesture of the target object can be extracted as the target gesture by identifying the target object in the image data. Two-dimensional skeleton data corresponding to the target pose is acquired, and a three-dimensional skeleton model of the target object can be constructed based on the two-dimensional skeleton data.

The model acquisition module is specifically configured to: input the image data into a preset first neural network model to obtain two-dimensional skeleton data of the target posture, and the first neural network model is used to identify the image The target posture of the target object in the data, and the corresponding two-dimensional skeleton data is generated based on the target posture; the two-dimensional skeleton data is input into the preset second neural network model, and the corresponding target posture is obtained. 3D Skeleton Model.

The attitude acquisition module 202 is used to map the three-dimensional skeleton model into the joint space of the robot, and obtain the attitude features of the robot and the skeleton attitude features of the three-dimensional skeleton model.

Specifically, the space formed by all the joint vectors is the joint space corresponding to the current robot arm, and the joint space corresponding to the torso of the robot may also be used as the joint space of the robot. The space angle between adjacent parts in the 3D skeleton model is the same as the space angle between adjacent parts corresponding to the robot, and the robot can present the same target pose as the 3D skeleton model. The 3D skeleton model and the robot are not in the same coordinate system. In this example, the 3D skeleton model is mapped to the joint space of the robot, and the pose feature of the robot is obtained. The pose feature includes the position of the vector of each joint of the robot and the 3D skeleton model. The skeleton pose feature includes the vector formed by each key point in the skeleton. For example, the magnitude of the vector where the joint 11 is located as shown in FIG. 2 may be the coordinate difference between the joint point 11 and the joint point 12 , and the direction may be the execution of the joint point 12 from the joint point.

The rotation angle acquisition module 203 is configured to adjust the posture feature of the robot to a target position matching the skeleton posture feature, and acquire the rotation angle information of each joint of the robot.

The rotation angle acquisition module is specifically used for: dividing the posture feature of the robot into a plurality of mapping parts; performing the following processing for each mapping part: transforming the position of the vector of each joint in the mapping part into the The position of the vector where the corresponding key point in the skeleton pose feature is located, and the rotation angle of the joint is obtained as the rotation angle information of the joint. Wherein, the plurality of mapping parts include: trunk, limbs, head and waist.

The motion control module 204 is configured to control the motion of the corresponding joint of the robot according to the rotation angle information of each joint to form the target posture.

The motion control module is specifically configured to: filter the rotation angle information of each joint; and control the rotation of the corresponding joint according to the processed rotation angle information of each joint to form the target posture.

In addition, in this embodiment, the control device for the robot posture further includes:

a motion data module, used for taking the corner information of each described joint as the motion data of the current frame;

a collision detection module for performing collision detection on the motion data of the current frame;

The motion control sub-module is configured to determine and execute the step of controlling the motion of the corresponding joint according to the rotation angle information of each joint to form the target posture if no collision is detected.

The device also includes:

The normalization module is used for normalizing the three-dimensional skeleton model.

The device also includes:

a video collection module, used for collecting the video data of the target object;

An image acquisition module, configured to acquire the image data from the video data.

Another embodiment of the present disclosure further provides a robot, whose structural block diagram is shown in FIG. 15 , including: at least one processor 301 ; and a memory 302 communicatively connected to the at least one processor 301 ; wherein the memory 302 stores Instructions executable by the at least one processor 301, the instructions are executed by the at least one processor 301, so that the at least one processor 301 can execute the above-mentioned method for controlling the posture of the robot.

The memory 302 and the processor 301 are connected by a bus, and the bus may include any number of interconnected buses and bridges, and the bus links one or more processors 301 and various circuits of the memory 302 together. The bus may also link together various other circuits, such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface between the bus and the transceiver. A transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing a means for communicating with various other devices over a transmission medium. The data processed by the processor is transmitted on the wireless medium through the antenna, and further, the antenna also receives the data and transmits the data to the processor.

Processor 301 is responsible for managing the bus and general processing, and may also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions. Instead, memory 302 may be used to store data used by the processor in performing operations.

In addition, an embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned method for controlling the attitude of the robot is implemented.

In addition, an embodiment of the present disclosure also provides a computer program, including instructions, which, when run on a computer, cause the computer to execute the above-mentioned method for controlling the posture of a robot.

Those skilled in the art can understand that all or part of the steps in the method of the above embodiments can be completed by instructing the relevant hardware through a program. The program is stored in a storage medium and includes several instructions to make a device (which may be a single-chip microcomputer) , chip, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

Those skilled in the art can understand that the above-mentioned embodiments are specific examples for realizing the present invention, and in practical applications, various changes in form and details can be made without departing from the spirit and the spirit of the present invention. scope.

Claims

A method for controlling the attitude of a robot, comprising:

According to the target posture of the target object in the image data, obtain a three-dimensional skeleton model corresponding to the target posture;

mapping the three-dimensional skeleton model into the joint space of the robot, and obtaining the robot's posture feature and the skeleton posture feature of the three-dimensional skeleton model;

Adjust the posture feature of the robot to a target position matching the skeleton posture feature, and obtain the rotation angle information of each joint of the robot;

The movement of the robot corresponding to the joint is controlled according to the rotation angle information of each joint to form the target posture.
The method for controlling robot posture according to claim 1, wherein the obtaining a three-dimensional skeleton model corresponding to the target posture according to the target posture of the target object in the image data comprises:

Inputting the image data into a preset first neural network model to obtain two-dimensional skeleton data of the target posture, and the first neural network model is used to identify the target posture of the target object in the image data, and generate corresponding two-dimensional skeleton data based on the target posture;

Inputting the two-dimensional skeleton data into a preset second neural network model to obtain a three-dimensional skeleton model corresponding to the target posture.
The method for controlling robot posture according to claim 1, wherein adjusting the posture feature of the robot to a target position matching the skeleton posture feature, and acquiring the rotation angle information of each joint of the robot, comprising:

dividing the posture feature of the robot into a plurality of mapping parts;

The following processing is performed for each mapping part: the position of the vector of each joint in the mapping part is transformed into the position of the vector of the corresponding key point in the skeleton pose feature, and the rotation angle of the joint is obtained as the joint corner information.
The method for controlling the posture of a robot according to claim 3, wherein the plurality of mapping parts comprise: a trunk part, a limb part, a head part and a waist part.
The control method of robot posture according to claim 1, is characterized in that, described method also comprises:

Taking the corner information of each described joint as the motion data of the current frame;

performing collision detection on the motion data of the current frame;

If no collision is detected, it is determined to execute the step of controlling the motion of the corresponding joint according to the rotation angle information of each joint to form the target posture.
The method for controlling the posture of a robot according to any one of claims 1 to 5, wherein the movement of the robot corresponding to the joint is controlled according to the rotation angle information of each joint to form the target posture ,include:

filtering the rotation angle information of each of the joints;

The rotation of the corresponding joint is controlled according to the processed rotation angle information of each joint to form the target posture.
The method for controlling the attitude of a robot according to any one of claims 1 to 5, wherein the three-dimensional skeleton model is mapped to the joint space of the robot, and the attitude features of the robot and the robot are acquired. Before the skeleton pose feature of the three-dimensional skeleton model, the method further includes:

The three-dimensional skeleton model is normalized.
The method for controlling robot posture according to claim 1, characterized in that, before the image data is input into the preset first neural network model and the two-dimensional skeleton data of the target posture is obtained, the method further comprises: include:

collecting video data of the target object;

The image data is obtained from the video data.
A control device for robot posture, comprising:

a model acquisition module, configured to acquire a three-dimensional skeleton model corresponding to the target posture according to the target posture of the target object in the image data;

an attitude acquisition module, configured to map the three-dimensional skeleton model into the joint space of the robot, and acquire the attitude features of the robot and the skeleton attitude features of the three-dimensional skeleton model;

A turning angle acquisition module, for adjusting the posture feature of the robot to a target position matched with the skeleton posture feature, to obtain the turning angle information of each joint of the robot;

A motion control module is configured to control the motion of the corresponding joint of the robot according to the rotation angle information of each joint to form the target posture.
The device according to claim 9, wherein the model acquisition module is specifically used for:

Inputting the image data into a preset first neural network model to obtain two-dimensional skeleton data of the target posture, and the first neural network model is used to identify the target posture of the target object in the image data, and generate corresponding two-dimensional skeleton data based on the target posture;

Inputting the two-dimensional skeleton data into a preset second neural network model to obtain a three-dimensional skeleton model corresponding to the target posture.
The device according to claim 9, wherein the corner acquiring module is specifically used for:

dividing the posture feature of the robot into a plurality of mapping parts;

The following processing is performed for each mapping part: the position of the vector of each joint in the mapping part is transformed into the position of the vector of the corresponding key point in the skeleton pose feature, and the rotation angle of the joint is obtained as the joint corner information.
11. The apparatus of claim 11, wherein the plurality of mapped parts comprises: a torso, a limb, a head, and a waist.
The apparatus according to claim 9, wherein the apparatus further comprises:

a motion data module, used for taking the corner information of each described joint as the motion data of the current frame;

a collision detection module for performing collision detection on the motion data of the current frame;

The motion control sub-module is configured to determine and execute the step of controlling the motion of the corresponding joint according to the rotation angle information of each joint to form the target posture if no collision is detected.
The device according to any one of claims 9 to 13, wherein the control motion module is specifically used for:

filtering the rotation angle information of each of the joints;

Control the rotation of the corresponding joints according to the processed rotation angle information of the joints to form the target posture.
The device according to any one of claims 9 to 13, wherein the device further comprises:

The normalization module is used for normalizing the three-dimensional skeleton model.
The device according to claim 1, wherein the device further comprises:

a video collection module, used for collecting the video data of the target object;

An image acquisition module, configured to acquire the image data from the video data.
A robot, characterized in that it includes:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any one of claims 1-8 The control method of robot attitude.
A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, the method for controlling the attitude of a robot according to any one of claims 1 to 8 is implemented.
A computer program comprising instructions, when run on a computer, causes the computer to execute the method for controlling the attitude of a robot according to any one of claims 1-8.