WO2022134702A1

WO2022134702A1 - Action learning method and apparatus, storage medium, and electronic device

Info

Publication number: WO2022134702A1
Application number: PCT/CN2021/120457
Authority: WO
Inventors: 张站朝; 黄晓庆
Original assignee: 达闼机器人股份有限公司
Priority date: 2020-12-24
Filing date: 2021-09-24
Publication date: 2022-06-30
Also published as: CN112847336A; CN112847336B

Abstract

The present disclosure relates to an action learning method and apparatus, a storage medium, and an electronic device. Said method comprises: acquiring human action image data of a target object; determining joint motion data of a robot corresponding to the human action image data; determining, by means of simulation data of a digital twin model of the robot, whether the joint motion data conforms to an atomic action constraint corresponding to the robot; and in cases where it is determined that the joint motion data conforms to the atomic action constraint corresponding to the robot, determining the joint motion data as the atomic action of the robot. By means of the described technical solution, a set of human actions can be converted into atomic actions of a plurality of different robots, and learning of human actions is completed on the basis of a digitized and virtual robot digital twin model, without the need of a physical robot, thereby greatly reducing costs of generating robot actions.

Description

Action learning method, device, storage medium and electronic device

CROSS-REFERENCE TO RELATED APPLICATIONS

This disclosure claims the priority of a Chinese patent application with application number 202011552528.1 and titled "Action Learning Method, Apparatus, Storage Medium and Electronic Equipment" filed with the China Patent Office on December 24, 2020, the entire contents of which are incorporated by reference in in this disclosure.

technical field

The present disclosure relates to the field of robotics, and in particular, to an action learning method, device, storage medium, and electronic device.

Background technique

Each action of the robot is a process of combining multiple joints. The method for robots to learn human actions is usually to first capture human motion postures through human wearable devices. Then control the physical robot to learn the action according to the human motion posture. The cost of action learning for physical robots is high, and the same action needs to be learned repeatedly on different robots. And a common way of motion capture is to set trackers, such as inertial measurement units (IMUs), at key parts of moving objects. The three-dimensional position and orientation information at each sensor of the inertial motion capture can be recorded and displayed in real time (with a slight delay), and the cost of small-scale use is relatively reasonable. However, the defects are also obvious. It needs to be demagnetized every time it is used, and it is often limited by the range and accuracy of the magnetic field, the spatial positioning is not accurate, and due to equipment reasons, there are many action limitations. And one person is a set of equipment, and the cost of multiple people increases exponentially.

SUMMARY OF THE INVENTION

The purpose of the present disclosure is to provide an action learning method, device, storage medium and electronic device, which can convert a group of human actions into atomic actions of multiple different robots, and the learning of human actions is based on digital and virtual robots The completion of the digital twin model does not require a physical robot, which can greatly reduce the cost of generating robot actions.

In order to achieve the above object, the present disclosure provides an action learning method, the method includes:

Obtain the human action image data of the target object;

determining the joint motion data of the robot corresponding to the human body motion image data;

Determine whether the joint motion data complies with the atomic motion constraints corresponding to the robot by using the simulation data of the digital twin model of the robot;

When it is determined that the joint motion data complies with the atomic motion constraint conditions corresponding to the robot, the joint motion data is determined as the atomic motion of the robot.

Optionally, the determining the joint motion data of the robot corresponding to the human motion image data includes:

performing denoising processing on the human action image data;

Corresponding three-dimensional gesture motion data is determined according to the denoised human motion image data, and the three-dimensional gesture motion data includes the motion data of each three-dimensional key point corresponding to the target object, and the three-dimensional key point for constructing the three-dimensional pose of the target object;

The joint motion data of the robot is obtained by mapping the three-dimensional posture motion data.

Optionally, judging whether the joint motion data complies with the atomic motion constraints corresponding to the robot by using the simulation data of the digital twin model of the robot includes:

Based on the simulation data of the digital twin model, it is determined whether the joint motion data meets the motion compliance conditions, and the motion compliance conditions include that the motion corresponding to the joint motion data is the atomic motion, and the joint motion data does not There is a joint movement that exceeds the physical limit of the joint, and the digital twin model does not have collision and imbalance when executing the joint movement data;

When the joint motion data does not meet any one of the motion compliance conditions, it is determined that the joint motion data does not meet the atomic motion constraint conditions corresponding to the robot.

Optionally, it is described whether the joint motion data is judged by the simulation data of the digital twin model of the robot to meet the atomic action constraint corresponding to the robot and also includes:

When the joint motion data satisfies all of the motion compliance conditions, determine whether the joint motion data satisfies motion optimization conditions based on the simulation data of the digital twin model, where the motion optimization conditions include that the joint motion data complies with the robot The motions corresponding to the kinematic laws and the joint motion data are smooth and smooth;

When the joint motion data does not meet the motion optimization condition, it is determined that the joint motion data does not meet the atomic motion constraint condition corresponding to the robot.

Optionally, judging whether the joint motion data complies with the atomic motion constraints corresponding to the robot by using the simulation data of the digital twin model of the robot further includes:

When the joint motion data satisfies the motion optimization condition and the motion compliance condition, execute the joint motion data through the digital twin model, and display the motion picture of the digital twin model;

In the case of receiving the first manual confirmation instruction, the robot is driven to execute the joint motion data through the digital twin model, and the first manual confirmation instruction is input by the user in response to the motion picture of the digital twin model instruction, used to indicate that the motion picture of the digital twin model meets the requirements;

In the case of receiving a second manual confirmation instruction, it is determined that the joint motion data complies with the atomic motion constraint corresponding to the robot, and the second manual confirmation instruction is determined by the user in response to the execution action of the robot. The input instructions are used to instruct the robot to perform actions that meet the requirements.

Optionally, the method further includes: when it is determined that the joint motion data does not meet the atomic motion constraint conditions corresponding to the robot through the simulation data of the digital twin model of the robot, correcting the joint motion data , and judge whether the revised joint motion data complies with the atomic motion constraints corresponding to the robot through the simulation data of the digital twin model of the robot, until it is determined that the joint motion data complies with the Describe the atomic action constraints.

The modifying the joint motion data includes:

When the joint motion data does not meet any one of the motion compliance conditions, receiving a manual modification instruction to modify the joint motion data;

When the joint motion data does not meet the motion optimization condition, perform a first optimization on the joint motion data, and re-judg whether the optimized joint motion data complies with the atomic motion constraint conditions corresponding to the robot, and the The first optimization includes performing data optimization on the joint motion data according to inverse kinematics, performing data optimization on the joint motion data according to confrontational generation network training, performing data optimization on the joint motion data according to data fitting, and performing data optimization on the joint motion data according to an interpolation algorithm. One or more of data optimizations are performed on the articulation data.

Optionally, when the robot is in an action imitation mode, the method further includes:

Judging by the simulation data of the digital twin model of the robot whether the joint motion data conforms to the action imitation conditions corresponding to the robot;

In the case that the joint motion data does not conform to the motion imitation conditions corresponding to the robot, a second optimization is performed on the joint motion data, and it is re-determined whether the optimized joint motion data conforms to the motion corresponding to the robot Imitating conditions, the second optimization includes performing data optimization on the joint motion data according to inverse kinematics, performing data optimization on the joint motion data according to adversarial generation network training, and performing data optimization on the joint motion data according to data fitting. One or more of optimizing and performing data optimization on the joint motion data according to an interpolation algorithm;

In the case that the joint motion data conforms to the motion simulation conditions corresponding to the robot, the robot is driven to execute the joint motion data through the digital twin model, and returns to the initial stage of the robot after the motion execution is completed. state.

The present disclosure also provides an action learning device, the device comprising:

an acquisition module, used to acquire human motion image data of the target object;

a determination module for determining joint motion data of the robot corresponding to the human body motion image data;

a first judging module, used for judging whether the joint motion data complies with the atomic motion constraints corresponding to the robot through the simulation data of the digital twin model of the robot;

The learning module is configured to determine the joint motion data as the atomic motion of the robot when it is determined that the joint motion data conforms to the atomic motion constraint conditions corresponding to the robot.

The present disclosure also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the steps of the above-described method.

The present disclosure also provides an electronic device, comprising:

a memory on which a computer program is stored;

A processor for executing the computer program in the memory to implement the steps of the above-described method.

The present disclosure also provides a computer program product comprising a computer program executable by a programmable apparatus, the computer program having code for performing the steps of the above-described method when executed by the programmable apparatus part.

Through the above technical solution, the joint motion data of the robot can be determined by the acquired human body motion image data of the target object, so that the corresponding robot can be determined according to a set of human body motion image data of the target object. The joint motion data accelerates the learning speed of different robots for actions; and, according to the digital twin model of the digital and virtual robot, it is possible to judge online whether the joint motion data conforms to the constraints of atomic motions, without the need for a physical robot. Dramatically reduce the cost of generating robot actions.

Other features and advantages of the present disclosure will be described in detail in the detailed description that follows.

Description of drawings

The accompanying drawings are used to provide a further understanding of the present disclosure, and constitute a part of the specification, and together with the following detailed description, are used to explain the present disclosure, but not to limit the present disclosure. In the attached image:

FIG. 1 is a flowchart of an action learning method according to an exemplary embodiment of the present disclosure.

Fig. 2 is a flowchart of an action learning method according to another exemplary embodiment of the present disclosure.

Fig. 3 is a flowchart of an action learning method according to another exemplary embodiment of the present disclosure.

Fig. 4 is a flowchart of an action learning method according to another exemplary embodiment of the present disclosure.

FIG. 5 is a structural block diagram of an action learning apparatus according to an exemplary embodiment of the present disclosure.

Fig. 6 is a block diagram of an electronic device according to an exemplary embodiment.

Fig. 7 is a block diagram of an electronic device according to an exemplary embodiment.

Detailed ways

The specific embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure, but not to limit the present disclosure.

FIG. 1 is a flowchart of an action learning method according to an exemplary embodiment of the present disclosure. As shown in FIG. 1 , the method includes steps 101 to 102 .

In step 101, the human body motion image data of the target object is acquired.

The target object may be any object within the acquisition range that can perform the action to be learned, such as a human being, an animal, or other robots capable of performing actions, and so on.

Human motion image data can be 2D data or 3D data, and can be obtained by any RGB camera, depth camera or lidar, for example, the depth camera can be a TOF-based depth camera, or an infrared-based depth camera, Or passive vision-based monocular depth cameras or binocular depth cameras, etc.

The device used to obtain human motion image data can be set at any position on the robot's head, chest, waist, etc., so that the robot can learn the movements made by the target object in front of it, or it can also be used not set on the robot. The human body motion image data can be acquired by the acquisition device on the scene environment, for example, the human body motion image data can be acquired by other devices set in the scene environment.

In the process of acquiring the human body motion image data, the position of the target object may be located first, and the human body motion image data of the target object may be acquired; or after the target object exists within the acquisition range, Image data is acquired for all objects within the range, and human motion image data related to the motion of the target object is extracted from all the acquired image data.

In step 102, the joint motion data of the robot corresponding to the human body motion image data is determined.

Since the joint structures corresponding to different types of robots are different, for a same action, the joint motion data corresponding to different robots will also be different. Therefore, after acquiring the human action image data, it is necessary to determine the robot to be learned. For example, the robot may be a single-arm robot, a dual-arm robot, a single-leg robot, or a biped robot, a robot with two-finger multi-joint hands, or a multi-fingered hand. Articulated robots and more.

The content included in the joint motion data can be determined according to the actual joints of the robot, as long as it is learned from the human body motion image data of the target object. For example, if the target object is a person, and the actions corresponding to the human body motion image data include the actions of the whole body such as legs and hands, and the robot is a monopod robot with two arms, it is determined that the obtained motion is the same as the action of the human body. The joint motion data corresponding to the image data may only include the motion data of the upper body such as the head, arms, and waist, so as to ensure that the motion corresponding to the joint motion data is similar to the motion corresponding to the human motion image data as much as possible. It can also ensure the balance, stability and safety of the robot.

Among them, the joint motion data is also composed of multiple sets of joint rotation angle values with time series attributes. For example, the joint motion data for a 6-axis robot arm can be:

[(0)(1:1:3.68)(2:1:3.09)(3:1:2.88)(4:1:1.98)(5:1:1.58)(6:1:1.87)],

[(100)(1:1:3.31)(2:1:2.85)(3:1:2.34)(4:1:1.54)(5:1:1.35)(6:1:0.85)],

Among them, (0) (100) represents the timestamp, (0) can be the starting time point, (100) can be the time interval from the last timestamp (unit can be, for example, milliseconds); (1:1:3.68) The first 1 represents the first joint in the 6-axis robot arm, the second 1 represents the forward rotation, and (3.68) represents the rotation angle position, where the joint sequence is defined in the robot, and the corresponding flag can be directly used For example, 2 represents the second joint, and 3 represents the third joint; in addition to the above-mentioned forward rotation represented by 1 on the corresponding sign bit, reverse rotation and reverse rotation can also be performed. can be represented by 0.

In step 103, it is judged whether the joint motion data complies with the atomic motion constraint conditions corresponding to the robot according to the simulation data of the digital twin model of the robot.

The digital twin model is a virtual robot obtained by digitizing the physical model of the robot. The physical model of the robot includes physical simulation models such as appearance, motors of each joint, sensors, gravity, and collision. The digital twin model can also receive the robot's physical simulation model. Perceives feedback and is able to synchronously control the robot. The digital twin model may be stored on the server side, and the server side judges whether the joint motion data satisfies the atomic action constraint condition through the simulation data of the digital twin model.

The atomic action constraint condition is used to constrain the action corresponding to the joint motion data, to avoid actions that affect the stability and safety of the robot, or actions that are not atomic actions to be determined as the atomic actions of the robot. The atomic action constraints can be set according to the actual situation of the robot. The atomic action is also the basic action used to compose other actions, such as raising the hand, letting go of the hand, etc. When the robot executes the atomic action, the execution process cannot be planned or split.

In step 104, when it is determined that the joint motion data complies with the atomic motion constraint conditions corresponding to the robot, the joint motion data is determined as the atomic motion of the robot.

Multiple atomic actions can form new robot behaviors, and the learned atomic actions can be stored in the robot or in a cloud server.

Fig. 2 is a flowchart of an action learning method according to another exemplary embodiment of the present disclosure. As shown in FIG. 2 , the method further includes steps 201 to 203 .

In step 201, denoising processing is performed on the human body motion image data. The denoising process can be performed by any denoising method, and by denoising and optimizing the human action image data, the actions included in the human action image data can be made more accurate.

In step 202, the corresponding three-dimensional posture motion data is determined according to the denoised human body motion image data, and the three-dimensional posture motion data includes the motion data of each three-dimensional key point corresponding to the target object, The three-dimensional key points are used to construct the three-dimensional pose of the target object.

That is, after obtaining the human body motion image data after the denoising process, the three-dimensional key points used to construct the three-dimensional posture of the target object can be determined in the human body motion image data, and then the three-dimensional key points can be used to construct the three-dimensional key points of the human body. The motion data of the motion image data, the determined three-dimensional gesture motion data corresponding to the human motion image data. The three-dimensional attitude motion data is composed of three-dimensional position values of multiple groups of multiple key points with time series attributes.

In step 203, the joint motion data of the robot is obtained by mapping the three-dimensional posture motion data.

After the robot is determined, joint information in the robot can be obtained, and then the three-dimensional pose motion data can be mapped to corresponding joints in the robot. The mapping method can be performed through a deep learning neural network, wherein different deep learning neural networks can be trained for different robots respectively, and a corresponding deep learning neural network can be selected for mapping according to the robot.

Fig. 3 is a flowchart of an action learning method according to another exemplary embodiment of the present disclosure. As shown in FIG. 3 , the method further includes steps 301 to 307 .

In step 301, based on the simulation data of the digital twin model, it is judged whether the joint motion data satisfies the motion compliance condition, if yes, go to step 303, if not, go to step 302. The motion compliance conditions include that the motion corresponding to the joint motion data is the atomic motion, there is no joint motion in the joint motion data that exceeds the physical limit of the joint, the digital twin model is executing all No collision or imbalance occurred when the joint motion data was described. When the joint motion data does not satisfy any one of the motion compliance conditions, it can be determined that the joint motion data does not satisfy the motion compliance condition, that is, does not satisfy the atomic motion constraint condition corresponding to the robot .

There are various methods for judging whether the action corresponding to the joint motion data is the atomic action. For example, the feature comparison between the joint motion data and the preset motion data of various atomic actions can be performed to obtain the feature similarity between the joint motion data and the motion data of various atomic actions. Or if the feature similarity between the motion data of two or more atomic actions exceeds a preset threshold, it can be determined that the action corresponding to the joint motion data is not the atomic action that needs to be learned.

The physical limits of the above joints are related to the specific joints in the robot. When judging whether there is a joint motion exceeding the physical limit of the joint in the joint motion data, the physical limit corresponding to all the joints in the robot can be obtained separately, and then the joint motion data of the joint is involved. exercise data for comparison.

The collision of the above-mentioned digital twin model may include self-collision. For example, the model arm overlaps with the model leg during movement. When the digital twin model is in the same 3D environment scene as the robot, a collision may also occur. Including the model colliding with surrounding objects.

In step 302, that is, when the joint motion data does not meet any of the motion compliance conditions, receive a manual modification instruction to modify the joint motion data; and re-judg the modified joint motion data Whether the atomic action constraint corresponding to the robot is met, that is, return to step 301 . The manual modification instruction may be a modification instruction generated by repairing action data based on the digital twin body model by a software tool, and the software tool may be a visualization tool, and the action performed by the digital twin lift model according to the joint motion data under modification may be is visible.

In step 303, that is, when the joint motion data satisfies all the motion compliance conditions, it is judged whether the joint motion data satisfies the motion optimization conditions based on the simulation data of the digital twin model, and if so, go to step 303 305, if not, go to step 304; the motion optimization condition includes that the joint motion data conforms to the kinematics of the robot and the motion corresponding to the joint motion data is smooth and smooth. When the joint motion data does not meet the motion optimization condition, it may also be determined that the joint motion data does not meet the atomic motion constraint condition corresponding to the robot.

The robot kinematics law may also be determined according to different robots. If the joint motion data does not satisfy the kinematics law of the robot or the preset fluency and smoothness, the actions performed by the robot will be stuck and unreasonable. Therefore, in order to ensure that the joint motion data is within the When executed by the robot, the smoothness and flow of the action can be guaranteed, and it will be judged whether the joint motion data satisfies the motion optimization condition.

Wherein, the judgment order of the above motion optimization condition and the above motion compliance condition may be adjusted according to the actual situation, and the sequence is not limited in the present disclosure.

In step 304, that is, when the joint motion data does not meet the motion optimization conditions, the first optimization is performed on the joint motion data; Atomic action constraints, that is, return to step 301 . The first optimization method may include optimizing the joint motion data according to inverse kinematics, optimizing the joint motion data according to adversarial generation network training, optimizing the joint motion data according to data fitting, One or more of optimizing data optimization, optimizing the joint motion data according to an interpolation algorithm, or may also include other optimization methods, which are not limited in this disclosure, and the above-mentioned first optimization can be performed without manual labor. achieved without intervention.

In step 305, that is, when the joint motion data satisfies both the motion compliance condition and the motion optimization condition, execute the joint motion data through the digital twin model, and analyze the digital twin model for the joint motion data. The moving picture is displayed.

In step 306, in the case of receiving a first manual confirmation instruction, drive the robot to execute the joint motion data through the digital twin model, and the first manual confirmation instruction is that the user responds to the digital twin model The instruction input by the moving picture of the digital twin model is used to indicate that the moving picture of the digital twin model meets the requirements.

When the joint motion data satisfies the motion optimization condition, that is, the joint motion data satisfies both the motion compliance condition and the motion optimization condition, in this case, it can be executed through the digital twin model. The joint motion data is given to the user, so that the user can manually confirm whether the joint motion data meets the requirements of action learning.

If the user thinks that the moving picture of the digital twin model meets the requirements, he can confirm by entering the first manual confirmation command, but the user still needs to make changes when he thinks that the moving picture of the digital twin model does not meet the requirements Next, a manual modification instruction may be directly inputted to modify a bunch of the joint motion data, that is, step 302 may be turned to modify the joint motion data. Alternatively, the user may directly determine that the joint motion data does not meet the constraints of the atomic motion corresponding to the robot, thereby ending the learning of the motion.

In step 307, when a second manual confirmation instruction is received, it is determined that the joint motion data complies with the atomic motion constraint corresponding to the robot, and the second manual confirmation instruction is that the user responds to the The command input by the robot's execution action is used to indicate that the robot's execution action meets the requirements.

When it is determined that the joint motion data complies with the atomic motion constraint conditions corresponding to the robot, the joint motion data may be determined as the atomic motion of the robot. However, if the user thinks that the action does not meet the requirements after viewing the action performed by the robot according to the joint motion data, the user can also return to step 302 to modify the joint motion data again through the manual modification instruction, and repeat the procedure. The judgment of whether the constraints of the atomic action are satisfied. Alternatively, the user may directly determine that the joint motion data does not meet the constraints of the atomic motion corresponding to the robot, thereby ending the learning of the motion.

Through the above technical solution, the joint motion data can be repaired or optimized by judging the motion compliance conditions and the motion optimization conditions, and the reliability of the learned atomic motion can also be ensured through double manual confirmation. sex.

FIG. 4 is a flowchart of an action learning method according to another exemplary embodiment of the present disclosure. As shown in FIG. 4 , the method further includes steps 401 to 403 .

In step 401, when the robot is in an action imitation mode, it is judged whether the joint motion data conforms to the action imitation condition corresponding to the robot according to the simulation data of the digital twin model of the robot. In the action imitation mode, the robot can learn and imitate the action made by the target object in real time.

The motion imitation conditions may include, for example, the above-mentioned motion compliance conditions, or may also include the above-mentioned motion optimization conditions. When judging whether the motion imitation conditions are met, the joint motion data can be scored according to the specific conditions of the motion imitation conditions. For example, if all the motion compliance conditions are met, 50 points will be awarded, and the robot that meets the motion optimization conditions will be scored. 10 points are awarded for any of the laws of kinematics, the preset smoothness, and the preset smoothness, and so on. Therefore, it can be judged whether the action imitation condition is satisfied according to the score. Alternatively, it is also possible to first judge whether the joint motion data meets the motion compliance condition, and when the motion compliance condition is met and the score exceeds a certain threshold, it is judged that the motion imitation condition is met.

In step 402, in the case that the joint motion data does not meet the motion imitation conditions corresponding to the robot, a second optimization is performed on the joint motion data, and it is re-determined whether the optimized joint motion data meets the requirements of the robot. Describe the corresponding action imitation conditions of the robot. In the case that the action imitation condition is not met, the second optimization performed may include optimizing data optimization of the joint motion data according to inverse kinematics, optimizing data optimization of the joint motion data according to confrontation generation network training , one or more of optimizing data optimization for the joint motion data according to data fitting, optimizing data optimization for the joint motion data according to an interpolation algorithm, and performing smooth interpolation, for example, on the joint motion data. The methods included in the second optimization and the above-mentioned first optimization may be the same or different.

In step 403, in the case that the joint motion data meets the motion imitation conditions corresponding to the robot, the robot is driven to execute the joint motion data through the digital twin model, and returns to the robot after the motion execution is completed. The initial state of the robot.

Through the above technical solutions, the function of imitating and learning the movements of the target object by the robot in real time can be realized under the condition of ensuring the safety of the robot.

FIG. 5 is a structural block diagram of an action learning apparatus according to an exemplary embodiment of the present disclosure. As shown in FIG. 5 , the device includes: an acquisition module 10 for acquiring human motion image data of a target object; a determination module 20 for determining joint motion data of the robot corresponding to the human motion image data; a first The judgment module 30 is used for judging whether the joint motion data complies with the atomic motion constraints corresponding to the robot through the simulation data of the digital twin model of the robot; the learning module 40 is used for judging that the joint motion data meets the In the case of the atomic motion constraint corresponding to the robot, the joint motion data is determined as the atomic motion of the robot.

In a possible implementation manner, the determining module 20 includes: a denoising sub-module for performing denoising processing on the human action image data; a determining sub-module for The human action image data determines the corresponding three-dimensional posture motion data, and the three-dimensional posture motion data includes the motion data of each three-dimensional key point corresponding to the target object, and the three-dimensional key point is used to construct the three-dimensional posture of the target object. Attitude; a mapping submodule for mapping the joint movement data of the robot according to the three-dimensional attitude movement data.

In a possible implementation manner, the first judging module 30 includes a first judging sub-module for judging whether the joint motion data satisfies the motion compliance condition based on the simulation data of the digital twin model, and the motion The compliance conditions include that the motion corresponding to the joint motion data is the atomic motion, the joint motion data does not contain a joint motion exceeding the physical limit of the joint, and the digital twin model executes the joint motion data when the joint motion data is executed. There is no collision and imbalance; and when the joint motion data does not meet any of the motion compliance conditions, it is determined that the joint motion data does not meet the atomic motion constraint conditions corresponding to the robot.

In a possible implementation manner, the first judging module 30 further includes a second judging sub-module, which judges based on the simulation data of the digital twin model when the joint motion data satisfies all the motion compliance conditions Whether the joint motion data satisfies the motion optimization conditions, the motion optimization conditions include that the joint motion data conforms to the laws of robot kinematics and the motion corresponding to the joint motion data is smooth and smooth; and if the joint motion data does not satisfy the When the motion optimization conditions are met, it is determined that the joint motion data does not conform to the atomic motion constraint conditions corresponding to the robot.

In a possible implementation manner, the first judging module 30 further includes a simulation sub-module, and when the joint motion data satisfies the motion optimization condition and the motion compliance condition, execute the execution by using the digital twin model The joint motion data is displayed, and the motion picture of the digital twin model is displayed; the action execution sub-module is used to drive the robot to execute all operations through the digital twin model when the first manual confirmation instruction is received. The joint motion data, the first manual confirmation instruction is the instruction input by the user in response to the motion picture of the digital twin model, and is used to indicate that the motion picture of the digital twin model meets the requirements; In the case of receiving a second manual confirmation instruction, it is determined that the joint motion data complies with the atomic motion constraint corresponding to the robot, and the second manual confirmation instruction is input by the user in response to the execution action of the robot The instructions are used to instruct the robot to perform actions that meet the requirements.

In a possible implementation manner, the apparatus further includes: a correction module, configured to determine that the joint motion data does not conform to the atomic motion constraint corresponding to the robot through the simulation data of the digital twin model of the robot When , the joint motion data is corrected, and it is judged by the simulation data of the digital twin model of the robot whether the corrected joint motion data complies with the atomic motion constraint conditions corresponding to the robot, until the joint is determined The motion data conforms to the atomic motion constraints corresponding to the robot.

In one possible implementation,

The first judging module 30 includes a first judging sub-module for judging whether the joint motion data satisfies the motion compliance condition based on the simulation data of the digital twin model, and the motion compliance condition includes the joint motion data. The corresponding action is the atomic action, the joint motion data does not have a joint motion exceeding the physical limit of the joint, and the digital twin model does not collide and unbalance when executing the joint motion data;

The first judging module 30 also includes a second judging sub-module, when the joint motion data satisfies all the motion compliance conditions, based on the simulation data of the digital twin model, it is judged whether the joint motion data satisfies the motion optimization. conditions, the motion optimization conditions include that the joint motion data conforms to the law of robot kinematics and the motion corresponding to the joint motion data is smooth and smooth;

The correction module also includes:

A first correction submodule, configured to receive a manual modification instruction to modify the joint motion data when the joint motion data does not meet any of the motion compliance conditions, and re-judg the modified joint motion Whether the data conforms to the atomic action constraints corresponding to the robot.

The second correction sub-module is configured to perform a first optimization on the joint motion data when the joint motion data does not meet the motion optimization condition, and re-judg whether the optimized joint motion data conforms to the corresponding robot The first optimization includes performing data optimization on the joint motion data according to inverse kinematics, performing data optimization on the joint motion data according to adversarial generation network training, and performing data optimization on the joint motion data according to data fitting. One or more of performing data optimization on the data and performing data optimization on the joint motion data according to an interpolation algorithm.

In a possible implementation manner, the apparatus further includes: a second judgment module, configured to judge the joint motion according to the simulation data of the digital twin model of the robot when the robot is in an action imitation mode Whether the data complies with the motion imitation conditions corresponding to the robot, the second optimization includes: performing data optimization on the joint motion data according to inverse kinematics; performing data optimization on the joint motion data according to the confrontation generation network training; One or more of data fitting to perform data optimization on the joint motion data, and data optimization for the joint motion data according to an interpolation algorithm; an optimization module for performing data optimization on the joint motion data when the joint motion data is inconsistent with the robot In the case of the corresponding action imitation conditions, the joint motion data is secondly optimized, and it is re-judged whether the optimized joint motion data conforms to the motion imitation conditions corresponding to the robot; the execution module is used to perform a second optimization on the joint motion data. When the motion data conforms to the motion imitation conditions corresponding to the robot, the robot is driven to execute the joint motion data through the digital twin model, and returns to the initial state of the robot after the motion execution is completed.

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

FIG. 6 is a block diagram of an electronic device 600 according to an exemplary embodiment. As shown in FIG. 6 , the electronic device 600 may include: a processor 601 and a memory 602 . The electronic device 600 may also include one or more of a multimedia component 603 , an input/output (I/O) interface 604 , and a communication component 605 .

The processor 601 is used to control the overall operation of the electronic device 600 to complete all or part of the steps in the above-mentioned action learning method. The memory 602 is used to store various types of data to support operations on the electronic device 600, such data may include, for example, instructions for any application or method operating on the electronic device 600, and application-related data, Such as contact data, messages sent and received, pictures, audio, video, and so on. The memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM for short), Electrically Erasable Programmable Read-Only Memory ( Electrically Erasable Programmable Read-Only Memory (EEPROM for short), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. Multimedia components 603 may include screen and audio components. The screen can be, for example, a touch screen, and the audio component is used for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in memory 602 or transmitted through communication component 605 . The audio assembly also includes at least one speaker for outputting audio signals. The I/O interface 604 provides an interface between the processor 601 and other interface modules, and the above-mentioned other interface modules may be a keyboard, a mouse, a button, and the like. These buttons can be virtual buttons or physical buttons. The communication component 605 is used for wired or wireless communication between the electronic device 600 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or one or more of them The combination is not limited here. Therefore, the corresponding communication component 605 may include: Wi-Fi module, Bluetooth module, NFC module and so on.

In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), digital signal processors (Digital Signal Processor, DSP for short), digital signal processing devices (Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor or other electronic components Implementation is used to perform the above-mentioned action learning method.

In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned action learning method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 602 including program instructions, and the above-mentioned program instructions can be executed by the processor 601 of the electronic device 600 to implement the above-mentioned action learning method.

FIG. 7 is a block diagram of an electronic device 700 according to an exemplary embodiment. For example, the electronic device 700 may be provided as a server. 7 , an electronic device 700 includes a processor 722 , which may be one or more in number, and a memory 732 for storing computer programs executable by the processor 722 . The computer program stored in memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 722 may be configured to execute the computer program to perform the above-described action learning method.

In addition, the electronic device 700 may also include a power supply component 726, which may be configured to perform power management of the electronic device 700, and a communication component 750, which may be configured to enable communication of the electronic device 700, eg, wired or wireless communication. Additionally, the electronic device 700 may also include an input/output (I/O) interface 758 . Electronic device 700 may operate based on an operating system stored in memory 732, such as Windows Server ^™ , Mac OS X ^™ , Unix ^™ , Linux ^™ , and the like.

In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned action learning method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 732 including program instructions, and the above-mentioned program instructions can be executed by the processor 722 of the electronic device 700 to implement the above-mentioned action learning method.

In another exemplary embodiment, there is also provided a computer program product comprising a computer program executable by a programmable apparatus, the computer program having, when executed by the programmable apparatus, for performing the above The code part of the action learning method.

The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings. However, the present disclosure is not limited to the specific details of the above-mentioned embodiments. Within the scope of the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure. These simple modifications all fall within the protection scope of the present disclosure.

In addition, it should be noted that each specific technical feature described in the above-mentioned specific implementation manner may be combined in any suitable manner under the circumstance that there is no contradiction. In order to avoid unnecessary repetition, various possible combinations are not described in the present disclosure.

In addition, the various embodiments of the present disclosure can also be arbitrarily combined, as long as they do not violate the spirit of the present disclosure, they should also be regarded as the contents disclosed in the present disclosure.

Claims

An action learning method, characterized in that the method comprises:

Obtain the human action image data of the target object;

determining the joint motion data of the robot corresponding to the human body motion image data;

Determine whether the joint motion data complies with the atomic motion constraints corresponding to the robot by using the simulation data of the digital twin model of the robot;

When it is determined that the joint motion data complies with the atomic motion constraint conditions corresponding to the robot, the joint motion data is determined as the atomic motion of the robot.
The method according to claim 1, wherein the determining the joint motion data of the robot corresponding to the human motion image data comprises:

performing denoising processing on the human action image data;

Corresponding three-dimensional gesture motion data is determined according to the denoised human motion image data, and the three-dimensional gesture motion data includes the motion data of each three-dimensional key point corresponding to the target object, and the three-dimensional key point for constructing the three-dimensional pose of the target object;

The joint motion data of the robot is obtained by mapping the three-dimensional posture motion data.
The method according to claim 1 or 2, wherein the judging whether the joint motion data complies with the atomic motion constraints corresponding to the robot by using the simulation data of the digital twin model of the robot comprises:

Based on the simulation data of the digital twin model, it is determined whether the joint motion data meets the motion compliance conditions, and the motion compliance conditions include that the motion corresponding to the joint motion data is the atomic motion, and the joint motion data does not There is a joint movement that exceeds the physical limit of the joint, and the digital twin model does not have collision and imbalance when executing the joint movement data;

When the joint motion data does not meet any one of the motion compliance conditions, it is determined that the joint motion data does not meet the atomic motion constraint conditions corresponding to the robot.
The method according to claim 3, wherein the judging whether the joint motion data complies with the atomic motion constraints corresponding to the robot by using the simulation data of the digital twin model of the robot further comprises:

When the joint motion data satisfies all the motion compliance conditions, determine whether the joint motion data satisfies motion optimization conditions based on the simulation data of the digital twin model, where the motion optimization conditions include that the joint motion data complies with the robot The motions corresponding to the kinematic laws and the joint motion data are smooth and smooth;

When the joint motion data does not meet the motion optimization condition, it is determined that the joint motion data does not meet the atomic motion constraint condition corresponding to the robot.
The method according to claim 4, wherein the judging whether the joint motion data complies with the atomic motion constraints corresponding to the robot by using the simulation data of the digital twin model of the robot further comprises:

When the joint motion data satisfies the motion optimization condition and the motion compliance condition, execute the joint motion data through the digital twin model, and display the motion picture of the digital twin model;

In the case of receiving the first manual confirmation instruction, the robot is driven to execute the joint motion data through the digital twin model, and the first manual confirmation instruction is input by the user in response to the motion picture of the digital twin model instruction, used to indicate that the motion picture of the digital twin model meets the requirements;

In the case of receiving a second manual confirmation instruction, it is determined that the joint motion data complies with the atomic motion constraint corresponding to the robot, and the second manual confirmation instruction is determined by the user in response to the execution action of the robot. The input instructions are used to instruct the robot to perform actions that meet the requirements.
The method according to any one of claims 1-5, wherein the method further comprises:

When it is determined by the simulation data of the digital twin model of the robot that the joint motion data does not meet the atomic motion constraints corresponding to the robot, the joint motion data is corrected, and the joint motion data is corrected through the digital twin model of the robot. It is judged whether the revised joint motion data complies with the atomic motion constraint conditions corresponding to the robot until it is determined that the joint motion data complies with the atomic motion constraint conditions corresponding to the robot.
The method according to claim 6, wherein the determining whether the joint motion data complies with the atomic motion constraints corresponding to the robot by using the simulation data of the digital twin model of the robot comprises:

Based on the simulation data of the digital twin model, it is determined whether the joint motion data meets the motion compliance conditions, and the motion compliance conditions include that the motion corresponding to the joint motion data is the atomic motion, and the joint motion data does not There is a joint movement that exceeds the physical limit of the joint, and the digital twin model does not have collision and imbalance when executing the joint movement data;

When the joint motion data satisfies all the motion compliance conditions, determine whether the joint motion data satisfies motion optimization conditions based on the simulation data of the digital twin model, where the motion optimization conditions include that the joint motion data complies with the robot The motions corresponding to the kinematic laws and the joint motion data are smooth and smooth;

The modifying the joint motion data includes:

When the joint motion data does not meet any one of the motion compliance conditions, receiving a manual modification instruction to modify the joint motion data;

When the joint motion data does not meet the motion optimization condition, perform a first optimization on the joint motion data, where the first optimization includes performing data optimization on the joint motion data according to inverse kinematics, generating a network according to confrontation One or more of performing data optimization on the joint motion data by training, performing data optimization on the joint motion data according to data fitting, and performing data optimization on the joint motion data according to an interpolation algorithm.
The method according to any one of claims 1-7, wherein when the robot is in an action imitation mode, the method further comprises:

Judging by the simulation data of the digital twin model of the robot whether the joint motion data conforms to the action imitation conditions corresponding to the robot;

In the case that the joint motion data does not conform to the motion imitation conditions corresponding to the robot, a second optimization is performed on the joint motion data, and it is re-determined whether the optimized joint motion data conforms to the motion corresponding to the robot Imitating conditions, the second optimization includes performing data optimization on the joint motion data according to inverse kinematics, performing data optimization on the joint motion data according to confrontational generation network training, and performing data optimization on the joint motion data according to data fitting. One or more of data optimization, performing data optimization on the joint motion data according to an interpolation algorithm;

In the case that the joint motion data conforms to the motion simulation conditions corresponding to the robot, the robot is driven to execute the joint motion data through the digital twin model, and returns to the initial stage of the robot after the motion execution is completed. state.
An action learning device, characterized in that the device comprises:

an acquisition module, used to acquire human motion image data of the target object;

a determination module for determining joint motion data of the robot corresponding to the human body motion image data;

a first judging module, used for judging whether the joint motion data complies with the atomic motion constraints corresponding to the robot through the simulation data of the digital twin model of the robot;

The learning module is configured to determine the joint motion data as the atomic motion of the robot when it is determined that the joint motion data conforms to the atomic motion constraint condition corresponding to the robot.
The device according to claim 9, wherein the determining module comprises:

a denoising sub-module for performing denoising processing on the human action image data;

A determination submodule, configured to determine corresponding three-dimensional posture motion data according to the denoised human body motion image data, where the three-dimensional posture motion data includes motion data of each three-dimensional key point corresponding to the target object , the three-dimensional key points are used to construct the three-dimensional pose of the target object;

The mapping submodule is used for mapping the joint motion data of the robot according to the three-dimensional attitude motion data.
The device according to claim 9 or 10, wherein the first judging module comprises a first judging sub-module for:

Based on the simulation data of the digital twin model, it is determined whether the joint motion data meets the motion compliance conditions, and the motion compliance conditions include that the motion corresponding to the joint motion data is the atomic motion, and the joint motion data does not There is a joint movement that exceeds the physical limit of the joint, the digital twin model does not have collision and imbalance when executing the joint movement data; and the joint movement data does not meet any of the movement compliance conditions. When there is one, it is determined that the joint motion data does not conform to the atomic motion constraints corresponding to the robot.
The device according to claim 11, wherein the first judging module further comprises a second judging sub-module for:

When the joint motion data satisfies all the motion compliance conditions, determine whether the joint motion data satisfies motion optimization conditions based on the simulation data of the digital twin model, where the motion optimization conditions include that the joint motion data complies with the robot The kinematic law and the motion corresponding to the joint motion data are smooth and smooth; and when the joint motion data does not satisfy the motion optimization condition, it is determined that the joint motion data does not conform to the atomic motion constraint condition corresponding to the robot.
The device according to claim 12, wherein the first judgment module further comprises:

The simulation submodule is used to execute the joint motion data through the digital twin model when the joint motion data satisfies the motion optimization condition and the motion compliance condition, and compare the motion picture of the digital twin model to display;

an action execution sub-module, configured to drive the robot to execute the joint motion data through the digital twin model in the case of receiving a first manual confirmation instruction, the first manual confirmation instruction being that the user responds to the digital The instruction input by the motion picture of the twin model is used to indicate that the motion picture of the digital twin model meets the requirements;

The determination sub-module is configured to determine that the joint motion data complies with the atomic motion constraints corresponding to the robot in the case of receiving a second manual confirmation instruction, and the second manual confirmation instruction is a response of the user to the The instruction inputted by the execution action of the robot is used to indicate that the execution action of the robot meets the requirements.
The device according to any one of claims 9-13, wherein the device further comprises a correction module for:

When it is determined by the simulation data of the digital twin model of the robot that the joint motion data does not meet the atomic motion constraints corresponding to the robot, the joint motion data is corrected, and the joint motion data is corrected through the digital twin model of the robot. It is determined whether the revised joint motion data complies with the atomic motion constraint corresponding to the robot until it is determined that the joint motion data complies with the atomic motion constraint corresponding to the robot.
The device according to claim 14, wherein the first judgment module comprises:

A first judgment sub-module, configured to judge whether the joint motion data satisfies the motion compliance condition based on the simulation data of the digital twin model, and the motion compliance condition includes that the action corresponding to the joint motion data is the atomic motion , in the joint motion data, there is no joint motion exceeding the physical limit of the joint, and the digital twin model does not have collision and imbalance when executing the joint motion data;

The second judging sub-module, when the joint motion data meets all the motion compliance conditions, judges whether the joint motion data satisfies motion optimization conditions based on the simulation data of the digital twin model, and the motion optimization conditions include all the motion optimization conditions. The joint motion data conforms to the law of robot kinematics and the motion corresponding to the joint motion data is smooth and smooth;

The correction module also includes:

A first correction submodule, configured to receive a manual modification instruction to modify the joint motion data when the joint motion data does not meet any of the motion compliance conditions, and re-judg the modified joint motion Whether the data conforms to the atomic action constraints corresponding to the robot.

The second correction sub-module is configured to perform a first optimization on the joint motion data when the joint motion data does not meet the motion optimization condition, and re-judg whether the optimized joint motion data conforms to the corresponding robot The first optimization includes performing data optimization on the joint motion data according to inverse kinematics, performing data optimization on the joint motion data according to adversarial generation network training, and performing data optimization on the joint motion data according to data fitting. One or more of performing data optimization on the data and performing data optimization on the joint motion data according to an interpolation algorithm.
The device according to any one of claims 9-15, wherein the device further comprises:

a second judging module, configured to judge whether the joint motion data complies with the motion imitation conditions corresponding to the robot by using the simulation data of the digital twin model of the robot when the robot is in an action imitation mode;

An optimization module, configured to perform a second optimization on the joint motion data when the joint motion data does not meet the motion imitation conditions corresponding to the robot, and re-judg whether the optimized joint motion data conforms to the motion simulation conditions of the robot. According to the corresponding action imitation conditions of the robot, the second optimization includes performing data optimization on the joint motion data according to inverse kinematics, performing data optimization on the joint motion data according to adversarial generation network training, and performing data optimization on the joint motion data according to data fitting. One or more of performing data optimization on the joint motion data and performing data optimization on the joint motion data according to an interpolation algorithm;

The execution module is used to drive the robot to execute the joint motion data through the digital twin model under the condition that the joint motion data conforms to the action imitation conditions corresponding to the robot, and return to the joint motion data after the action execution is completed. The initial state of the robot.
A non-volatile computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the steps of the method according to any one of claims 1-8 are implemented.
An electronic device, comprising:

a memory on which a computer program is stored;

A processor for executing the computer program in the memory to implement the steps of the method of any one of claims 1-8.
A computer program product, characterized in that the computer program product comprises a computer program executable by a programmable device, the computer program having, when executed by the programmable device, for performing any one of claims 1-8 The code portion of the steps of the method described in item .