CN116079703A

CN116079703A - Robot teaching method, apparatus, device and computer readable storage medium

Info

Publication number: CN116079703A
Application number: CN202111308490.8A
Authority: CN
Inventors: 杜广龙; 苏金绍; 郑宇�
Original assignee: South China University of Technology SCUT; Tencent Technology Shenzhen Co Ltd
Current assignee: South China University of Technology SCUT; Tencent Technology Shenzhen Co Ltd
Priority date: 2021-11-05
Filing date: 2021-11-05
Publication date: 2023-05-09

Abstract

The application discloses a robot teaching method, a device, equipment and a computer readable storage medium, and belongs to the technical field of robots. The method comprises the following steps: providing haptic feedback in response to the physical robot colliding with the obstacle during controlling the physical robot to perform the first teaching action; and adjusting the first teaching action by using an adjustment instruction acquired after the haptic feedback is provided to obtain a target teaching action meeting the non-collision condition, and completing the teaching of the physical robot based on the target teaching action. According to the method, automatic detection of the collision event is increased, when the collision event is detected, the tactile feedback for prompting the existence of the collision event is provided, the reliability of the automatic detection of the collision event is higher, the adjustment instruction obtained after the tactile feedback is provided is a more reliable adjustment instruction generated by a demonstrator according to the prompting of the tactile feedback, the quality of adjusting the teaching action is higher, and the improvement of the teaching effect of the robot is facilitated.

Description

Robot teaching method, apparatus, device and computer readable storage medium

Technical Field

The embodiment of the application relates to the technical field of robots, in particular to a robot teaching method, a device, equipment and a computer readable storage medium.

Background

With the development of robot technology, more and more physical robots can complete various tasks such as carrying, assembling, track tracking and the like, and before a certain task is completed by using the physical robots, a demonstrator needs to teach the physical robots so as to determine target teaching actions required to be executed by the physical robots to complete the task, and further, the teaching of the physical robots is completed based on the target teaching actions.

In the process of teaching the entity robot by the demonstrator, the demonstrator can observe the process of the entity robot controlled by the computer equipment to execute a certain teaching action in real time. In the related art, a demonstrator artificially observes a collision event between a physical robot and an obstacle, and generates an adjustment instruction after observing the collision event, and a computer device adjusts a teaching action by using the adjustment instruction to obtain a final target teaching action.

The reliability of the artificial observation collision event is low, the collision event occurring in the visual blind area is easily missed or the non-collision event is mistaken as the collision event, the reliability of an adjustment instruction generated after the demonstrator observes the collision event is low, the quality of adjusting the teaching action is poor, and the teaching effect of the robot is poor.

Disclosure of Invention

The embodiment of the application provides a robot teaching method, a device, equipment and a computer readable storage medium, which can be used for improving the quality of adjusting teaching actions so as to improve the teaching effect of a robot. The technical scheme is as follows:

in one aspect, an embodiment of the present application provides a robot teaching method, where the method includes:

in the process of controlling the entity robot to execute the first teaching action, responding to collision of the entity robot and an obstacle, and providing haptic feedback, wherein the haptic feedback is used for prompting that a collision event exists;

and adjusting the first teaching action by using an adjustment instruction acquired after the haptic feedback is provided to obtain a target teaching action meeting a non-collision condition, and completing the teaching of the physical robot based on the target teaching action.

In another aspect, there is provided a robot teaching device, the device including:

the control unit is used for responding to collision of the entity robot and the obstacle in the process of controlling the entity robot to execute the first teaching action and providing tactile feedback, wherein the tactile feedback is used for prompting the existence of a collision event;

And the adjusting unit is used for adjusting the first teaching action by utilizing an adjusting instruction acquired after the tactile feedback is provided to obtain a target teaching action meeting a non-collision condition, and the teaching of the physical robot is completed based on the target teaching action.

In one possible implementation, the control unit is configured to determine a target current based on a collision force between the physical robot and the obstacle; the target current is applied to a haptic feedback device such that the haptic feedback device provides the haptic feedback under the influence of the target current.

In one possible implementation, the apparatus further includes:

the acquisition unit is used for acquiring a second teaching action corresponding to the virtual robot; and acquiring the first teaching action based on the second teaching action.

In one possible implementation manner, the acquiring unit is configured to acquire teaching information, where the teaching information includes at least one of gesture information and voice information; based on the teaching information, acquiring a teaching instruction, wherein the teaching instruction is used for indicating sub-teaching actions; virtual teaching is carried out on the virtual robot by utilizing the sub-teaching action indicated by the teaching instruction; and responding to the virtual teaching process meeting the first condition, and obtaining the second teaching action.

In one possible implementation manner, the teaching information includes gesture information and voice information, and the acquiring unit is configured to acquire a fusion text based on the gesture information and the voice information; invoking a classification model to classify the fusion text to obtain the matching probability of each candidate instruction, wherein the classification model is obtained based on a sample text and instruction labels corresponding to the sample text; and acquiring the teaching instruction based on the candidate instruction with the matching probability meeting the selection condition.

In one possible implementation manner, the acquiring unit is configured to correct the second teaching action to obtain a corrected second teaching action; and acquiring the first teaching action based on the corrected second teaching action.

In one possible implementation, the virtual robot is built by an augmented reality device based on the physical robot.

In one possible implementation manner, the acquiring unit is configured to send the teaching instruction to an augmented reality device, and the augmented reality device controls the virtual robot to execute the sub-teaching action indicated by the teaching instruction.

In one possible implementation manner, the physical robot is provided with a force sensor, and the control unit is further configured to determine that the physical robot collides with the obstacle in response to the force detected by the force sensor meeting a collision detection condition.

In one possible implementation, the classification model is a maximum entropy model.

In another aspect, a computer device is provided, where the computer device includes a processor and a memory, where at least one computer program is stored in the memory, where the at least one computer program is loaded and executed by the processor, so that the computer device implements any one of the robot teaching methods described above.

In another aspect, there is also provided a computer readable storage medium having at least one computer program stored therein, the at least one computer program being loaded and executed by a processor to cause a computer to implement any one of the above-described robot teaching methods.

In another aspect, there is also provided a computer program product comprising a computer program or computer instructions loaded and executed by a processor to cause a computer to implement any of the above-described robot teaching methods.

The technical scheme provided by the embodiment of the application at least brings the following beneficial effects:

according to the technical scheme, in the process of controlling the entity robot to execute teaching actions, collision events between the entity robot and the obstacle are automatically detected, and when the collision events are detected, tactile feedback for prompting the existence of the collision events is provided, so that a demonstrator can intuitively perceive the collision events according to the tactile feedback. The reliability of the automatic collision event detection is higher, the adjustment instruction obtained after the haptic feedback is provided is a more reliable adjustment instruction generated by a demonstrator according to the prompt of the haptic feedback, the quality of adjusting teaching actions by using the adjustment instruction obtained after the haptic feedback is provided is higher, and the improvement of the teaching effect of the robot is facilitated.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic diagram of an implementation environment of a robot teaching method provided in an embodiment of the present application;

FIG. 2 is a schematic illustration of a teaching environment for a robot provided in an embodiment of the present application;

FIG. 3 is a flow chart of a robot teaching method provided in an embodiment of the present application;

FIG. 4 is a schematic illustration of a three-dimensional hand model provided in an embodiment of the present application;

FIG. 5 is a schematic diagram of a coordinate system referenced by various subjects in a teaching environment provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of a haptic feedback device provided by an embodiment of the present application;

fig. 7 is a schematic view of a robot teaching device according to an embodiment of the present application;

fig. 8 is a schematic diagram of a robot teaching device provided in an embodiment of the present application;

fig. 9 is a schematic structural diagram of a terminal according to an embodiment of the present application;

Fig. 10 is a schematic structural diagram of a server according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

It should be noted that the terms "first," "second," and the like herein are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be implemented in sequences other than those illustrated or otherwise described herein. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.

Fig. 1 shows a schematic diagram of an implementation environment of a robot teaching method according to an embodiment of the present application. The implementation environment comprises: a computer device 11, a physical robot 12, an augmented reality device 13, a somatosensory device 14 and a haptic feedback device 15.

Where the physical robot 12 refers to a device for performing certain tasks instead of a human, the physical robot is, for example, a robot applied in industry, which may also be referred to as an industrial robot. The type of task that the entity robot 12 can perform is not limited in this embodiment of the present application, which relates to an actual application scenario, a structure of the entity robot, and the like, and exemplary tasks that the entity robot 12 can perform include, but are not limited to, an assembly task, a track-following task, and the like. The present embodiment does not limit the structure of the physical robot 12, and the physical robot 12 is, for example, a robot having six degrees of freedom, that is, a robot having six joints. Of course, the physical robot 12 may be a robot of other structures.

The augmented reality (Augmented Reality, AR) device 13 is a device integrated with augmented reality technology. Augmented reality technology is a new technology that integrates real and virtual environments "seamlessly" with the goal of having the virtual environment fit within the real environment on the screen and for the user to interact. In the embodiment of the present application, the augmented reality device 13 can construct a virtual robot based on an entity robot located in a real environment, and add motion attributes to the virtual robot, and then can control the motion of the virtual robot in a manner of controlling the entity robot. The embodiment of the present application does not limit the type of the augmented reality device 13, and the augmented reality device 13 is exemplified by AR glasses such as holonens (a head mounted display). Illustratively, the augmented reality device 13 is worn by a teach pendant, which by wearing the augmented reality device 13 is able to observe not only the movements of the virtual robot, but also the movements of the physical robot 12.

The somatosensory device 14 is used to acquire teaching information of a demonstrator, such as gesture information, voice information, and the like. Illustratively, the somatosensory device 14 has multiple functions of real-time dynamic capturing, image recognition, microphone input, voice recognition, and the like. The type of the Motion sensing device 14 is not limited in the embodiments of the present application, and the Motion sensing device 14 is exemplified by Kinect (a Motion sensing device), leep Motion (a Motion sensing device), and the like. Illustratively, the somatosensory device 14 is coupled to the physical robot 12 to facilitate the collection of teaching information of the teach pendant. The embodiments of the present application do not limit the location where the somatosensory device 14 is attached to the physical robot 12. For example, in the case that the physical robot 12 is a robot having six joints, and the fifth joint of the physical robot 12 is a translational joint and the sixth joint is a rotational joint, the motion sensing device 14 is connected to the fifth joint of the physical robot 12 to ensure the stability of the motion sensing device 14. Of course, in exemplary embodiments, the somatosensory device 14 may also be unattached to the physical robot 12, e.g., placed on some stationary object.

The haptic feedback device 15 is used to provide haptic feedback to the teach pendant. The teach pendant can receive haptic feedback by wearing the haptic feedback device 15. The wearing manner of the haptic feedback device 15 is not limited in the embodiments of the present application, and the haptic feedback device 15 is worn on the finger of the demonstrator by way of example, in which case the haptic feedback device 15 may also be referred to as a fingertip haptic feedback device.

The computer device 11 is used for implementing the robot teaching method provided in the embodiment of the present application. In an exemplary embodiment, the computer device 11 is capable of acquiring the teaching information acquired by the somatosensory device 14, converting the teaching information into a teaching instruction for controlling the virtual robot, and issuing the teaching instruction to the augmented reality device 13, so that the augmented reality device 13 controls the virtual robot according to the teaching instruction. The computer device 11 can also acquire teaching actions corresponding to the virtual robot, convert the teaching actions corresponding to the virtual robot into teaching actions corresponding to the physical robot 12, and control the physical robot 12 to execute the teaching actions. The computer device 11 is also capable of controlling the haptic feedback device 15 to provide haptic feedback to the teach pendant in response to the physical robot 12 colliding with an obstacle during the performance of the teaching action.

The computer device 11 may be a terminal or a server, which is not limited in the embodiment of the present application. By way of example, the terminal may be any electronic product that can interact with a user by one or more of a keyboard, a touch pad, a touch screen, a remote control, a voice interaction or handwriting device, such as a PC (Personal Computer, a personal computer), a mobile phone, a smart phone, a PDA (Personal Digital Assistant, a personal digital assistant), a wearable device, a PPC (Pocket PC, palm computer), a tablet computer, a smart car set, a smart television, a smart sound box, a car terminal, etc. The server may be a server, a server cluster comprising a plurality of servers, or a cloud computing service center. The computer device 11 establishes a communication connection with the physical robot 12, the augmented reality device 13, the somatosensory device 14 and the haptic feedback device 15 through a wired or wireless network.

Those skilled in the art will appreciate that the above-described computer device 11, physical robot 12, augmented reality device 13, somatosensory device 14, and haptic feedback device 15 are by way of example only, and that other devices now known or hereafter may be present as applicable to the present application are also intended to be within the scope of the present application and are incorporated herein by reference.

In an exemplary embodiment, for the case where the physical robot performs a trajectory tracking task on a production line, a teaching environment of a robot including the physical robot, a somatosensory device, an augmented reality device, a haptic feedback device, a production line, and a virtual robot is shown in fig. 2. The virtual robot is constructed by the augmented reality device based on the physical robot, and the base of the virtual robot coincides with the base of the physical robot. The somatosensory device is mounted on a fifth joint of the physical robot. The augmented reality device is worn on the eyes of the teach pendant, and the augmented reality device is capable of providing visual feedback to the teach pendant. The haptic feedback device is worn on the digit of the teach pendant and is capable of providing haptic feedback to the teach pendant. The production line is provided with a track tracking accessory.

Based on the implementation environment shown in fig. 1 described above, the embodiment of the present application provides a robot teaching method, which is executed by the computer device 11. As shown in fig. 3, the robot teaching method provided in the embodiment of the present application includes the following

steps

301 and 302.

In step 301, in response to the physical robot colliding with the obstacle during control of the physical robot to perform a first teaching action, haptic feedback is provided for alerting of the presence of a collision event.

The first teaching action refers to a certain teaching action required to be executed by the physical robot determined in the process of teaching the physical robot, and illustratively, the first teaching action is composed of one or more sub-teaching actions. The first teaching action may be a finally determined teaching action or a teaching action that needs to be further adjusted, which is not limited in the embodiment of the present application.

In an exemplary embodiment, the first teaching action needs to be acquired before step 301 is performed, and the manner of acquiring the first teaching action is not limited in this embodiment of the present application. Illustratively, the first teaching action is manually programmed on the teach pendant by the teach pendant. The first teaching action is illustratively derived based on a corresponding second teaching action of the virtual robot.

In this case, the teaching of the physical robot can be realized in a mode of off-line fusion teaching, and the demonstrator can directly and safely perform virtual teaching on the virtual robot in a real scene through off-line fusion teaching, and then the physical robot reproduces the motion of the virtual robot to complete the teaching of the physical robot.

Before teaching by utilizing the off-line fusion teaching mode, a virtual robot needs to be constructed based on the physical robot. Illustratively, the process of building a virtual robot is performed by the augmented reality device, i.e., the virtual robot is built by the augmented reality device based on the physical robot. By wearing the augmented reality apparatus, the demonstrator can observe not only the physical robot in the real environment but also the virtual robot superimposed and displayed in the real environment. Illustratively, the base of the constructed virtual robot coincides with the base of the physical robot in the real environment within the field of view of the augmented reality device.

In the case where the first teaching action is obtained based on the second teaching action corresponding to the virtual robot, before executing step 301, the steps to be executed include: acquiring a second teaching action corresponding to the virtual robot; based on the second teaching action, a first teaching action is acquired.

The second teaching action corresponding to the virtual robot refers to a teaching action which needs to be executed by the virtual robot and is determined for enabling the virtual robot to complete the virtual task. The virtual task is obtained by virtualizing a task that needs to be executed by the physical robot, for example, if the task that needs to be executed by the physical robot is that an end effector of the physical robot tracks a target track, the virtual task is that the end effector of the virtual robot tracks a virtual track, where the virtual track coincides with the target track.

In one possible implementation, the process of acquiring the second teaching action corresponding to the virtual robot includes the following steps 3001 to 3004.

Step 3001: and acquiring teaching information, wherein the teaching information comprises at least one of gesture information and voice information.

The gesture information is used for representing gestures of the demonstrator, the voice information is used for representing voices of the demonstrator, and the demonstrator teaches the physical robot through at least one natural interaction mode of the gestures and the voices. For the situation that a demonstrator teaches the physical robot in a natural interaction mode of gestures, the teaching information acquired by the computer equipment only comprises gesture information; for the situation that a demonstrator teaches the physical robot in a natural interaction mode of voice, the teaching information acquired by the computer equipment only comprises voice information; for the situation that a demonstrator teaches the physical robot through two natural interaction modes of gestures and voices, teaching information acquired by the computer equipment comprises gesture information and voice information. Illustratively, the gesture of the demonstrator may refer to a dynamic gesture for drawing a track to be tracked by the robot, or may refer to a static gesture for indicating a direction in which the robot moves, which is not limited in the embodiment of the present application.

The teaching of the physical robot by the demonstrator in the natural interaction mode is not limited, and the teaching of the physical robot can be realized.

Illustratively, the teaching information acquired in step 3001 refers to teaching information acquired within a certain period of time. For the case that the teaching information includes gesture information, the number of gesture information included in the teaching information may be one or more; in the case where the teaching information includes voice information, the teaching information may include one or more of the voice information.

The gesture information is obtained by the following way: the somatosensory equipment collects gesture images of the demonstrator according to the image collection frequency, and sends the collected gesture images to the computer equipment; the computer equipment performs gesture recognition on the gesture image, and acquires gesture information based on information obtained by recognition. The number of gesture information acquired by the computer device in a certain time period is the same as the number of gesture images acquired by the somatosensory device in the time period. Illustratively, if the duration of a certain period of time is 3 seconds and the image acquisition frequency is 40 images per second, the computer device can acquire 120 gesture information in the certain period of time.

The image acquisition frequency is set empirically or flexibly adjusted according to the type of somatosensory device, and the embodiment of the application is not limited to this, for example, the image acquisition frequency is 40 images acquired per second. Illustratively, the motion sensing device coordinate system referred by the motion sensing device is different from the hand coordinate system referred by the hands of the demonstrator, and in the process of acquiring the gesture image, the motion sensing device converts the position of the hands of the demonstrator under the hand coordinate system into the motion sensing device coordinate system according to the transformation relationship between the hand coordinate system and the motion sensing device coordinate system, so as to ensure the accuracy of subsequent teaching. The transformation relation between the hand coordinate system and the somatosensory equipment coordinate system is calibrated in advance.

Illustratively, the hand coordinate system is a coordinate system determined based on a three-dimensional hand model. Three-dimensional hand model as shown in fig. 4, the three-dimensional hand model may provide the location of the index finger tip a, the metacarpal joint B, and the palm center C. From A to B, a vector AB is obtained. C is taken as the origin of the hand coordinate system, and the axis parallel to the vector AB is taken as the Y-axis (Y _H ) An axis perpendicular to the vector AB in the plane ABC is taken as the X-axis (X _H ) Since the X-axis, Y-axis and Z-axis of the hand coordinate system are perpendicular to each other, the X-axis, Y-axis and Z-axis can be determined by Y _H And X _H The Z-axis (Z _H ). Note that X shown in fig. 4 _B ^R 、Y _B ^R And Z _B ^R Refers to three coordinate axes of a base reference coordinate system of the physical robot.

The motion sensing device is illustratively a cuboid-shaped device, and the motion sensing device coordinate system has a center point of the motion sensing device as an origin, and an axis parallel to an optical axis of the motion sensing device as a Z-axis (Z _K ) An axis parallel to the long side of the somatosensory device was taken as the Y axis (Y _K ) Will be in contact with Z _K And Y _K The axes perpendicular to each other are taken as X-axis (X _K )。

For example, a computer device performs gesture recognition on a gesture image using a gesture recognition model that is capable of outputting information characterizing the recognized gesture. The gesture recognition model can be obtained by training a sample gesture image and an information label in a supervision training mode.

Illustratively, a gesture tracking system (e.g., 3 Gear Systems) is integrated into the computer device, which uses the gesture tracking system to gesture the gesture image. Illustratively, the gesture tracking system operates in a client/server mode, and the computer device may communicate with the gesture recognition server via a UDP (User Datagram Protocol ) socket and obtain information characterizing the gesture via an API (Application Programming Interface, application program interface) provided by the gesture recognition server to enable gesture recognition of the gesture image.

After gesture recognition is performed on the gesture image, information for characterizing the gesture in the gesture image can be obtained, and the embodiment of the application is not limited to a representation form of the information for characterizing the gesture, and the information for characterizing one gesture includes a state of a finger tip of an index finger, for example. Illustratively, the state of the index finger tip includes, but is not limited to, the position, velocity, and acceleration at which the index finger tip is located in the somatosensory device coordinate system. The velocity and acceleration can be calculated from two adjacent gesture images. Illustratively, the position of the index finger tip in the somatosensory device coordinate system is uniquely determined by the positions of the index finger tip in the X-axis, Y-axis and Z-axis of the somatosensory coordinate system respectively, and the velocity of the index finger tip in the somatosensory device coordinate system is uniquely determined by the velocities of the index finger tip in the X-axis, Y-axis and Z-axis of the somatosensory coordinate system respectivelyThe acceleration in the motion sensing device coordinate system is uniquely determined by the acceleration of the index finger tip in the X-axis, Y-axis and Z-axis of the motion sensing coordinate system respectively. Illustratively, let x _k Information representing a gesture characterizing time k, x _k Represented by formula (1):

wherein p is _x，k ，v _x，k ，a _x，k Respectively representing the position, the speed and the acceleration of the tip of the index finger on the X axis of the somatosensory coordinate system; p is p _y，k ，v _y，k ，a _y，k Respectively representing the position, the speed and the acceleration of the tip of the index finger on the Y axis of the somatosensory coordinate system; p is p _z，k ，v _z，k ，a _z，k Respectively representing the position, the speed and the acceleration of the tip of the index finger on the Z axis of the somatosensory coordinate system;

representing the transpose of the matrix.

In an exemplary embodiment, the information for characterizing the gesture may include, in addition to the state of the tip of the index finger, a direction indicated by the index finger, and the like, which is not limited in the embodiment of the present application.

After gesture recognition is performed on the gesture image, gesture information is acquired based on the information obtained by the recognition. In one possible implementation, based on the information obtained by the recognition, the manner of acquiring the gesture information refers to: and directly taking the information obtained by recognition as gesture information. In this case, the obtained gesture information is the originally recognized information itself, no additional operation is required to be performed, and the efficiency of obtaining the gesture information is high.

In another possible implementation manner, based on the information obtained by the recognition, the gesture information is obtained by: and filtering the information obtained by recognition, and taking the information obtained by filtering as gesture information. The identified information may have a certain measurement error and have a certain noise, and by filtering the identified information, more reliable gesture information can be obtained, so as to obtain more accurate teaching instructions. In the case where the number of gesture images is plural, the number of pieces of information to be recognized is also plural, and the filtering of the pieces of information to be recognized means the filtering of each piece of information to be recognized.

The filtering manner of the information obtained by recognition can be set empirically, and can be flexibly adjusted according to actual application scenes, which is not limited in the embodiment of the application. Illustratively, the identified information is either linearly filtered (e.g., kalman filtered) or non-linearly filtered (e.g., particle filtered).

In an exemplary embodiment, the process of filtering the identified information to obtain gesture information is implemented based on equation (2):

x′ _k ＝Φ _k x _k +B _k u _k +w _k (2)

wherein x' _k Representing gesture information obtained after filtering; x is x _k Information obtained by the identification is represented by the formula (1); phi _k Representing a state transition matrix; u (u) _k Representing an input matrix; b (B) _k Representing a control input model applied to the input matrix; b (B) _k u _k Representing a system input matrix; w (w) _k Representing a process noise matrix.

Illustratively, in filtering the identified information, a state transition matrix Φ _k As shown in formula (3):

wherein t represents an image acquisition time interval of the somatosensory device.

Since there is no control input for the position state, the system inputs matrix B _k ·u _k As shown in formula (4):

process noise matrix w _k As shown in formula (5):

/>

wherein w is _x ，w _y ，w _z The process noise of acceleration of the index finger tip is represented on the X-axis, Y-axis and Z-axis of the motion sensing device coordinate system, respectively. w (w) _x ，w _y ，w _z The value of (2) is set empirically or flexibly adjusted according to the actual application scenario, which is not limited in the embodiment of the present application.

By substituting expression (1), expression (3), expression (4), and expression (5) into expression (2), gesture information can be obtained.

The voice information is obtained by the following way: the voice acquisition device acquires voices sent by a demonstrator in a certain period of time and sends voice information used for representing the acquired voices to the computer equipment; the computer device obtains the voice information. The embodiment of the application does not limit the representation form of the voice information for representing the voice, and can refer to the audio composed of the voice. The type of the voice acquisition device is not limited in the embodiments of the present application, for example, the voice acquisition device refers to a microphone array. The voice capture device is illustratively built into the device with which the computer device is in communication, the voice capture device is built into the somatosensory device, or the voice capture device is built into the augmented reality device.

Based on the above-mentioned gesture information and the manner of acquiring the voice information and the specific case where the teaching information includes the gesture information and the voice information, the teaching information can be acquired.

Step 3002: based on the teaching information, a teaching instruction is acquired, and the teaching instruction is used for indicating sub-teaching actions.

After the teaching information is acquired, a teaching instruction for controlling the virtual robot is acquired based on the teaching information, wherein the teaching instruction is used for indicating a sub-teaching action, and the sub-teaching action refers to an action which needs to be controlled to be executed by the virtual robot. In the exemplary embodiment, since the connection relationship between the respective joints of the physical robot is known and the connection relationship between the respective joints of the virtual robot is the same as the connection relationship between the respective joints of the physical robot, the connection relationship between the respective joints of the virtual robot is also known, and the motion of the respective joints of the virtual robot can be calculated on the basis of the inverse kinematics from the connection relationship between the respective joints and the sub-teaching motion performed by the end effector. Illustratively, the connection relationships between the individual joints of the physical robot are determined by modeling the physical robot, for example, using a DH (Denavit-Hartenberg ) model.

The embodiment of the present application does not limit the manner of expressing the teaching instruction, as long as one kind of teaching operation can be clearly indicated. Illustratively, the teach instruction is composed of four attributes (C _opt ,C _dir ,C _val ,C _unit ) And (5) defining. Wherein C is _opt Representing the type of sub-teaching action; c (C) _dir Indicating the direction of the sub-teaching action; c (C) _val Representing an action value; c (C) _unit Representing units of action. Illustratively, the teach instruction is represented by a coordinate point for indicating the position of the end effector of the virtual robot, the coordinate point being in a coordinate system to which the end effector of the virtual robot refers.

Illustratively, the process of acquiring the teaching instruction based on the teaching information refers to a process of acquiring the teaching instruction by comprehensively considering all information in the teaching information, and the accuracy of the acquired teaching instruction can be improved in this way. For example, in the case where the teaching information includes only gesture information, the teaching instruction is an instruction obtained by taking the gesture information into consideration; for the case where the teaching information includes only voice information, the teaching instruction is an instruction obtained by taking the voice information into consideration; for the case where the teaching information includes gesture information and voice information, the teaching instruction is an instruction obtained by comprehensively considering the gesture information and the voice information.

In an exemplary embodiment, in the case where the teaching information includes gesture information and voice information, the process of acquiring the teaching instruction includes the following

steps

1 and 2 based on the teaching information.

Step 1: and acquiring the fusion text based on the gesture information and the voice information.

The fusion text is a text based on which teaching instructions are acquired on the basis of comprehensively considering gesture information and voice information. In an exemplary embodiment, acquiring the fusion text based on the gesture information and the voice information may refer to acquiring the fusion text based on all the acquired gesture information and all the acquired voice information; the method may also refer to acquiring the fusion text based on the acquired latest reference number of gesture information and all voice information, which is not limited in the embodiment of the present application. The reference number is set empirically or flexibly adjusted according to the application scenario, for example, the reference number is 2. The latest reference number of gesture information refers to gesture information acquired based on the latest reference number of gesture images acquired by the somatosensory device in a period of time.

In an exemplary embodiment, based on gesture information and voice information, the manner of acquiring the fused text is: acquiring a first text based on gesture information; acquiring a second text based on the voice information; and fusing the first text and the second text to obtain a fused text. Illustratively, the manner in which the first text is obtained based on the gesture information is: and converting the direction in the gesture information into a direction text, converting the coordinate point in the gesture information into a position text, and forming a first text by the direction text and the position text. Illustratively, the second text is obtained based on the speech information in the following manner: and identifying a second text corresponding to the voice information. The embodiment of the application does not limit the manner of identifying the text corresponding to the voice information. Illustratively, a speech recognition SDK (Software Development Kit ) is invoked to recognize text corresponding to the speech information. Illustratively, a speech recognition model is invoked to recognize text corresponding to the speech information.

It should be noted that, for the case that the number of acquired gesture information is plural, a first text is acquired based on each gesture information, and the number of the first texts is plural; for the case where the number of acquired voice information is plural, one second text is acquired based on each voice information, and the number of the second texts is plural.

After the first text and the second text are acquired, the first text and the second text are fused, and the fused text is obtained. For example, in the case that the number of the first texts and the number of the second texts are multiple, the multiple first texts and the multiple second texts are fused to obtain a fused text.

Illustratively, the way to fuse the first text and the second text is: and inputting the first text and the second text into a text fusion model or a text fusion program, and fusing the first text and the second text by using the text fusion model or the text fusion program. The text fusion model or the text fusion program is constructed according to fusion experience, and can fuse the text acquired based on gesture information and the text acquired based on voice information.

Step 2: invoking a classification model to classify the fusion text to obtain the matching probability of each candidate instruction, wherein the classification model is obtained based on the sample text and instruction labels corresponding to the sample text; and acquiring a teaching instruction based on the candidate instruction with the matching probability meeting the selection condition.

And after the fusion text is acquired, a classification model is called to classify the fusion text, so that a classification result is obtained, and the classification result comprises the matching probability of each candidate instruction. Each candidate instruction refers to a preset alternative teaching instruction, and can be set according to experience or flexibly adjusted according to an actual application scene of robot teaching, which is not limited in the embodiment of the present application.

In an exemplary embodiment, the manner of calling the classification model to classify the fused text is as follows: extracting text features of the fusion text, inputting the text features of the integrated text into a classification model for classification processing, and obtaining a classification result output by the classification model. The embodiment of the present application does not limit the manner of extracting the text feature of a certain text, and illustratively extracts the text feature of a certain text based on the IF-IDF (Term Frequency Inverse Document Frequency, word frequency-reverse file frequency) algorithm. Illustratively, the process of extracting text features of a certain text based on the IF-IDF algorithm is implemented based on equation (6):

wherein TF is _i，j Word frequency representing text i; n is n _i，j Representing the number of occurrences of text i in document j in the corpus;

representing the number of all text contained in document j; IDF (IDF) _i Reverse file frequency representing text i; |d| represents the number of all documents in the corpus; | { j: t is t _i ∈d _j The } | represents the number of documents in the corpus that contain text i; TFIDF (tfIDF) _i，j Representing the text characteristics of text i.

It should be noted that, the above description of one exemplary embodiment for extracting the text feature of a certain text is not limited thereto, and the word2vec model (a word vector model) may be used to extract the text feature of a certain text, for example.

The classification result comprises the matching probability of each candidate instruction, and the higher the matching probability of a certain candidate instruction is, the higher the matching degree of the candidate instruction and the fusion text is. After the matching probability of each candidate instruction is obtained, the teaching instruction is obtained based on the candidate instruction with the matching probability meeting the selection condition. The matching probability satisfies the selection condition and is set empirically or flexibly adjusted according to the application scene, which is not limited in the embodiment of the present application.

In an exemplary embodiment, the matching probability satisfying the selection condition means that the matching probability is a matching probability that the front K (K is an integer not less than 1) is large. In this case, the number of candidate instructions whose matching probabilities satisfy the selection condition is K. In an exemplary embodiment, the matching probability satisfying the selection condition means that the matching probability is not less than a probability threshold value, which is set empirically or flexibly adjusted according to an application scenario, for example, the probability threshold value is 0.8.

The number of candidate instructions whose matching probabilities satisfy the selection condition may be one or more, which is not limited in the embodiment of the present application. If the number of candidate instructions with the matching probability meeting the selection condition is one, the mode of acquiring the teaching instruction based on the candidate instructions with the matching probability meeting the selection condition is as follows: and directly taking a candidate instruction with the matching probability meeting the selection condition as a teaching instruction. If the number of candidate instructions with the matching probability meeting the selection condition is a plurality of, the mode of acquiring the teaching instruction based on the candidate instructions with the matching probability meeting the selection condition is as follows: and fusing the candidate instructions with the matching probabilities meeting the selection conditions to obtain the teaching instruction. Illustratively, fusing the candidate instructions with the plurality of matching probabilities meeting the selection condition refers to fusing candidate instructions with target types in the candidate instructions with the plurality of matching probabilities meeting the selection condition, and the target types refer to the instruction types with the most candidate instructions with the plurality of matching probabilities meeting the selection condition.

The classification model is obtained through training in a supervision training mode based on the sample text and the instruction label corresponding to the sample text, and the instruction label corresponding to the sample text is a certain candidate instruction in the candidate instructions. Before the step 2 is performed, a classification model needs to be trained. The embodiment of the application is not limited to the types of the classification models, and the training process of the classification models of different types is different. Illustratively, the classification model is a neural network model, a support vector machine model, a naive bayes model, a maximum entropy model, or the like.

Illustratively, the classification model is taken as the maximum entropy model for illustration. The core idea of the maximum entropy model is to satisfy a known condition when predicting the probability distribution of random variables. At this point, the entropy of the probability distribution is the greatest, which preserves the various possibilities and minimizes the risk of prediction. Let x be the text feature of one sample text and y be the corresponding instruction tag. The maximum entropy model is to model the conditional probability p (y|x) to obtain the most uniform distribution model. The conditional entropy H (p (y|x)) needs to be introduced in the maximum entropy model to measure the uniformity of the conditional probability p (y|x) distribution. The calculation formula of H (p (y|x)) is shown in formula (7):

wherein,,

representing the empirical distribution of text features x in the training set; p (y|x) represents the conditional probability distribution in the maximum entropy model that needs to be solved. />

The problem of solving the maximum entropy model can be generalized to an optimization problem represented by equation (8):

wherein f _i Representing a feature function constructed based on the sample text; n (n is an integer not less than 1) is the number of feature functions.

According to the lagrangian multiplier method, the maximum entropy probability distribution p (y|x) can be obtained by solving under the constraint of the formula (8), and the calculation formula of the p (y|x) is shown as the formula (9):

Wherein f _i (x, y) represents an ith feature function; lambda (lambda) _i Represents f _i (x, y); z (x) represents a normalization factor, and the calculation formula of Z (x) is shown as formula (10):

through researching a sample text, the weight value of each characteristic function can be obtained, and the maximum entropy probability distribution p (y|x) can be calculated based on the weight value of each characteristic function, the formula (9) and the formula (10), namely the maximum entropy model is obtained.

The implementation manner of acquiring the teaching instruction is described by taking the example that the teaching information comprises gesture information and voice information only, and the teaching instruction is directly acquired based on the first text for the case that the teaching information comprises gesture information only; and for the case that the teaching information only comprises voice information, acquiring the teaching instruction directly based on the second text.

Illustratively, based on the manner provided by the embodiments of the present application, the demonstrator need not always give a complete command during the teaching process, allowing the use of appropriate default values. The computer device can fill in the missing semantics through the context of the command. For example, the demonstrator first issues a command "move 3mm in this direction" and points in one direction P. If the next command "continue moving 1mm" is given, the computer device combines that command with the previous command, which can be interpreted as "move 1mm in the P direction". Therefore, the demonstrator does not need much attention to ensure the semantic integrity of each command, is more in line with the habit of daily communication of human beings, and improves the naturalness of the teaching process.

In an exemplary embodiment, for the case where the teaching information includes gesture information, dynamic coordinate registration is required before the teaching instruction is acquired based on the teaching information, so that coordinate points in the hand coordinate system can be converted into other coordinate systems. In the embodiment of the present application, taking an example in which the somatosensory device is mounted on the fifth joint of the physical robot, a coordinate system to which each subject (the device or the hand of the demonstrator) in the teaching environment refers is shown in fig. 5.

The base reference coordinate system of the physical robot is consistent with the world coordinate system, and the three axes of the base coordinate system are respectively marked as X _B ^R 、Y _B ^R And Z _B ^R The method comprises the steps of carrying out a first treatment on the surface of the The three axes of the fifth joint coordinate system referenced by the fifth joint of the physical robot are respectively marked as X ₅ ^R 、Y ₅ ^R And Z ₅ ^R The method comprises the steps of carrying out a first treatment on the surface of the Three axes of the physical end effector coordinate system to which the end effector of the physical robot refers are respectivelyDenoted as X _E ^R 、Y _E ^R And Z _E ^R The method comprises the steps of carrying out a first treatment on the surface of the Three axes of the motion sensing device coordinate system referred to by the motion sensing device are respectively marked as X _K 、Y _K And Z _K The method comprises the steps of carrying out a first treatment on the surface of the The three axes of the coordinate system of the augmented reality device referenced by the augmented reality device are respectively marked as X _A 、Y _A And Z _A The method comprises the steps of carrying out a first treatment on the surface of the The three axes of the calibration box coordinate system to which the calibration box refers are respectively denoted as X _C 、Y _C And Z _C The method comprises the steps of carrying out a first treatment on the surface of the The three axes of the hand coordinate system referenced by the hands of the demonstrator are respectively denoted as X _H 、Y _H And Z _H The method comprises the steps of carrying out a first treatment on the surface of the The three axes of the virtual end effector coordinate system referenced by the end effector of the virtual robot are denoted as X, respectively _E ^V 、Y _E ^V And Z _E ^V (not shown in the drawings). The calibration box is used for positioning the position of the physical robot, so that the augmented reality device can construct and display a virtual robot with a base coincident with the base of the physical robot in the field of view of the demonstrator according to the position of the calibration box.

In order to obtain the transformation relationship between the various coordinate systems in fig. 5, it is necessary to establish the transformation relationship between the different coordinate systems. Illustratively, the transformation order between the different coordinate systems is in turn: a hand coordinate system, a somatosensory device coordinate system, a fifth joint coordinate system, a reference coordinate system, a calibration box coordinate system and an augmented reality device coordinate system. The motion sensing device is fixed on a fifth joint of the physical robot, and the transformation relation between the motion sensing device coordinate system and the fifth joint coordinate system is calibrated in advance. The transformation relationship between the fifth joint coordinate system and the reference coordinate system can be established through the forward kinematics model of the physical robot. The relationship between the calibration box coordinate system and the reference coordinate system has also been calibrated in advance. After the calibration box is captured by the glasses of the augmented reality device, the transformation relation between the coordinate system of the augmented reality device and the coordinate system of the calibration box can be calibrated.

Since the reference coordinate system of the base reference of the virtual robot overlaps with the reference coordinate system of the base reference of the physical robot, the coordinate values in the hand coordinate system can be converted into the coordinate values in the reference coordinate system of the virtual robot according to the conversion relationship between the coordinate systems, and can be used for teaching the virtual robot. Illustratively, after converting the coordinate values in the hand coordinate system into the coordinate values in the reference coordinate system, the coordinate values in the reference coordinate system can be further converted into the coordinate values in the virtual end effector coordinate system to which the end effector of the virtual robot refers, according to the relationship between joints of the virtual robot.

Step 3003: virtual teaching is performed on the virtual robot by using the sub-teaching action indicated by the teaching instruction.

After the teaching instruction is acquired, virtual teaching is performed on the virtual robot by utilizing the sub-teaching action indicated by the teaching instruction, so that the virtual robot executes the sub-teaching action indicated by the teaching instruction. Illustratively, the implementation of virtual teaching of the virtual robot using the sub-teaching action indicated by the teaching instruction includes: the computer device sends the teaching instruction to the augmented reality device, and the augmented reality device controls the virtual robot to execute the sub-teaching action indicated by the teaching instruction.

Since the virtual robot is constructed by the augmented reality device based on the physical robot, the process of controlling the virtual robot to perform actions is implemented by the augmented reality device. The augmented reality device, after receiving the teaching instruction sent by the computer device, can identify the sub-teaching action indicated by the teaching instruction, and then control the virtual robot to execute the sub-teaching action indicated by the teaching instruction. Illustratively, the sub-teaching action indicated by the teaching instruction refers to an action that the end effector of the virtual robot needs to execute, and the sub-teaching action indicated by the teaching instruction is controlled to be executed by the end effector of the virtual robot. In the process of controlling the end effector of the virtual robot to execute the sub-teaching action indicated by the teaching instruction, one or more joints of the virtual robot also execute the action, and the action executed by the one or more joints is obtained by analyzing the sub-teaching action indicated by the teaching instruction through a reverse kinematic model.

Step 3004: and responding to the virtual teaching process meeting the first condition to obtain a second teaching action.

After virtual teaching is performed on the virtual robot by using the sub-teaching action indicated by the teaching instruction, judging whether the virtual teaching process meets the first condition, if the virtual teaching process does not meet the first condition, acquiring a new teaching instruction according to the mode from step 3001 to step 3003, and then performing virtual teaching on the virtual robot by using the sub-teaching action indicated by the new teaching instruction until the virtual teaching process meets the first condition. And when the virtual teaching process meets the first condition, obtaining a second teaching action corresponding to the virtual robot. The second teaching action is composed of sub-teaching actions which are sequentially executed by the virtual robot in the virtual teaching process.

In an exemplary embodiment, the virtual teaching process satisfies the first condition and is set empirically or flexibly adjusted according to an actual application scenario, which is not limited in the embodiment of the present application. Illustratively, the virtual teaching process meeting the first condition means that no teaching information is acquired within a reference period, which is set empirically or flexibly adjusted according to an actual application scenario, and the embodiment of the present application is not limited to this, for example, the reference period refers to 3 minutes. Illustratively, the virtual teaching process meeting the first condition means that the end effector of the virtual robot reaches a target position point, where the end effector of the virtual robot needs to be located when the virtual task is completed, and the target position point is determined according to the type of the virtual task, which is not limited in the embodiment of the present application.

After the second teaching action is acquired, the first teaching action required to be executed by the physical robot is acquired based on the second teaching action. The number of sub-teaching actions included in the first teaching action is the same as the number of sub-teaching actions included in the second teaching action. In one possible implementation, based on the second teaching action, acquiring the first teaching action may refer to acquiring the first teaching action directly based on the second teaching action; the first teaching operation may be acquired first after the correction, and then the first teaching operation may be acquired based on the corrected second teaching operation.

For example, for the case of acquiring the first teaching action based on the second teaching action, the first teaching action is acquired directly based on the second teaching action, and the acquiring process of the first teaching action is as follows: the second teaching action is down-converted from the virtual end effector coordinate system to the physical end effector coordinate system, and the first teaching action is obtained. The first teaching action is obtained by down-converting the second teaching action from the virtual end effector coordinate system to the reference coordinate system, and then down-converting the third teaching action from the reference coordinate system to the physical end effector coordinate system.

For example, for the case of acquiring the first teaching action based on the second teaching action, the corrected second teaching action is acquired first, and then the first teaching action is acquired based on the corrected second teaching action, before the first teaching action is acquired, the second teaching action needs to be corrected first, and the corrected second teaching action is acquired. One or more sub-teaching actions with low accuracy may exist in the second teaching action, the second teaching action is corrected, the sub-teaching action with low accuracy can be corrected, and the corrected second teaching action is high in accuracy. The computer device may perform correction on the second teaching action according to the intention of the human being, or may perform correction on the second teaching action according to a correction instruction of the staff, which is not limited in the embodiment of the present application. After the corrected second teaching action is acquired, the first teaching action is acquired based on the corrected second teaching action. The principle of acquiring the first teaching action based on the corrected second teaching action is the same as the principle of acquiring the first teaching action directly based on the second teaching action, and will not be described here again.

In an exemplary embodiment, the process of correcting the second teaching action includes, but is not limited to, workpiece alignment, track correction, and the like. Workpiece alignment refers to: for the assembly task of placing the workpiece into the hole, automatic alignment of the virtual workpiece with the actual hole site is provided. Firstly, a segmentation algorithm (such as a watershed algorithm) is applied to segment an image of a real workpiece, the image is divided into different areas, and then the areas and the corresponding edges and centroids of the areas are extracted. Based on the depth image, coordinates of edge points in a somatosensory device coordinate system are calculated and then converted into a world coordinate system. Finally, the position and the direction of the virtual workpiece can be automatically adjusted to align the virtual workpiece with the hole, namely, the workpiece alignment is completed, and the second teaching action is corrected from the aspect that the virtual workpiece can be aligned with the hole. Trajectory correction refers to: for some trajectory tracking tasks, a teaching trajectory may be inferred, and a second teaching action corrected based on a difference between the inferred teaching trajectory and the trajectory performing the second teaching action.

In an exemplary embodiment, after the second teaching action is acquired, it is determined whether the second teaching action satisfies the second condition, and when it is determined that the second teaching action satisfies the second condition, a process based on the second teaching action is executed to acquire the first teaching action. That is, in response to the second teaching action satisfying the second condition, the first teaching action is acquired based on the second teaching action. In this case, if the second teaching operation does not satisfy the second condition, the virtual teaching is continued on the virtual robot until the second teaching operation satisfying the second condition is obtained. The second condition is satisfied and is set empirically or flexibly adjusted according to the application scene. For example, the second teaching action meeting the second condition means that the number of sub-teaching actions included in the second teaching action is not greater than a number threshold, and the number threshold is set empirically, so that the task execution process can be prevented from being excessively complicated.

Illustratively, the second teaching action satisfying the second condition means that a difference between a task completed by the virtual robot by executing the second teaching action and the virtual task is not greater than a difference threshold, and a determination manner of the difference between the two tasks is related to a type of the task, which is not limited in this embodiment of the present application, for example, the difference between the two trajectory tracking tasks means a difference between the tracked trajectories. The difference threshold is empirically set, and the accuracy of the second teaching action can be ensured by the setting mode of the second condition.

After the first teaching action is acquired, the computer equipment controls the entity robot to execute the first teaching action. In one possible implementation manner, the control of the physical robot to perform the first teaching action refers to controlling an end effector of the physical robot to perform the first teaching action, and in the process of controlling the end effector of the physical robot to perform the first teaching action, one or more joints of the physical robot also perform the action, where the action performed by the one or more joints is obtained by analyzing through a inverse kinematics model according to the first teaching action.

In an exemplary embodiment, the physical robot is directly communicatively coupled to a computer device, in which case the computer device is capable of directly controlling the physical robot to perform the first teaching action. In an exemplary embodiment, the physical robot is communicatively coupled to the computer device via the controller, in which case the computer device sends a first teaching action to the controller, which controls the physical robot to perform the first teaching action.

In the process of controlling the entity robot to execute the first teaching action, the computer equipment monitors whether the entity robot collides with the obstacle in real time. In response to the physical robot colliding with the obstacle, haptic feedback is provided, the haptic feedback being used to indicate the presence of a collision event. By providing haptic feedback, the demonstrator can intuitively perceive the collision event, which is beneficial to the demonstrator to generate more targeted and reliable adjustment instructions, thereby improving the teaching effect of the physical robot.

In one possible implementation, a force sensor is mounted on the physical robot, and the physical robot is determined to collide with the obstacle in response to the force detected by the force sensor satisfying a collision detection condition. The position where the force sensor is mounted on the physical robot is related to a position where the force sensor needs to detect that the force sensor may collide with an obstacle, for example, the collision between the physical robot and the obstacle refers to that the end effector of the physical robot collides with the obstacle, that is, the position where the force sensor needs to detect that the force sensor may collide with the obstacle is the end effector of the physical robot, in which case the force sensor is mounted on the end effector of the physical robot. Illustratively, an obstacle refers to a body that is not expected to collide during the course of a physical robot performing a task. For example, in the trajectory tracking task, a subject who does not wish to collide may indicate a certain workpiece, in which case a phenomenon in which the physical robot collides with the workpiece may occur due to the physical robot gripping the workpiece, and at this time, the collision force between the physical robot and the obstacle may be referred to as a gripping force.

The force sensor is used for detecting force, the detected force is sent to the computer equipment, the computer equipment can judge whether the force detected by the force sensor meets the collision detection condition, and if the force detected by the force sensor meets the collision detection condition, the physical robot is determined to collide with the obstacle. Illustratively, the force satisfying the collision detection condition means that the magnitude of the force is not less than a first threshold value, which is empirically set, and if the first threshold value is 0, the force detected by the force sensor satisfies the collision detection condition. Illustratively, the force meeting the collision detection condition refers to a stage in which the force is detected being a particular stage in the process of controlling the physical robot to perform the first teaching action. The specific stage is a stage in which collision with an obstacle is not desired, and is not limited to the embodiment of the present application, as to a task that is actually required to be performed.

Upon determining that the physical robot collides with an obstacle, haptic feedback is provided. In an exemplary embodiment, the teach pendant is worn with a haptic feedback device, and the computer device providing haptic feedback means that the computer device controls the haptic feedback device to provide haptic feedback. In one possible implementation, the process of controlling the haptic feedback device to provide haptic feedback includes: determining a target current based on a collision force between the physical robot and the obstacle; a target current is applied to the haptic feedback device such that the haptic feedback device provides haptic feedback under the influence of the target current. The collision force between the physical robot and the obstacle is the force detected by the force sensor, and in one possible implementation, the manner of determining the target current based on the collision force is: based on the collision force, the target current is calculated according to maxwell's equations.

In an exemplary embodiment, the haptic feedback device has a coil and a magnet therein, and the target current refers to a current that needs to be applied to the coil in the haptic feedback device, and for an exemplary case where a plurality of coils are included in the haptic feedback device, the number of target currents is also a plurality. The coils in the haptic feedback device are capable of causing the magnets in the haptic feedback device to generate magnetic forces that are used to provide haptic feedback to the teach pendant after the application of an electrical current.

In an exemplary embodiment, a schematic diagram of a haptic feedback device is shown in FIG. 6. FIG. 6 (1) is a schematic diagram of a haptic feedback device worn on a digit of a teach pendant; FIG. 6 (2) is a schematic diagram of a haptic feedback device; a product diagram of the haptic feedback device is shown in fig. 6 (3). As shown in (2) and (3) of fig. 6, the tactile feedback device has a coil, a magnet, and a cotton pad therein. The coil and the magnet are distributed on four rotatable footholds, and the cotton pad is distributed at the contact part with the finger. The magnet is capable of generating a magnetic force under the influence of an electric current applied to the coil.

The size and the structure of the coil are not limited, the size and the type of the magnet are not limited, and the coil can be flexibly adjusted according to actual application scenes. For example, each coil is 12.5mm in diameter and 34.5mm in height, and the coil is composed of 350 turns of enamelled copper. The magnet is a cylindrical neodymium magnet, and the diameter and the height of the magnet are respectively 10mm and 2mm. The haptic feedback device adjusts fingertip haptic by controlling the current through the coil. The demonstrator can feel the collision between the end effector of the physical robot and the obstacle only by simply wearing the designed tactile feedback device, and the immersion of the operation can not be influenced.

According to the embodiment of the application, the tactile feedback device can be used for providing tactile feedback for the demonstrator in man-machine interaction, the tactile feedback device allows the non-contact gesture interface to provide the tactile feedback, and the collision event which is measured remotely can be transmitted to the demonstrator through sense organs. The demonstrator can teach the robot through gestures, then receive visual feedback through the augmented reality device, obtain the tactile feedback through the tactile feedback device fixed on the fingertip for remote operation. In the teaching process, workpieces are easily damaged in a vision blind area due to accidental collision only by means of visual feedback. Through tactile feedback, a demonstrator can quickly detect occurrence of collision events and timely adjust teaching actions.

In an exemplary embodiment, during the computer device controlling the physical robot to perform the first teaching action, the teach pendant can receive visual feedback through the augmented reality device in addition to tactile feedback. According to the visual feedback, the demonstrator can observe the movement of the physical robot in real time. The visual feedback and the tactile feedback are combined, so that more comprehensive feedback can be provided for a demonstrator, and the reliability of an adjustment instruction generated by the demonstrator is further improved.

In step 302, the first teaching action is adjusted by using the adjustment instruction obtained after the haptic feedback is provided, so as to obtain a target teaching action meeting the non-collision condition, and the teaching of the physical robot is completed based on the target teaching action.

After providing the haptic feedback, the computer device is able to adjust the first teaching action using the adjustment instructions obtained after providing the haptic feedback. The adjustment instruction obtained after the haptic feedback is provided is an adjustment instruction generated by a demonstrator according to the prompt of the haptic feedback after the haptic feedback is received, and the demonstrator can intuitively sense that the physical robot collides with the obstacle in the process of executing the first teaching action according to the haptic feedback, so that the demonstrator can generate a more reliable adjustment instruction according to the prompt of the haptic feedback after the haptic feedback is received, and the computer equipment can obtain the adjusted teaching action capable of effectively reducing the collision probability of the physical robot and the obstacle according to the adjustment instruction.

In the exemplary embodiment, since the demonstrator can observe the motion of the physical robot in real time according to the visual feedback, the demonstrator can determine the possible cause of the collision by combining the prompt of the tactile feedback and the observed motion condition after receiving the tactile feedback, thereby generating an adjustment instruction capable of effectively reducing the collision probability.

The final purpose of the computer device for adjusting the first teaching action by using the adjustment instruction acquired after the haptic feedback is provided is to obtain a target teaching action meeting the non-collision condition, wherein the target teaching action is the action required to be executed by the finally determined physical robot to realize the task, and after the target teaching action meeting the non-collision condition is obtained, the teaching of the physical robot is completed based on the target teaching action. Illustratively, the teaching of the entity robot based on the target teaching action is to control the entity robot to execute the target teaching action and enable the entity robot to record a manner of executing the target teaching action, so that the entity robot can automatically execute the target teaching action to complete a corresponding task.

In an illustrative example, the adjustment instructions may be generated by at least one of a gesture or a natural interaction of speech, which is not limited by embodiments of the present application. After the computer equipment provides the tactile feedback, the computer equipment can acquire an adjustment instruction generated by a demonstrator according to the prompt of the tactile feedback through interaction with the somatosensory equipment or the augmented reality equipment.

In an exemplary embodiment, the instruction for adjusting the first teaching action generated by the demonstrator according to the prompt of the haptic feedback is an instruction for directly adjusting the first teaching action, and it can be clearly indicated how to adjust the first teaching action. In this case, the computer device adjusts the first teaching action using the adjustment instruction acquired after providing the haptic feedback means that the first teaching action is directly adjusted using the adjustment instruction acquired after providing the haptic feedback.

In an exemplary embodiment, in a case where the first teaching action is obtained based on the second teaching action corresponding to the virtual robot, the adjustment instruction generated by the demonstrator according to the indication of the haptic feedback may also be an instruction for adjusting the second teaching action, and it may be explicitly indicated how to adjust the second teaching action. In this case, the computer device adjusting the first teaching action by using the adjustment instruction obtained after providing the haptic feedback means that the computer device adjusts the second teaching action by using the adjustment instruction obtained after providing the haptic feedback, and obtains the adjusted first teaching action based on the adjusted second teaching action, thereby indirectly realizing the adjustment of the first teaching action.

After the first teaching action is adjusted by using the adjustment instruction obtained after the haptic feedback is provided, it is determined whether the adjusted first teaching action satisfies the non-collision condition. In an exemplary embodiment, the one teaching action satisfying the non-collision condition includes the physical robot not colliding with the obstacle during the controlling of the physical robot to perform the one teaching action. In an exemplary embodiment, the meeting of the non-collision condition by the one teaching action includes that, in addition to the physical robot not colliding with the obstacle in the process of controlling the physical robot to perform the one teaching action, a difference between a task completed by the physical robot and an actual task is smaller than a threshold value after the control of the physical robot to perform the one teaching action. In either case, if one teaching action satisfies the non-collision condition, it is at least ensured that the physical robot does not collide with the obstacle during the control of the physical robot to execute the one teaching action.

After the adjusted first teaching action is determined, whether the adjusted first teaching action meets the non-collision condition can be judged, and if the adjusted first teaching action meets the non-collision condition, the adjusted first teaching action is taken as a target teaching action meeting the non-collision condition. If the adjusted first teaching action does not meet the non-collision condition, the adjusted first teaching action still needs to be adjusted. For example, in the course of the subsequent adjustment, if the computer device provides the tactile feedback to the demonstrator, the demonstrator generates the adjustment instruction by combining the visual feedback and the tactile feedback, and if the computer device does not provide the tactile feedback to the demonstrator, the demonstrator may generate the adjustment instruction according to the visual feedback, which is not limited in the embodiment of the present application. After the first teaching action after adjustment is adjusted, judging whether the first teaching action after adjustment again meets the non-collision condition, and the like until the target teaching action meeting the non-collision condition is obtained.

In the embodiment of the present application, the physical robot is described as an example of collision with an obstacle during the process of controlling the physical robot to perform the first teaching operation, and the embodiment of the present application is not limited thereto. In an exemplary embodiment, in the process of controlling the physical robot to execute the first teaching action, the physical robot may not collide with the obstacle, in this case, the computer device does not need to provide haptic feedback, the computer device may determine whether the first teaching action meets the non-collision condition, and if the first teaching action meets the non-collision condition, the first teaching action may be directly taken as the target teaching action; if the first teaching action does not satisfy the non-collision condition, a prompt for adjustment may be sent to the teach-in person, and then the first teaching action may be adjusted by the teach-in person according to an adjustment instruction generated by visual feedback. In either case, the final objective is to obtain a target teaching action that satisfies the non-collision condition to complete teaching of the physical robot based on the target teaching action.

Illustratively, the embodiment of the application can teach the robot in an off-line fusion manner. Through virtual-real fusion interaction technology, a demonstrator can safely conduct virtual teaching on the virtual robot in a real scene directly, and then the physical robot reproduces the motion of the virtual robot to complete the teaching process. The teaching method combines gesture and voice interaction modes, and can provide visual feedback and tactile feedback for a demonstrator. In addition, the teaching method can rapidly verify teaching results while ensuring safety of a demonstrator and avoiding damage to an entity robot or a workpiece. Once errors exist between the physical robot motion and the motion to be taught, a demonstrator can finely tune the physical robot motion in real time through a gesture and voice teaching fusion algorithm.

According to the robot teaching method, in the process of controlling the entity robot to execute teaching actions, collision events between the entity robot and the obstacle are automatically detected, and when the collision events are detected, tactile feedback for prompting the existence of the collision events is provided, so that a demonstrator can intuitively perceive the collision events according to the tactile feedback. The reliability of the automatic collision event detection is higher, the adjustment instruction obtained after the haptic feedback is provided is a more reliable adjustment instruction generated by a demonstrator according to the prompt of the haptic feedback, the quality of adjusting teaching actions by using the adjustment instruction obtained after the haptic feedback is provided is higher, and the improvement of the teaching effect of the robot is facilitated.

Referring to fig. 7, an embodiment of the present application provides a robot teaching device, including:

a control unit 701, configured to provide haptic feedback in response to collision of the physical robot with the obstacle during control of the physical robot to perform the first teaching action, where the haptic feedback is used to prompt that a collision event exists;

and the adjusting unit 702 is configured to adjust the first teaching action by using an adjustment instruction acquired after the haptic feedback is provided, obtain a target teaching action that meets the non-collision condition, and complete teaching of the physical robot based on the target teaching action.

In one possible implementation, the control unit 701 is configured to determine the target current based on a collision force between the physical robot and the obstacle; a target current is applied to the haptic feedback device such that the haptic feedback device provides haptic feedback under the influence of the target current.

In one possible implementation, referring to fig. 8, the apparatus further includes:

an acquiring unit 703, configured to acquire a second teaching action corresponding to the virtual robot; based on the second teaching action, a first teaching action is acquired.

In one possible implementation, the obtaining unit 703 is configured to obtain teaching information, where the teaching information includes at least one of gesture information and voice information; based on the teaching information, acquiring a teaching instruction, wherein the teaching instruction is used for indicating sub-teaching actions; virtual teaching is carried out on the virtual robot by utilizing the sub-teaching action indicated by the teaching instruction; and responding to the virtual teaching process meeting the first condition to obtain a second teaching action.

In one possible implementation, the teaching information includes gesture information and voice information, and the acquiring unit 703 is configured to acquire a fusion text based on the gesture information and the voice information; invoking a classification model to classify the fusion text to obtain the matching probability of each candidate instruction, wherein the classification model is obtained based on the sample text and instruction labels corresponding to the sample text; and acquiring a teaching instruction based on the candidate instruction with the matching probability meeting the selection condition.

In one possible implementation manner, the acquiring unit 703 is configured to correct the second teaching action to obtain a corrected second teaching action; and acquiring the first teaching action based on the corrected second teaching action.

In one possible implementation, the virtual robot is built by the augmented reality device based on a physical robot.

In one possible implementation, the obtaining unit 703 is configured to send a teaching instruction to the augmented reality device, where the augmented reality device controls the virtual robot to perform a sub-teaching action indicated by the teaching instruction.

In one possible implementation, the physical robot is mounted with a force sensor, and the control unit 701 is further configured to determine that the physical robot collides with the obstacle in response to the force detected by the force sensor meeting the collision detection condition.

According to the robot teaching device, in the process of controlling the entity robot to execute teaching actions, collision events between the entity robot and obstacles are automatically detected, and when the collision events are detected, tactile feedback for prompting the existence of the collision events is provided, so that a demonstrator can intuitively perceive the collision events according to the tactile feedback. The reliability of the automatic collision event detection is higher, the adjustment instruction obtained after the haptic feedback is provided is a more reliable adjustment instruction generated by a demonstrator according to the prompt of the haptic feedback, the quality of adjusting teaching actions by using the adjustment instruction obtained after the haptic feedback is provided is higher, and the improvement of the teaching effect of the robot is facilitated.

It should be noted that, when the apparatus provided in the foregoing embodiment performs the functions thereof, only the division of the functional units is used as an example, and in practical application, the functional allocation may be performed by different functional units according to needs, that is, the internal structure of the device is divided into different functional units, so as to perform all or part of the functions described above. In addition, the apparatus and the method embodiments provided in the foregoing embodiments belong to the same concept, and specific implementation processes of the apparatus and the method embodiments are detailed in the method embodiments and are not repeated herein.

In an exemplary embodiment, a computer device is also provided, the computer device comprising a processor and a memory, the memory having at least one computer program stored therein. The at least one computer program is loaded and executed by one or more processors to cause the computer arrangement to implement any one of the robot teaching methods described above. The computer device may be a terminal or a server, which is not limited in this embodiment of the present application. Next, the structures of the terminal and the server will be described, respectively.

Fig. 9 is a schematic structural diagram of a terminal according to an embodiment of the present application. The terminal may be: PC, cell-phone, smart mobile phone, PDA, wearable equipment, PPC, panel computer, intelligent car machine, smart TV, intelligent audio amplifier, on-vehicle terminal. Terminals may also be referred to by other names as user equipment, portable terminals, laptop terminals, desktop terminals, etc.

Generally, the terminal includes: a processor 901 and a memory 902.

Processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing ), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array ). Processor 901 may also include a main processor, which is a processor for processing data in an awake state, also called a CPU, and a coprocessor; a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 901 may integrate a GPU (Graphics Processing Unit, image processor) for taking care of rendering and drawing of content that the display screen needs to display. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence ) processor for processing computing operations related to machine learning.

The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 902 is configured to store at least one instruction for execution by processor 901 to cause the terminal to implement the robot teaching method provided by the method embodiments in the present application.

In some embodiments, the terminal may further optionally include: a peripheral interface 903, and at least one peripheral. The processor 901, memory 902, and peripheral interface 903 may be connected by a bus or signal line. The individual peripheral devices may be connected to the peripheral device interface 903 via buses, signal lines, or circuit boards. Specifically, the peripheral device includes: at least one of radio frequency circuitry 904, a display 905, a camera assembly 906, audio circuitry 907, a positioning assembly 908, and a power source 909.

The peripheral interface 903 may be used to connect at least one peripheral device associated with an I/O (Input/Output) to the processor 901 and the memory 902. In some embodiments, the processor 901, memory 902, and peripheral interface 903 are integrated on the same chip or circuit board; in some other embodiments, either or both of the processor 901, the memory 902, and the peripheral interface 903 may be implemented on separate chips or circuit boards, which is not limited in this embodiment.

The Radio Frequency circuit 904 is configured to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 904 communicates with a communication network and other communication devices via electromagnetic signals. The radio frequency circuit 904 converts an electrical signal into an electromagnetic signal for transmission, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 904 includes: antenna systems, RF transceivers, one or more amplifiers, tuners, oscillators, digital signal processors, codec chipsets, subscriber identity module cards, and so forth. The radio frequency circuit 904 may communicate with other terminals via at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and/or WiFi (Wireless Fidelity ) networks. In some embodiments, the radio frequency circuit 904 may also include NFC (Near Field Communication ) related circuits, which are not limited in this application.

The display 905 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display 905 is a touch display, the display 905 also has the ability to capture touch signals at or above the surface of the display 905. The touch signal may be input as a control signal to the processor 901 for processing. At this time, the display 905 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard. In some embodiments, the display 905 may be one and disposed on the front panel of the terminal; in other embodiments, the display 905 may be at least two, respectively disposed on different surfaces of the terminal or in a folded design; in other embodiments, the display 905 may be a flexible display disposed on a curved surface or a folded surface of the terminal. Even more, the display 905 may be arranged in an irregular pattern other than rectangular, i.e., a shaped screen. The display 905 may be made of LCD (Liquid Crystal Display ), OLED (Organic Light-Emitting Diode) or other materials.

The camera assembly 906 is used to capture images or video. Optionally, the camera assembly 906 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal and the rear camera is disposed on the rear surface of the terminal. In some embodiments, the at least two rear cameras are any one of a main camera, a depth camera, a wide-angle camera and a tele camera, so as to realize that the main camera and the depth camera are fused to realize a background blurring function, and the main camera and the wide-angle camera are fused to realize a panoramic shooting and Virtual Reality (VR) shooting function or other fusion shooting functions. In some embodiments, camera assembly 906 may also include a flash. The flash lamp can be a single-color temperature flash lamp or a double-color temperature flash lamp. The dual-color temperature flash lamp refers to a combination of a warm light flash lamp and a cold light flash lamp, and can be used for light compensation under different color temperatures.

The audio circuit 907 may include a microphone and a speaker. The microphone is used for collecting sound waves of users and the environment, converting the sound waves into electric signals, and inputting the electric signals to the processor 901 for processing, or inputting the electric signals to the radio frequency circuit 904 for voice communication. For the purpose of stereo acquisition or noise reduction, a plurality of microphones can be respectively arranged at different parts of the terminal. The microphone may also be an array microphone or an omni-directional pickup microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a conventional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only the electric signal can be converted into a sound wave audible to humans, but also the electric signal can be converted into a sound wave inaudible to humans for ranging and other purposes. In some embodiments, the audio circuit 907 may also include a headphone jack.

The location component 908 is used to locate the current geographic location of the terminal to enable navigation or LBS (Location Based Service, location-based services). The positioning component 908 may be a positioning component based on the United states GPS (Global Positioning System ), the Beidou system of China, the Granati system of Russia, or the Galileo system of the European Union.

The power supply 909 is used to supply power to the various components in the terminal. The power supply 909 may be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 909 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery may also be used to support fast charge technology.

In some embodiments, the terminal further includes one or more sensors 910. The one or more sensors 910 include, but are not limited to: acceleration sensor 911, gyroscope sensor 912, pressure sensor 913, fingerprint sensor 914, optical sensor 915, and proximity sensor 916.

The acceleration sensor 911 can detect the magnitudes of accelerations on three coordinate axes of a coordinate system established with a terminal. For example, the acceleration sensor 911 may be used to detect components of gravitational acceleration in three coordinate axes. The processor 901 may control the display 905 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal acquired by the acceleration sensor 911. The acceleration sensor 911 may also be used for the acquisition of motion data of a game or a user.

The gyro sensor 912 may detect a body direction and a rotation angle of the terminal, and the gyro sensor 912 may collect a 3D motion of the user to the terminal in cooperation with the acceleration sensor 911. The processor 901 may implement the following functions according to the data collected by the gyro sensor 912: motion sensing (e.g., changing UI according to a tilting operation by a user), image stabilization at shooting, game control, and inertial navigation.

The pressure sensor 913 may be provided at a side frame of the terminal and/or at a lower layer of the display 905. When the pressure sensor 913 is disposed on a side frame of the terminal, a grip signal of the terminal by a user may be detected, and the processor 901 performs left-right hand recognition or quick operation according to the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is provided at the lower layer of the display 905, the processor 901 performs control of the operability control on the UI interface according to the pressure operation of the user on the display 905. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

The fingerprint sensor 914 is used for collecting the fingerprint of the user, and the processor 901 identifies the identity of the user according to the fingerprint collected by the fingerprint sensor 914, or the fingerprint sensor 914 identifies the identity of the user according to the collected fingerprint. Upon recognizing that the user's identity is a trusted identity, the processor 901 authorizes the user to perform relevant sensitive operations including unlocking the screen, viewing encrypted information, downloading software, paying for and changing settings, etc. The fingerprint sensor 914 may be provided on the front, back or side of the terminal. When a physical key or a manufacturer Logo (trademark) is provided on the terminal, the fingerprint sensor 914 may be integrated with the physical key or the manufacturer Logo.

The optical sensor 915 is used to collect the intensity of ambient light. In one embodiment, the processor 901 may control the display brightness of the display panel 905 based on the intensity of ambient light collected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display luminance of the display screen 905 is turned up; when the ambient light intensity is low, the display luminance of the display panel 905 is turned down. In another embodiment, the processor 901 may also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity collected by the optical sensor 915.

A proximity sensor 916, also referred to as a distance sensor, is typically provided on the front panel of the terminal. The proximity sensor 916 is used to collect the distance between the user and the front of the terminal. In one embodiment, when the proximity sensor 916 detects that the distance between the user and the front of the terminal gradually decreases, the processor 901 controls the display 905 to switch from the bright screen state to the off screen state; when the proximity sensor 916 detects that the distance between the user and the front surface of the terminal gradually increases, the processor 901 controls the display 905 to switch from the off-screen state to the on-screen state.

It will be appreciated by those skilled in the art that the structure shown in fig. 9 is not limiting of the terminal and may include more or fewer components than shown, or may combine certain components, or may employ a different arrangement of components.

Fig. 10 is a schematic structural diagram of a server provided in the embodiment of the present application, where the server may include one or more processors (Central Processing Units, CPU) 1001 and one or more memories 1002, where the one or more memories 1002 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1001, so that the server implements the robot teaching method provided in the foregoing method embodiments. Of course, the server may also have a wired or wireless network interface, a keyboard, an input/output interface, and other components for implementing the functions of the device, which are not described herein.

In an exemplary embodiment, there is also provided a computer-readable storage medium having stored therein at least one computer program loaded and executed by a processor of a computer apparatus to cause the computer to implement any one of the robot teaching methods described above.

In one possible implementation, the computer readable storage medium may be a Read-Only Memory (ROM), a random-access Memory (Random Access Memory, RAM), a compact disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

In an exemplary embodiment, a computer program product is also provided, which comprises a computer program or computer instructions that are loaded and executed by a processor to cause the computer to implement any one of the robot teaching methods described above.

It should be understood that references herein to "a plurality" are to two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.

The foregoing description of the exemplary embodiments of the present application is not intended to limit the invention to the particular embodiments disclosed, but on the contrary, the intention is to cover all modifications, equivalents, alternatives, and alternatives falling within the spirit and scope of the invention.

Claims

1. A robot teaching method, the method comprising:

2. The method of claim 1, wherein the providing haptic feedback comprises:

determining a target current based on a collision force between the physical robot and the obstacle;

the target current is applied to a haptic feedback device such that the haptic feedback device provides the haptic feedback under the influence of the target current.

3. The method of claim 1, wherein the method further comprises, prior to providing haptic feedback in response to the physical robot colliding with an obstacle during control of the physical robot to perform the first teaching action:

acquiring a second teaching action corresponding to the virtual robot;

and acquiring the first teaching action based on the second teaching action.

4. The method of claim 3, wherein the obtaining a second teaching action corresponding to the virtual robot comprises:

acquiring teaching information, wherein the teaching information comprises at least one of gesture information and voice information;

Based on the teaching information, acquiring a teaching instruction, wherein the teaching instruction is used for indicating sub-teaching actions;

virtual teaching is carried out on the virtual robot by utilizing the sub-teaching action indicated by the teaching instruction;

and responding to the virtual teaching process meeting the first condition, and obtaining the second teaching action.

5. The method of claim 4, wherein the teaching information includes gesture information and voice information, and wherein the acquiring teaching instructions based on the teaching information includes:

acquiring a fusion text based on the gesture information and the voice information;

invoking a classification model to classify the fusion text to obtain the matching probability of each candidate instruction, wherein the classification model is obtained based on a sample text and instruction labels corresponding to the sample text;

and acquiring the teaching instruction based on the candidate instruction with the matching probability meeting the selection condition.

6. The method according to any one of claims 3-5, wherein the obtaining the first teaching action based on the second teaching action includes:

correcting the second teaching action to obtain a corrected second teaching action;

And acquiring the first teaching action based on the corrected second teaching action.

7. The method of any of claims 3-5, wherein the virtual robot is constructed by an augmented reality device based on the physical robot.

8. The method according to claim 4, wherein the virtual teaching of the virtual robot using the sub-teaching action indicated by the teaching instruction includes:

and sending the teaching instruction to augmented reality equipment, and controlling the virtual robot to execute sub-teaching actions indicated by the teaching instruction by the augmented reality equipment.

9. The method of any of claims 1-5, 8, wherein the physical robot has a force sensor mounted thereon, and wherein the method further comprises, prior to providing haptic feedback in response to the physical robot colliding with an obstacle:

and determining that the physical robot collides with the obstacle in response to the force detected by the force sensor meeting a collision detection condition.

10. The method of claim 5, wherein the classification model is a maximum entropy model.

11. A robot teaching device, the device comprising:

12. A computer device, characterized in that it comprises a processor and a memory, in which at least one computer program is stored, which is loaded and executed by the processor, so that the computer device implements the robot teaching method according to any of claims 1 to 10.

13. A computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor, so that the computer implements the robot teaching method according to any one of claims 1 to 10.

14. A computer program product, characterized in that the computer program product comprises a computer program or computer instructions that are loaded and executed by a processor to cause the computer to implement the robot teaching method according to any of claims 1 to 10.