Classifications

• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
  • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
  • B25J9/00—Programme-controlled manipulators
  • B25J9/16—Programme controls
  • B25J9/1679—Programme controls characterised by the tasks executed
  • B25J9/1689—Teleoperation
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B2219/00—Program-control systems
  • G05B2219/30—Nc systems
  • G05B2219/35—Nc in input of data, input till input file format
  • G05B2219/35444—Gesture interface, controlled machine observes operator, executes commands
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B2219/00—Program-control systems
  • G05B2219/30—Nc systems
  • G05B2219/37—Measurements
  • G05B2219/37567—3-D vision, stereo vision, with two cameras

Definitions

• the present invention relates to a computer-implemented method for the realtime control of an anthropomorphic robot and to a related system for the realtime control.
• the main aim of the present invention is to devise a computer-implemented method for the real-time control of an anthropomorphic robot and a related system for the real-time control which allows effective real-time control of a robot.
• Figure 1 is a schematic view of a robotic cell of the system according to the invention.
• Figure 2 is a schematic view of a real-time control station of the system according to the invention.
• Figure 3 schematically illustrates a possible image displayed inside the control station and collected by a camera on the anthropomorphic robot inside the robotic cell;
• Figure 4 is a general block diagram of the computer-implemented method according to the invention.
• reference numeral 1 globally relates to a system for the real-time control of an anthropomorphic robot 2.
• the system 1 comprises a robotic cell 3, schematically shown in Figure 1.
• the robotic cell 3 comprises the anthropomorphic robot 2 and at least one area 4 intended to house at least one object 5 to be handled by means of the anthropomorphic robot 2.
• the robotic cell 3 comprises a main area 4 intended to house a plurality of objects 5 which are heterogeneous with respect to each other to be sorted and at least one secondary area 6 intended to house the sorted objects 5 of the same type.
• the main area 4 inside the robotic cell 3 may comprise at least one bin containing a plurality of objects 5 which are heterogeneous with respect to each other to be sorted by means of the anthropomorphic robot 2, while the secondary area 6 may comprise a plurality (two or more) of other bins intended to house the sorted objects 5.
• the anthropomorphic robot 2 comprises grippers 7 which are adapted to grasp and handle the object 5.
• the grippers 7 of the anthropomorphic robot 2 may have two or three jaws.
• the grippers 7 are preferably made in a claw fashion (i.e. with a compass closure). The use of grippers 7 of different types cannot however be ruled out.
• the anthropomorphic robot 2 comprises at least one camera 8 arranged at the point where the grippers 7 are located.
• the camera 8 is arranged in a substantially central position between the grippers 7.
• the camera 8 is built into the wrist of the anthropomorphic robot 2 in a central position between the grippers 7.
• the robotic cell 3 comprises an additional camera 9 associated with the anthropomorphic robot 2 in a different position from the camera 8 and/or arranged in a fixed position inside the robotic cell itself.
• the additional camera 9 may be positioned on the wrist of the anthropomorphic robot 2 but externally to the grippers 7.
• the additional camera 9 may be positioned inside the robotic cell 3, at the top and in a fixed position, so as to frame the entire working area of the anthropomorphic robot 2.
• system 1 comprises at least one distance sensor 10 associated with the grippers 7 of the anthropomorphic robot 2.
• the distance sensor 10 is used to detect the height (vertical distance) of the grippers 7 with respect to an object 5 to be picked up.
• the distance sensor 10 may comprise at least one of either an ultrasonic sensor, a laser sensor, or a contact sensor.
• the system 1 also comprises a real-time control station 11 of the anthropomorphic robot 2, schematically shown in Figure 2.
• the real-time control station 11 comprises at least one processing unit 12 operationally connected to the anthropomorphic robot 2.
• the processing unit 12 may be composed of one or more processors of the type of personal computers or the like.
• the real-time control station 11 comprises graphical user interface means 13 accessible by an operator O, operationally connected to the processing unit 12 and configured to display the images detected by said camera 8 and/or the images detected by the additional camera 9.
• the real-time control station 11 also comprises at least one three-dimensional vision sensor 14 operationally connected to the processing unit 12 and configured to detect the movements of at least the hands of the operator O.
• the three-dimensional vision sensor 14 is a high-speed stereoscopic sensor.
• the three-dimensional vision sensor 14 is preferably positioned in front of the operator O and frames the operator’ s hands to detect the movements thereof.
• the three-dimensional vision sensor 14 sends the acquired images to the processing unit 12 in a continuous mode.
• the processing unit 12 comprises processing means 15 for the execution of the computer-implemented method described below for the real-time control of the anthropomorphic robot 2.
• Such processing means 15 are implemented by means of one or more dedicated software programs.
• processing means 15 for the execution of the computer- implemented method comprise a software program for the recognition of human gestures (and in particular all joints, or axes of movement, of human limbs).
• the recognition software of human gestures is able to recognize the joint of the elbow, of the wrist, the knuckles of the fingers, the phalanges, the ends of the fingers of the right and left hand.
• processing means 15 comprise a software program for the management of the robotic cell 3 configured to receive, at input, commands from the software for the recognition of gestures, input images from the camera 8, as well as from one or more additional cameras 9, as well as distance data from the distance sensor 10.
• the software program for the management of the robotic cell 3 also receives input data from the control of the anthropomorphic robot 2, such as linear and reorientation speed, motion status, and status of the grippers 7.
• the software program for the management of the robotic cell 3 sends predefined (linear or reorientation) movement instructions and commands to the anthropomorphic robot 2, as well as data regarding the speed to be set (linear and reorientation).
• the computer- implemented method 100 for the real-time control of an anthropomorphic robot is described in detail below and is schematically shown in Figure 4.
• the computer-implemented method 100 comprises at least the following steps:
• step 140 moving the anthropomorphic robot 2 depending on the detected movements of the operator O for the real-time handling of the object 5
• step 130 of detecting the movements of at least the hands of the operator O comprises at least the following steps:
• the step of identifying at least one command for the real-time movement of the anthropomorphic robot 2 comprises at least the following steps:
• RX, RY, RZ a rotary-translation matrix of the hands and/or of other parts of the body of the operator O in the space starting from the coordinates (X, Y, Z);
• the coordinates (X, Y, Z) of the axes of movement comprise coordinates relating to the knuckles of the index, middle, ring and little fingers of one hand of the operator O.
• the method 100 comprises at least one step of selecting a (linear and reorientation) speed of movement of said anthropomorphic robot.
• the anthropomorphic robot 2 is first switched to the automatic mode (motors on and automatic status).
• the computer- implemented method 100 comprises at least one preliminary step of positioning the anthropomorphic robot 2 in an initial home position which is retracted at the center of the robotic cell 3, as a result of a predefined command which is detected using the three-dimensional vision sensor 14.
• the initial home position command may be given by the operator O with his left hand (closed fist and thumb down).
• the anthropomorphic robot 2 moves until it reaches the home position (out of the way of all bins and in a retracted position at the center of the cell). If the operator O intends to stop the movement, he/she can raise his/her left open hand (stop).
• Such commands are detected by the three-dimensional vision sensor 14 and interpreted by the software program for the gestures control implemented on the processing unit 12.
• the method 100 comprises, as a result of a predefined command, which is detected by means of the three-dimensional vision sensor 14, at least one positioning step of the anthropomorphic robot 2 in a start picking position, wherein the grippers 7 are open and arranged above the object 5 (or objects) to be handled.
• the operator O may give the command for the start picking position with his/her left hand (closed fist and thumb upwards).
• the anthropomorphic robot 2 moves until it reaches the start picking position above and at the center of the bin of the objects 5 to be picked up, with the grippers 7 open. If the operator O wishes to stop the movement, he/she may raise his/her left open hand (stop).
• Such commands are detected by the three-dimensional vision sensor 14 and interpreted by the software program for the gesture control implemented on the processing unit 12.
• the method 100 also comprises, as a result of a predefined command which is detected by means of said three-dimensional vision sensor, at least one step of starting a real-time operating mode of the anthropomorphic robot 2, wherein the anthropomorphic robot 2 is ready to receive and execute real-time commands.
• the anthropomorphic robot 2 is switched to the run mode in real time (e.g., by means of closed fist command without thumb up or down).
• the anthropomorphic robot 2 is ready to receive movement commands in the various directions by means of the relevant gestures of the operator O. If the operator wishes to stop the movement, he/she will raise his/her left open hand (stop).
• Such commands are detected by the three-dimensional vision sensor 14 and interpreted by the software program for the gesture control implemented on the processing unit 12.
• the operator O sees the image of the objects O to be picked up inside the bin by means of the camera 8 on the wrist of the anthropomorphic robot 2, thus from the “point of view” of the grippers 7.
• the position and rotation of the grippers 7 (and consequently of the jaws) is always correctly visible, since the rotational data of the wrist axis are continuously sent by the anthropomorphic robot 2 to the management software program, which will update the graphics displayed by the graphical user interface means 13 accordingly.
• the jaws are displayed superimposed on the displayed image and are scaled in size depending on the distance between the grippers 7 and the objects 5 in the bin, detected by the distance sensor 10.
• the aforementioned steps of 130 and 140 of detecting the movements of the operator O and of moving the anthropomorphic robot 2 comprise, respectively, at least the following steps:
• step 141 moving the grippers 7 along such at least one direction in the horizontal and/or vertical plane (step 141).
• the operator O by means of gestures, may position himself/herself on the object 5 he/she wishes to pick up.
• the movement of the anthropomorphic robot 2 in the selected direction continues until the operator O carries out the opposite movement (if he/she has moved his/her hand forward, he/she must move it slightly backward): the change of direction is interpreted as a stop. Forward-right, backward-right, forward-left, backward-left diagonal movements are also possible.
• the operator O by means of the gestures, can move the grippers 7 of the anthropomorphic robot 2 also in the vertical plane, varying the distance with respect to the object to be picked up. If he/she wishes to move in the vertical plane, he/she has to move his/her hand up, down, right, left in the vertical plane. These movements are detected by the three-dimensional vision sensor 14 and interpreted by the software program for the gesture control implemented on the processing unit 12. Linear movement instructions with the previously selected linear speed are sent to the anthropomorphic robot 2.
• the method 100 also comprises at least the following steps:
• step 150 detecting the distance of the grippers 7 from the object 5 to be handled
• step 160 visually signaling to the operator O the distance of the grippers 7 from the object O to be handled (step 160).
• the step 160 of visually signaling the distance of the grippers 7 from the object O to be handled comprises displaying signals of different colors according to the distance.
• the graphic user interface means 13 display a signal of different color according to the distance: green in the event of the grippers 7 being out of the way, yellow in the event of these being at a short distance from the element to be picked up, red in the event of these being in a near collision position with the object O to be picked up (or in general with the elements inside a bin).
• the step 160 of visually signaling the distance of the grippers 7 from the object O to be handled comprises displaying the approximate distance between the grippers and the object themselves.
• step 130 and 140 of detecting the movements of the operator O and of moving the anthropomorphic robot 2 comprise, respectively, at least the following steps:
• step 132 - detecting, by means of the three-dimensional vision sensor 14, at least one rotation of a hand of the operator O around at least one axis of rotation (step 132);
• the method 100 according to the invention also allows interpolated and not only linear movements to be carried out with the anthropomorphic robot 2.
• angle 0 in the three directions of space the position of the right hand with more or less horizontal knuckles (consider the forearm in a horizontal and forward position). If the hand (and consequently the knuckles) is rotated in a clockwise direction, the anthropomorphic robot 2 will orient the wrist axis, and consequently the grippers 7, in a clockwise direction. If the hand (and consequently the knuckles) is rotated counterclockwise, the anthropomorphic robot 2 will orient the wrist axis, and consequently the grippers 7, counterclockwise. This is necessary to align the jaws of the grippers 7 with the object O in order to perform an optimal grip.
• the anthropomorphic robot 2 will be sent wrist reorientation movement instructions (clockwise and counterclockwise, respectively, for the wrist axis relative to the current angular position, incrementally) with the previously selected reorientation speed.
• the reorientation movement of the wrist axis is substantially a rotational movement around the axis Z of the tool robot, whose origin of the reference triad is positioned at the end of the grippers 7 (Z exiting the grippers and parallel to the vertical axis thereof).
• the reorientation movement of the anthropomorphic robot 2 in the selected direction continues until the operator carries out the opposite movement (if he/she has rotated his/her hand clockwise, he must bring it back to the zero or home position or carry out the opposite movement): the change of direction is interpreted as a stop.
• the operator O can also carry out more complex interpolated movements with the anthropomorphic robot 2, in order to manage the possible inclined grip of an object O.
• Such movements substantially represent a rotation around the axes X and Y of the tool robot, whose origin of the reference triad is positioned at the end of the grippers 7 (Z exiting the grippers and parallel to the vertical axis thereof).
• the anthropomorphic robot 2 will carry out the respective tool rotational movement around the axis X (identified by the left/right hand rotation) or axis Y (identified by the forward/backward hand rotation).
• the tool origin triad, positioned at the end of the grippers 7, will remain the fixed point of rotation.
• the method 100 comprises at least one step of managing two-axis interpolated rotation movements as a result of the detection of the forward-right, forward-left, backward-right, backward-left inclination of the operator hand knuckles.
• the anthropomorphic robot 2 is sent respective rotational movement instructions around the axes X and Y of the tool and with respect to the current angular position, incrementally, with the previously selected reorientation speed.
• the reorientation movement of the anthropomorphic robot 2 in the selected direction continues until the operator O carries out the opposite movement (if he/she has rotated his/her hand in one direction, he/she must bring it back to 0 position or carry out the opposite movement): the change of direction is interpreted as a stop.
• the aforementioned steps 130 and 140 of detecting the movements of the operator O and of moving the anthropomorphic robot 2 comprise, respectively, at least the following steps:
• step 133 detecting at least one closing command of the grippers or one opening command of the grippers
• the anthropomorphic robot 2 is brought to stop mode by opening the left hand in the air.
• the operator O can close in joint position the thumb and the index finger of the right hand: this operation is interpreted by the anthropomorphic robot 2 as a gripper closing command.
• the gripper closing can only occur with the robot in stop mode: gestures of closing and opening fingers during the movement phase will not be taken over by the management software.
• the operator O after having set the anthropomorphic robot 2 back in the movement condition with his/her left hand (closed fist without thumb up or down), can bring the anthropomorphic robot 2 back to the out of the way position above the bin with the object 5 picked up in his/her hand, using, for the ascent, the movements previously described for the right hand.
• the start picking position command closed fist and thumb upwards, like the OK gesture
• steps 130 and 140 of detecting the movements of the operator O and of moving the anthropomorphic robot 2 comprise, respectively, at least the following steps:
• step 144 moving the anthropomorphic robot 2 between a picking position and a storage position (step 144).
• the operator O moves the forearm completely clockwise (the robot makes a clockwise reorientation movement on the axis 1, at its base) or counterclockwise (the robot makes a counterclockwise reorientation movement on the axis 1, at its base).
• the movement of the anthropomorphic robot 2 in the selected direction continues until the operator O carries out the opposite movement (if he/she has rotated his/her forearm clockwise, he/she must rotate it again to the zero position, straight ahead, therefore with a counterclockwise movement): the change of direction is interpreted as a stop.
• the speed used for the movement of the axis 1 of the anthropomorphic robot 2 is the previously selected reorientation speed.
• the operator O can also use all the other movements of the linear type previously described for the correct positioning over the selected storage bin.
• the first camera 8 is dimmed by the presence of the object 5 being gripped.
• the operator O will therefore use as a reference the additional camera 9 located on the robot wrist (but externally to the grippers) or located high above the cell.
• the operator O judges that he/she is correctly positioned in the storage position, above the desired bin, he/she may stop the movement, bringing the anthropomorphic robot 2 to the stop mode.
• the operator O can open the fingers of his/her right hand, by stretching them completely: this operation is interpreted as a grippers opening command, thus depositing the object 5 in the storage bin.
• the opening of the grippers 7 can only occur with the anthropomorphic robot 2 in the stop mode: closing and opening gestures of the fingers during the movement phase will not be taken over by the management software.
• the robot By means of a run command and the various movements described above, the robot can be placed back in the out of the way position above the rough parts picking bin to start a new cycle.
• the start picking position command can be given with the left hand, which automatically returns the robot to the same position.

Landscapes

• Engineering & Computer Science (AREA)
• Robotics (AREA)
• Mechanical Engineering (AREA)
• Manipulator (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=74184764

Family Applications (1)

Country Status (3)

Family Cites Families (1)

• 2020
  • 2020-10-28 IT IT102020000025567A patent/IT202000025567A1/it unknown
• 2021
  • 2021-10-25 WO PCT/IB2021/059822 patent/WO2022090895A1/en unknown
  • 2021-10-25 EP EP21807253.6A patent/EP4237210A1/en not_active Withdrawn

Also Published As

Similar Documents

Legal Events