WO2021118388A1

WO2021118388A1 - Object handling system

Info

Publication number: WO2021118388A1
Application number: PCT/RU2019/000928
Authority: WO
Inventors: Андрей Владимирович НОВИКОВ; Владимир Николаевич ГЕРАСИМОВ; Роман Александрович ГОРБАЧЕВ; Никита Евгеньевич ШВИНДТ; Владимир Иванович НОВИКОВ; Андрей Евгеньевич ЕФРЕМЕНКО; Дмитрий Леонидович ШИШКОВ; Михаил Нилович ЗАРИПОВ; Филипп Александрович КОЗИН; Алексей Михайлович СТАРОСТЕНКО
Original assignee: Общество С Ограниченной Ответственностью "Нейроассистивные Технологии"
Priority date: 2019-12-10
Filing date: 2019-12-10
Publication date: 2021-06-17

Abstract

The invention relates to robotic systems for handling objects by means of a manipulator with a gripping device and can be used for assisting people with motor function disorders. An object handling system comprises a manipulator module, at least one gripping device module mounted on the latter, a control module, a user interface module and a fastening module. The manipulator module comprises at least one manipulator actuator comprising a manipulator motor and a manipulator reduction gear. The control module comprises a manipulator module control unit and a gripping device control unit. The user interface module comprises an eye-tracking unit, a computer vision unit and a neuro headset unit. By means of the user interface module, a user can open and close doors, switch on a light, pour a drink, hold a glass of water, etc. The handling system provides highly accurate determination of an object for handling among a multiplicity of surrounding objects, and also clear determination of the user's intentions regarding the handling of the objects selected for handling, and is compact and lightweight.

Description

OBJECT MANIPULATION SYSTEM

The invention relates to robotics, in particular, to robotic systems for manipulating objects by means of a manipulator with a gripping device, and can be used, inter alia, to help people with impaired motor functions caused by the consequences of brain and spinal cord injuries, stroke, neurodegenerative diseases and dr.

The declared object manipulation system performs the functions of gripping, holding and moving objects in the environment of a user with limited or impaired motor functions, often moving in a wheelchair. The system can have a modular architecture, personalized for a specific user, and can be used in a hospital or at home.

The use of existing automated components or the modification of existing manipulators is not able to meet all the necessary requirements, such as a light and compact design that will not compromise the balance of the user's wheelchair or increase its width. The manipulator must be capable of gripping objects of various shapes under the control of the user, while it must be protected from adverse factors such as dust and water.

RF patent RU2700246 discloses a method and a system for gripping an object using a robotic object, in which the robotic device is trained using a machine learning algorithm to recognize and memorize the gripping points of objects. Training is performed on data characterizing photographic images of objects and corresponding three-dimensional models of objects in various forms. This makes it possible to improve the accuracy of recognition of the area of object capture by the robotic device. Despite this, the recognition accuracy of the capture area of the object may be insufficient for the application of the known method and system by a user with limited or impaired motor functions. In addition, such a system requires rather long training, and the choice of an object of manipulation from several objects is not provided at all.

From US patent application US2017021500, systems and methods for determining displacement data of a robotic installation in a working environment are known. A robot operating in a work environment can be instructed to determine the work offset. Work offset can describe the location and angular orientation of the work plane of the work environment relative to the base plane of the robot. In response to the indication, the robot can identify the work plane. The robot can be controlled to contact one or more points on the work plane. The robot can determine the corresponding locations of the points of the contacting points relative to the reference plane based on the corresponding positions of the robot at the corresponding moments of contact. The robot can determine the location and angular orientation of the work plane relative to the reference plane based on the determined corresponding point locations of the contact points.

U.S. Patent Application US2019240843 describes a method for manipulating a deformable object, which includes determining the appropriate three-dimensional position of one or more markers on a deformable object held by a robotic arm, as well as determining a deformable model of the deformable object by displaying the movement of the robot arm and the movement of one or more markers. A robotic arm is controlled based on a defined deformation model to manipulate the deformable object and move one or more markers to the appropriate position. US patent application US2019143506 describes a system that is a robotic arm including a daisy chain containing a first link, a first link and a second link. The second link is between the first link and the first link in the daisy chain. The system further includes a processing unit including one or multiple processors. The processing unit is configured to receive data of the first communication line from the first system of sensors located in the first communication line, generate a first estimate of the connection state of the first connection based on the data of the first communication line and the kinematic model of the robotic arm, and control the first link based on the first estimate of the state connections.

The systems and methods of manipulation disclosed in the above US patent applications also do not provide for the ability to select a manipulated object from several objects, which is critical for a user with limited or impaired motor functions. The well-known 0 manipulators KINO V A® Gen2 Ultra lightweight robot, iArm and others contain a joystick for these purposes, the control of which is not always convenient or even possible, not to mention the accuracy of such control.

Thus, there is a problem of creating such a manipulator with a gripper that could grip, hold and move objects in the environment of a user with limited or impaired motor functions, while providing the user with precise, easy and simple control of the gripper of the manipulator.

The technical result of the invention is to provide high accuracy in determining the object of manipulation among a variety of surrounding objects with a compact and lightweight design and a clear definition of the user's intentions to manipulate the selected objects of manipulation.

The problem is solved, and the claimed technical result is achieved in the proposed system for manipulating objects (hereinafter also the manipulation system) for performing operations on objects, in particular, for capturing, 5 holding and moving objects. The object manipulation system comprises a manipulator module, at least one gripper module, a control module, a user interface module, and a fastening module. The manipulator module contains at least one manipulator drive, which includes a manipulator motor and a manipulator gear. Gripper module is installed on the manipulator module and contains at least one gripper drive including the gripper motor. The control module contains a control unit for the manipulator module for controlling the manipulator drive and executing the control algorithm for the manipulator module, 5 as well as a control unit for the gripping device module for controlling the gripping device drive and executing the control algorithm for the gripping device module. The user interface module is designed to control the manipulation system, select an object and perform an operation on the object and contains an eye tracking unit for determining the direction of the user's gaze, a computer vision unit 0 for performing a stereo reconstruction of the area around the object (that is, an area in the service area of the manipulator where required manipulation object), constructing a disparity map and determining the characteristics of the object, including at least the geometric dimensions of the object and the distance to the object, as well as a neuro-headset unit for recording bioelectric potentials of the user's brain activity and transferring them to the control module. The attachment module is designed to attach the manipulator module to an external device, such as a wheelchair.

The declared object manipulation system allows you to determine the objects that the user wants to control, and actually control them, where O under control is understood, in particular, to capture an object, hold an object and move objects.

The problem is solved, and the claimed technical result is also achieved in particular embodiments of the claimed manipulation system, which, however, the present invention is not limited to.

5 For example, an absolute angle sensor can be installed on the manipulator motor shaft.

To control the manipulator motor, the control module may additionally contain PID controllers for current, speed, position and torque. An absolute angle sensor can be installed on the output shaft of the manipulator gearbox.

It is preferable if the eye tracking unit is configured to determine the coordinates of the gaze direction vector. In this case, the eye tracking unit can be configured to determine the coordinates of the intersection of the gaze direction vector with the plane in which the object is located.

It is also preferable if the vision unit is configured to perform stereo reconstruction of the area around the object with a given frequency. In another preferred embodiment of the invention, the technical vision unit contains a stereo pair, including the stereo pair cameras, a stereo image processing unit and a unit for matching images of the area around the object obtained from different angles from the stereo pair cameras and from the eye tracking unit camera. The technical vision unit is preferably located above the head of the disabled operator, and the eye tracking unit is located on the head of the operator wearing glasses, and thus two images are obtained from different angles.

In addition, the headset unit can contain dry active electrodes for recording bioelectric potentials of the user's brain activity. Next, the claimed object manipulation system and its operation are explained in more detail with reference to the figure, which schematically shows the manipulation system.

In the figure, reference numerals indicate:

1 - manipulator module (MM); 2 - gripper module (MZU);

3 - control module (MU);

3.1 - manipulator module control unit (BUMM);

3.2 - gripper module control unit (BUMZU);

4 - user interface module (MIP); 4.1 - eye tracking unit (BTG);

4.2 - unit of technical vision (BTZ);

4.3 - neuro-headset unit (BN);

5 - mounting module (MK).

5 As mentioned above, the object manipulation system is designed to perform operations on objects, in particular, to grip, hold and move objects. The manipulation system should allow the specified operations to be performed by a user with limited mobility, in particular, with limited mobility of the upper limbs, and at the same time guarantee the exact execution of exactly those movements that the user wishes to perform.

In general, the manipulation system contains a manipulator module 1 (also MM), at least one gripper module 2 (also MZU), a control module 3 (also MU), a user interface module 4 (also MIP) and a fastening module 5 (also MK ).

5 MM 1 is designed to move in the space of MM 2 and can be installed on any object, including a wheelchair, by means of MK 5 installed at one end of MM 1 or near it. As MM 1, any known design of a robotic arm can be selected, for example, described in RU2700246 or US2007095582. Nevertheless, it is preferable that O the number of links MM 1 is from four to six: a smaller number of links may not provide accurate and fast positioning of the MM 2 in a space with many objects, in a larger number of links not only complicates the design of the entire handling system, but also increases its weight, which is critical, in particular, when using it on a wheelchair. In US2007095582 also: 5 shows a possible option for attaching MM 1 to a wheelchair by means of MK 5.

MM 1 contains at least one manipulator drive, which includes a manipulator motor and a manipulator gearbox (not shown in the figure). It is preferable, but not necessary, if each link MM 1 contains its own manipulator drive, which makes it possible to more accurately position the MM 2 in space.

An angle sensor can be additionally installed on the shaft of the manipulator motor of one or more manipulator drives. Even when using precision motors of the manipulator, errors in the positioning of the MM 1 sections are possible, especially critical for the section farthest from the MK 5, on which the MZU 2 is installed, in a confined space and / or a large number of objects. The presence of a rotation angle sensor on the manipulator motor shaft allows precise control of the shaft rotation angle. In this case, the most preferable is the use of an absolute encoder of the angle of rotation, which avoids the accumulation of errors in determining the angle of rotation of the shaft.

The same applies to the output shaft of the manipulator gearbox of one or more manipulator drives: the higher the accuracy of determining the rotation of this shaft, the more accurately the MZU 2 is positioned in space. Therefore, in this case, too, it is preferable to install a rotation angle sensor on the output shaft of the manipulator gearbox of one or more manipulator drives, most preferably an absolute rotation angle sensor, which avoids accumulation of errors in determining the shaft rotation angle.

MZU 2 is installed at the other end of MM 1 from MK 5 or near it and is designed to grip and hold objects to be manipulated. Any known device for a similar purpose, for example, described in RU118579 or US8936289, can be selected as MZU 2. In this case, it is preferable if the design of the MCD 2 is made in such a way that one drive of the gripper, including the motor of the gripper, is sufficient to control the fingers of the MCD 2. This provides MZU 2 compactness and low weight, which means accurate and fast positioning of MZU 2 in a space with many objects.

MU 3 is an electronic device and is designed to execute the control algorithm MM 1 and, accordingly, control MM 1, as well as to execute the control algorithm MZU 2 and, accordingly, control MZU 2. For for these purposes, MU 3 contains, respectively, a control unit for the manipulator module 3.1 (also BUMM) and a control unit for the gripper module 3.2 (also BUMZU).

In a private embodiment, MU 3 can be an electronic device based on a 32-bit microcontroller, for example, STM32F103C8T6. On the MU 3 board, voltage stabilizers, a CAN bus driver, drivers for controlling the manipulator motors (in BUMM 3.1), drivers for controlling the gripper motors (in BUMZU 3.2), ports for connecting absolute position and speed sensors, if any of they are used in MM 1 and / or MZU 2, an expansion port for connecting external modules, interfaces to encoders and strain gauges.

BUMM 3.1 and BUMZU 3.2 can share the specified microcontroller to execute the MM 1 control algorithm and the MCU 2 control algorithm.

In a private version, MU 3 additionally contains proportional-integral-differentiating controllers (PID controllers) of current, speed, position and torque to control the manipulator motors and the gripper motors.

With the help of MIP 4, the user controls the object manipulation system, selects the object and performs the desired operations on the object. MIP 4 contains an eye tracking unit 4.1 (also BTG), a technical vision unit 4.2 (also BTZ), and a neuro headset unit 4.3 (also WB).

BTG 4.1 is designed to determine the direction of the user's gaze, which is then used to accomplish the task of user interaction with the manipulation system. In particular, BTG 4.1, by means of the camera (or cameras) included in it, can provide the determination of the coordinates of the gaze direction vector and, additionally, the coordinates of the intersection of the gaze direction vector with the plane in which the manipulation object is located.

Any known device can be used as BTG 4.1, for example, Tobii Pro or SMI Eye Tracking Glasses. BTZ 4.2 carries out a stereo reconstruction of the area around the object, that is, the area in the movement zone of the MZU 2, in which the manipulation object can be located. Stereo reconstruction methods are widely presented in the prior art and do not require further explanation. In this case, stereo reconstruction can be carried out at a given frequency. BTZ 4.2 also builds a disparity map, that is, a map of the relative position of points displayed on the retinas of the left and right eyes. In addition, another function of BTZ 4.2 is to determine the characteristics of the manipulated object, such as the geometric dimensions of the object, the distance to the object.

0 To perform its functions, BTZ 4.2 may contain a stereopair including stereo pair cameras, a stereo image processing unit and a unit for comparing images of the area around the object obtained from different angles from the stereo pair cameras and from the BTG 4.1 camera. Different angles are achieved in particular by the fact that BTZ 4.2 is placed approximately above the user's head, and BTG 4.1 - in glasses, thus 5 providing two images of the area around the object from different angles.

BN 4.3 is designed to record bioelectric potentials of the user's brain activity and transfer them to MU 3. For this, BN 4.3 can contain dry active electrodes.

О The declared object manipulation system works as follows.

The user fixes attention on the object with which he wants to perform manipulations.

BTG 4.1 determines the direction of the user's gaze.

BTZ 4.2, through its cameras, for example, stereo pair 5 cameras, captures the surrounding environment, based on data from BTG 4.1, calculates the fixation point of the user's gaze on the resulting image, recognizes the object at this point, determines the distance from the user's head to the object and the distance from MZU 2 to the object. Next, the user chooses the action he wants to perform with the recognized object, for example, move the object from one point in space to another, press a key or switch, etc. Recognition of the user's intention in the claimed manipulation system can be implemented in 5 different ways. The first option for recognizing the user's intention is carried out by means of BN 4.3, in which signals from the surface electroencephalogram are recorded. The second option for recognizing the user's intention is carried out by means of BTG 4.1, in which the user selects one of the proposed actions by means of fixing his gaze on one of 0 icons displayed on the optically transparent display of augmented reality BTG 4.1.

For example, let's assume that the user has selected an action to move an object from one point in space to another. Then, after pointing to the manipulation object and choosing the intention to move the manipulation object to a new 5 point, the user fixes his attention on this new point in space where he wants to move the manipulation object. BTG 4.1 and BTZ 4.2 jointly define a new point.

Further, MU 3, by means of BUMM 3.1 and BUMZU 3.2, gives the corresponding commands to MM 1 to bring MZU 2 to the point of location of the manipulation object, about capturing and holding this object MZU 2, moving the manipulation object by means of MM 1 to a new point and release of this object MZU 2.

In everyday life, the user can use the object manipulation system for different needs. So the manipulation system can open and close doors, turn on the light, pour drinks, hold a glass of water 5, etc. This is not the end of its functions: it can also remove toys after children, feed a dog, support a person during trips, etc.

A distinctive feature of the declared manipulation system is the high accuracy of determining the object of manipulation among many others. objects, compact and lightweight design and a clear definition of the user's intentions to manipulate the selected manipulation objects.

Claims

CLAIM

1. A system for manipulating objects for performing operations on objects, in particular, for gripping, holding and moving objects, containing:

5 - a manipulator module containing at least one manipulator drive, including a manipulator motor and a manipulator gear,

- at least one gripping device module mounted on the manipulator module and containing at least one gripping device drive including a gripping device motor,

.0 - control module, including: a manipulator module control unit, configured to control the manipulator drive and execute the manipulator module control algorithm, and a gripping device module control unit, made with .5 the ability to control the gripping device drive and execute the gripping device module control algorithm,

- a user interface module for controlling the manipulation system, selecting an object and performing an operation on an object, including: an eye tracking unit made with the ability to determine the direction! 0 of the user's gaze, a technical vision unit made with the possibility of performing a stereo reconstruction of the area around the object, constructing a disparity map, and determining the characteristics of the object, including at least the geometric dimensions of the object and the distance to the object, and> 5 a neuro-headset unit configured to register bioelectric potentials of the user's brain activity and transmit them to the control module, and

- mounting module for attaching the manipulator module.

2. A manipulation system according to claim 1, in which an absolute rotational angle sensor is installed on the manipulator motor shaft.

3. A manipulation system according to claim 1 or 2, in which the control module

5 additionally contains PID controllers for current, speed, position and torque to control the manipulator motor.

4. The manipulation system according to claim 1, in which an absolute sensor of the angle of rotation is installed on the output shaft of the manipulator reducer.

0

5. A manipulation system according to claim 1, wherein the eye tracking unit is configured to determine the coordinates of the gaze direction vector.

6. A manipulation system according to claim 5, in which the eye tracking unit is configured to determine the coordinates of the intersection of the gaze direction vector with the plane in which the object is located.

7. A manipulation system according to claim 1, wherein the vision unit is configured to perform stereo reconstruction with a predetermined frequency.

: 0

8. A manipulation system according to claim 1, 5 or 6, in which the vision unit contains a stereopair including stereopair cameras, a stereoimage processing unit and a unit for comparing images of the area around the object obtained from different angles from the stereopair cameras and from the eye tracking unit camera.

!five

8. The manipulation system according to claim 1, in which the neuro-headset unit contains dry active electrodes for recording bioelectric potentials of the user's brain activity.