WO2021133186A1 - Procédé de commande de manipulateur robotisé - Google Patents

Procédé de commande de manipulateur robotisé Download PDF

Info

Publication number
WO2021133186A1
WO2021133186A1 PCT/RU2019/001001 RU2019001001W WO2021133186A1 WO 2021133186 A1 WO2021133186 A1 WO 2021133186A1 RU 2019001001 W RU2019001001 W RU 2019001001W WO 2021133186 A1 WO2021133186 A1 WO 2021133186A1
Authority
WO
WIPO (PCT)
Prior art keywords
laser emitter
scene
coordinate system
target point
point
Prior art date
Application number
PCT/RU2019/001001
Other languages
English (en)
Russian (ru)
Inventor
Андрей Владимирович НОВИКОВ
Владимир Николаевич ГЕРАСИМОВ
Роман Александрович ГОРБАЧЕВ
Никита Евгеньевич ШВИНДТ
Владимир Иванович НОВИКОВ
Андрей Евгеньевич ЕФРЕМЕНКО
Дмитрий Леонидович ШИШКОВ
Михаил Нилович ЗАРИПОВ
Филипп Александрович КОЗИН
Алексей Михайлович СТАРОСТЕНКО
Original Assignee
федеральное государственное автономное образовательное учреждение высшего образования "Московский физико-технический институт (национальный исследовательский университет)"
Общество С Ограниченной Ответственностью "Нейроассистивные Технологии"
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by федеральное государственное автономное образовательное учреждение высшего образования "Московский физико-технический институт (национальный исследовательский университет)", Общество С Ограниченной Ответственностью "Нейроассистивные Технологии" filed Critical федеральное государственное автономное образовательное учреждение высшего образования "Московский физико-технический институт (национальный исследовательский университет)"
Priority to PCT/RU2019/001001 priority Critical patent/WO2021133186A1/fr
Publication of WO2021133186A1 publication Critical patent/WO2021133186A1/fr

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals

Definitions

  • the invention relates to robotics, in particular, to robotic systems for easy control of pointing devices such as a laser pointer.
  • one of the urgent tasks is direct interaction with environmental objects and other people. That said, it is sometimes important to be able to point to the object of interest directly, rather than through the application interface displayed on the screen. This can be done, for example, with a laser pointer.
  • US 8442661 known telepresence system, which includes a mobile robot equipped with a laser and a user interface device.
  • the laser can be targeted by the robot operator through a user interface device.
  • the user interface device contains a head tracking module and an input module including, for example, a joystick, mouse, keyboard, trackball, touchpad.
  • the disadvantage of this system is the inability to use it by users with limited hand mobility.
  • the scene recorded by the video camera of the mobile platform is streamed to the patient's device.
  • the patient controls the mobile platform by moving or keeping the gaze on the patient's device.
  • the disadvantages of this solution include the following: the patient is forced to point only to objects in the field of view of the video camera of the mobile platform; the patient is forced to control the mobile platform through the patient's device, that is, the patient himself perceives the environment not directly, but through the screen; the mode of providing streaming video requires a sufficiently reliable communication channel.
  • the technical result of the invention is to improve the control accuracy of the light pointer, in particular the laser pointer, when using only the user's gaze.
  • the problem is solved, and the claimed technical result is achieved in the proposed method for controlling a robotic manipulator containing a laser emitter, a stereo scene camera for obtaining images of the scene and a stereo eye camera for obtaining images of the left eye and right eye.
  • the method includes selecting a projection area of a beam of a laser emitter by determining a direction of sight; obtaining an image of the scene around the projection area; determination of the possibility of beam projection onto the projection area; selection of the optimal point for the position of the laser emitter; determination of the current position of the laser emitter in the coordinate system of the scene stereo camera; approach of the laser emitter to the optimal point; and projection of the laser emitter beam onto the projection area.
  • the problem is solved, and the claimed technical result is also achieved in particular embodiments of the method for controlling a robotic arm, which, however, the present invention is not limited to.
  • the first target point, the second target point, the third target point and the fourth target point it is preferable if: select the first target point, the second target point, the third target point and the fourth target point, not lying on the same plane; sequentially direct the beam to the specified target points and determine the coordinates of the specified target points in the coordinate system of the scene stereo camera; determining the distance from the first target point to the laser emitter, the distance from the second target point to the laser emitter and the distance from the third target point to the laser emitter in the coordinate system of the scene stereo camera, and the position of the fourth target point is used to unambiguously determine the indicated distances; build an orthonormal coordinate system Oxyz with unit unit vectors i, j, k, while: select the first point by the origin, select the unit unit vector passing from the first target point to the second target point, with the direction of the unit unit unit i in the direction Ox, perform Gram orthogonalization -Schmidt of the vector passing from the first target point to the third target point, relative to the
  • FIG. 1 schematically shows a block diagram of an algorithm for implementing the projection of a laser emitter beam onto a user-selected projection area.
  • the figure uses the following abbreviations: LI - laser emitter; SC - coordinate system.
  • any known robotic system can be used, which is a manipulator with a laser emitter installed on it, as well as including a stereo camera of the scene and a stereo camera of the eyes.
  • any known directional light source can be used, for example a semiconductor LED with a focusing lens system, therefore the term "laser emitter” should not be understood as limiting this invention to the use of only a laser emitter therein.
  • a stereo scene camera is designed to capture an image of the surrounding area, also called a scene.
  • the stereo eye camera is used to acquire images of the left eye and right eye through its constituent left eye cameras and right eye cameras.
  • an image of the left eye and an image of the right eye are obtained containing glare from the light sources of the left eye and the right eye, which may be located near the stereo camera of the eyes.
  • the position of the pupil of the eye is determined, the position and numbering of the highlights on the cornea of the eye are determined, the optical axis of the direction of the gaze of each eye is determined, and the direction of gaze is already determined from it.
  • a preliminary search for the pupil is performed, a preliminary ellipse of the pupil is constructed, and an ellipse of the pupil is constructed from its nodal points.
  • searching for flares searching for flares, calculating the size of the iris, eliminating flares outside the iris, and numbering the flares to determine the gaze direction vector.
  • the optical axis of the gaze direction of each eye the nodal point of the eye, the refractive point for the center of the pupil, and the position of the center of the pupil in the coordinate system of the scene camera are determined. The determination of the gaze direction is performed based on the determined optical gaze direction and the gaze direction calibration.
  • a preliminary position of the center of the pupil can be determined, as well as the number of pixels in the area of the pupil, which preliminarily characterizes its size.
  • the binarization threshold is found in the preliminary pupil area and binarization is performed to determine the pupil boundary to construct the preliminary pupil ellipse.
  • the search for highlights on the cornea of the eye can be performed, for example, by thresholding the image of the pelvis with the selection of clusters and filtering the clusters by brightness, size and the parameter of deviation from roundness.
  • the size of the iris it is preferable to use information about the average size of the human iris and information about the distance from the corresponding camera of the left eye or the camera of the right eye to the pupil, which improves the accuracy of determining the direction of gaze.
  • Flare numbering can be performed from one flare from the upper pair closest to the bridge of the nose, in a circle, away from the bridge of the nose, i.e. clockwise for the right eye and counterclockwise for the left eye.
  • the step of determining the gaze direction may include calibration of the gaze direction, which is performed either at one of the indicated steps or in advance. In this case, it is enough to calibrate once for a specific user, and re-calibration is no longer required.
  • calibrating the direction of gaze the individual characteristics of the user and the relative position of the cameras of the left and right eyes and the scene camera are taken into account.
  • the step of obtaining information about the scene around the projection area by means of the scene stereo camera is carried out.
  • a method for obtaining a set of objects of a three-dimensional scene in which images of frames from the left camera and the right camera are simultaneously obtained as part of the scene stereo camera, for each image point with pixel coordinates a disparity map is formed by the method of semi-global establishment of stereo correspondences , according to the disparity map, the true coordinates of the specified point are determined (i.e., the coordinates of the point in the coordinate system of the scene stereo camera), the map of the depths of the points in true coordinates is formed, a two-dimensional image in the gray scale is formed, in which the brightness of the point depends on the true distance to the point, and on the obtained two-dimensional image in gray scale, detection and identification of objects is performed by one of the methods selected from the Viola-Jones method (R.
  • the disparity map is formed by the method of semi-global establishment of stereo correspondences (Semi-Global Matching, or SGM; the method is described, for example, in Heiko Hirschmuller. Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information. IEEE Conference on Computer Vision and Pattern Recognition ( CVPR), San Diego, CA, USA, June 20-26, 2005).
  • SGM Stereo-Global Matching
  • the true coordinates of a point are determined taking into account focal lengths of cameras in the scene stereo camera and the distance between them.
  • the brightness of a point is assumed to be zero if the true distance to it is outside the specified range.
  • the disparity map is a visual display of shifts between equally spaced fragments of images of the left and right cameras of the scene stereo camera (the closer the scene point is, the greater these shifts).
  • this "discrepancy" can be represented as a numeric array, the elements of which show the difference in pixels of the points of the right and left images, tied to one of them. Rectification of images from different angles (alignment of the right and left images horizontally) allows you to reduce the dimension of the array - to reduce it to two-dimensional. For ease of perception, this matrix is presented in a graphical form: the greater the discrepancy between the images, the brighter the corresponding pixels in the image.
  • Local algorithms calculate disparity separately for each pixel, while taking into account information only from its narrow neighborhood.
  • the algorithms mainly use square or rectangular windows of a fixed size and, according to some metric, compare the sums of the absolute values of the brightness within these windows.
  • Such algorithms are characterized by high speed and computational efficiency. However, acceptable performance is only ensured if the pixel intensity function is smooth. At the boundaries of objects, where the intensity function breaks, the algorithms make a significant number of errors. Further development of the methods led to the emergence of multi-window algorithms and windows with an adaptive structure, which improved the quality of disparity calculation. But the "payment" for this was a significant increase in operating time, which often leads to the impossibility of analyzing images in real time.
  • Global algorithms are based on calculating the disparity simultaneously for the entire image, while each pixel of the image affects the solution in all other pixels. Global algorithms differ both in the form of unary and pair potentials, and in the minimization algorithms and the structure of the graph. Despite the fact that, as a rule, in terms of efficiency, global algorithms are superior to local ones, the resulting disparity maps are not free from errors caused by the simplifications that were originally incorporated into the formula for the energy functional. Moreover, global algorithms are slower.
  • Semi-global, or partially global, methods are a reasonable compromise between fast but imprecise local methods and more accurate but slow global methods, allowing rational use of their strengths.
  • the idea of the methods consists in the independence of the solution for each pixel, taking into account the influence of all (or a part, not limited by the local neighborhood) of the remaining pixels of the image.
  • a disparity map is formed by the method of semi-global establishment of stereo correspondences with obtaining disparity d (x, y) for each image point with pixel coordinates (x, y).
  • a depth map D (x, y) is formed, where D is the true distance from the left camera or right camera to the point with pixel coordinates (x, y).
  • a two-dimensional gray-scale image is formed, on which the brightness ⁇ (x, y) of a point with pixel coordinates (x, y) is set by the formulas:
  • ⁇ (x, y) 255, if D (x, y)> Dmax,
  • ⁇ (x, y) 255 (D (x, y) - Dmin) / (Dmax - Dmin) - in other cases, where Dmin and Dmax are the specified minimum and maximum depth values, respectively, determined from the context of the application of the claimed method.
  • objects are detected and identified by one of the methods selected from the Viola-Jones method, the SSD-mobilenet neural network method and the Mask R-CNN neural network method, to obtain a set of objects in a three-dimensional scene.
  • the position of the manipulator and / or the direction of the laser emitter must be adjusted so that the beam can easily reach the projection area, taking into account the resulting scene, including obstacle objects and their sizes.
  • there may be several suitable positions for trouble-free projection of the beam onto the projection area and from such points of the position of the laser emitter the most optimal point is selected, for example, the one closest to the current position, or one that is distant from the boundaries of the area of possible movement of the manipulator, or according to some other criterion ...
  • the first target point, the second target point, the third target point and the fourth target point which do not lie on the same plane, are arbitrarily selected in space, as well as with the condition that they must be visible to the scene stereo camera and available to the laser beam.
  • the numbering of target points is indicated solely for the convenience of understanding the further description.
  • the laser emitter beam is sequentially directed at the indicated target points and the coordinates of the indicated target points are determined in the coordinate system of the scene stereo camera.
  • the distance from the first target point to the laser emitter, the distance from the second target point to the laser emitter and the distance from the third target point to the laser emitter are determined in the coordinate system of the scene stereo camera, and the position of the fourth target point is used to unambiguously determine the indicated distances , which will be discussed in more detail below.
  • Ar is the second target point
  • Az is the third target point
  • the coordinates of the points Ai, Ar, Az, Ad are known in the coordinate system of the scene stereo camera, and the coordinates of the point O are known in the coordinate system of the manipulator.
  • Points O, Ai, Ar, Az should not lie in the same plane, which is almost always done automatically.
  • an orthonormal coordinate system Oxyz is constructed with unit unit vectors i, j, k.
  • the origin of coordinates for example, the first point Ab, is chosen.
  • the unit vectors of the vector passing from Ai to the second point Ai are selected by the direction of the unit unit unit r in the direction Ox.
  • the Gram-Schmidt orthogonalization of the vector passing from Ai to the third point Az is performed relative to the unit reference unit / to obtain a unit unit reference unit j in the direction ⁇ réelle.
  • the vector product of unit unit vectors and j is performed to obtain unit unit unit unit vectors k in the direction Oz.
  • the vector of the position of the laser emitter is decomposed in the Oxyz coordinate system in terms of the coordinate system of the scene stereo camera.
  • the constructed system Oxyz allows for any point or vector to recalculate its coordinates from the coordinate system of the scene stereo camera to the coordinate system of the manipulator (or laser emitter), or in the opposite direction. Indeed, since the construction is performed simultaneously in the manipulator coordinate system and the scene stereo camera coordinate system, for any vector in the manipulator coordinate system and the scene stereo camera coordinate system, it is possible to determine its coordinates in the Oxyz coordinate system by scalar multiplying this vector and the unit vectors i, j, k.
  • the claimed method of controlling a robotic manipulator was tested under the following conditions: linear displacement between the attachment point (origin of the coordinate system) of the laser emitter and the attachment point (origin of the coordinate system) of the stage stereo camera from 1 to 5 meters; angular displacement within 1.5 radians along each axis; for the scene stereo camera, the modes of the left and right cameras are 1280x720, the distance between the left and right cameras is 0.1 m; distances between target points from 1 to 3 meters. According to the test results, the positioning error did not exceed 1 cm.
  • the method is quick and easy to use, for distributed robotics with difficultly predictable movement and rotation of the scene stereo camera relative to the laser emitter, it allows you to quickly determine or constantly monitor the relationship between the coordinate systems of the scene stereo camera and the laser emitter.

Abstract

L'invention concerne des systèmes robotisés pour la manipulation d'objets à l'aide d'un manipulateur comprenant un dispositif de saisie. L'invention concerne un procédé de commande de manipulateur robotisé qui consiste à: choisir la zone de projection d'un rayon d'un émetteur laser en déterminant la direction du regard; obtenir une image de la scène entourant la zone de projection; déterminer la possibilité de projection du rayon vers la zone de projection; choisir le point optimal de la position de l'émetteur laser; déterminer la position courante de l'émetteur laser dans un repère de caméra stéréo de la scène; amener l'émetteur laser au point optimal; et effectuer une projection du rayon de l'émetteur laser vers la zone de projection. Ce procédé peut être mis en oeuvre en utilisant un système robotisé comprenant un manipulateur sur lequel est installé un émetteur laser, une caméra stéréo de scène, et une caméra stéréo pour les yeux. Le résultat technique de la présente invention consiste en une augmentation de la précision de commande du pointeur lumineux, notamment un pointeur laser, en utilisant uniquement le regard de l'utilisateur.
PCT/RU2019/001001 2019-12-23 2019-12-23 Procédé de commande de manipulateur robotisé WO2021133186A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/RU2019/001001 WO2021133186A1 (fr) 2019-12-23 2019-12-23 Procédé de commande de manipulateur robotisé

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/RU2019/001001 WO2021133186A1 (fr) 2019-12-23 2019-12-23 Procédé de commande de manipulateur robotisé

Publications (1)

Publication Number Publication Date
WO2021133186A1 true WO2021133186A1 (fr) 2021-07-01

Family

ID=76573119

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/RU2019/001001 WO2021133186A1 (fr) 2019-12-23 2019-12-23 Procédé de commande de manipulateur robotisé

Country Status (1)

Country Link
WO (1) WO2021133186A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060100642A1 (en) * 2002-09-25 2006-05-11 Guang-Zhong Yang Control of robotic manipulation
US20100137880A1 (en) * 2007-06-19 2010-06-03 Medtech S.A. Multi-application robotized platform for neurosurgery and resetting method
US8731720B1 (en) * 2008-10-24 2014-05-20 Anybots 2.0, Inc. Remotely controlled self-balancing robot including kinematic image stabilization
US20180160035A1 (en) * 2009-06-17 2018-06-07 Lc Technologies, Inc. Robot System for Controlling a Robot in a Tele-Operation

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060100642A1 (en) * 2002-09-25 2006-05-11 Guang-Zhong Yang Control of robotic manipulation
US20100137880A1 (en) * 2007-06-19 2010-06-03 Medtech S.A. Multi-application robotized platform for neurosurgery and resetting method
US8731720B1 (en) * 2008-10-24 2014-05-20 Anybots 2.0, Inc. Remotely controlled self-balancing robot including kinematic image stabilization
US20180160035A1 (en) * 2009-06-17 2018-06-07 Lc Technologies, Inc. Robot System for Controlling a Robot in a Tele-Operation

Similar Documents

Publication Publication Date Title
US20240000295A1 (en) Light field capture and rendering for head-mounted displays
US11867978B2 (en) Method and device for determining parameters for spectacle fitting
US6373961B1 (en) Eye controllable screen pointer
US11922711B2 (en) Object tracking assisted with hand or eye tracking
WO2018030515A1 (fr) Dispositif de détection de ligne de visée
JP3064928B2 (ja) 被写体抽出方式
JP4825971B2 (ja) 距離算出装置、距離算出方法、構造解析装置及び構造解析方法。
US11348262B1 (en) Three-dimensional imaging with spatial and temporal coding for depth camera assembly
RU2001101469A (ru) Получение трехмерных планов одной ручной съемочной камерой
JP6631951B2 (ja) 視線検出装置及び視線検出方法
US10782780B2 (en) Remote perception of depth and shape of objects and surfaces
Kondori et al. Head operated electric wheelchair
US20220100268A1 (en) Eye tracking device and a method thereof
CN110895433A (zh) 用于增强现实中用户交互的方法和装置
JPH1066678A (ja) 非接触視線測定装置
CN109587304B (zh) 电子设备和移动平台
WO2021133186A1 (fr) Procédé de commande de manipulateur robotisé
WO2019220464A1 (fr) Système et procédé d'alignement entre des objets réels et virtuels dans un affichage transparent optique frontal
US20200292307A1 (en) Method and apparatus for determining 3d coordinates of at least one predetermined point of an object
KR20180116044A (ko) 증강현실용 디바이스와 그 증강현실 출력 방법
Schönbein omnidirectional Stereo Vision for autonomous Vehicles
WO2020044809A1 (fr) Dispositif de traitement d'informations, procédé de traitement d'informations et programme
Vámossy et al. PAL Based Localization Using Pyramidal Lucas-Kanade Feature Tracker
Kacperski et al. Calibration of vision systems operating in separate coordinate systems
US20230132986A1 (en) Information processing apparatus, information processing method, and program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19957127

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19957127

Country of ref document: EP

Kind code of ref document: A1