CN104739519B - Force feedback surgical robot control system based on augmented reality - Google Patents
Force feedback surgical robot control system based on augmented reality Download PDFInfo
- Publication number
- CN104739519B CN104739519B CN201510184133.3A CN201510184133A CN104739519B CN 104739519 B CN104739519 B CN 104739519B CN 201510184133 A CN201510184133 A CN 201510184133A CN 104739519 B CN104739519 B CN 104739519B
- Authority
- CN
- China
- Prior art keywords
- force feedback
- augmented reality
- unit
- information
- virtual environment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 44
- 238000006073 displacement reaction Methods 0.000 claims abstract description 6
- 230000001575 pathological effect Effects 0.000 claims description 15
- 230000037361 pathway Effects 0.000 claims description 9
- 238000001356 surgical procedure Methods 0.000 claims description 8
- 230000003993 interaction Effects 0.000 claims description 4
- 210000000056 organ Anatomy 0.000 claims description 3
- 230000008520 organization Effects 0.000 claims description 3
- 230000015572 biosynthetic process Effects 0.000 claims description 2
- 238000005728 strengthening Methods 0.000 claims description 2
- 230000002452 interceptive effect Effects 0.000 abstract description 2
- 238000000034 method Methods 0.000 description 7
- 238000002432 robotic surgery Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 3
- 210000004247 hand Anatomy 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 206010044565 Tremor Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 231100000915 pathological change Toxicity 0.000 description 1
- 230000036285 pathological change Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B17/00—Surgical instruments, devices or methods, e.g. tourniquets
- A61B17/00234—Surgical instruments, devices or methods, e.g. tourniquets for minimally invasive surgery
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Surgery (AREA)
- Heart & Thoracic Surgery (AREA)
- Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Manipulator (AREA)
Abstract
The invention relates to a force feedback surgical robot control system based on augmented reality, and belongs to the technical field of man-machine interactive control. The system comprises an operating terminal, a 3D (three-dimensional) display, a movement controller, a feedback unit, an augmented reality processing unit, a virtual environment unit, an executing terminal and a 3D endoscope, wherein the operating terminal transmits coordinate information to the movement controller, the movement controller transmits a movement control command to the executing terminal, the executing terminal transmits displacement information to the virtual environment unit, the virtual environment unit transmits image information to the augmented reality processing unit and transmits haptical information to the force feedback unit, the force feedback unit transmits haptical feedback to the operating terminal, the 3D endoscope transmits an image video of a surgical site to the augmented reality processing unit, and the augmented reality processing unit transmits an augmented reality video to the 3D display. According to the force feedback surgical robot control system based on augmented reality, operating precision, safety and reliability of a surgical robot can be effectively improved.
Description
Technical field
The invention belongs to interaction control technology field, it is related to a kind of force feedback operating robot based on augmented reality
Control system.
Background technology
Micro-wound operation robot is made up of with machine actuating station doctor operating side: doctor passes through endoscopic visualisation operative region
Image information, and by operating side send operation technique instruction, actuating station undergo surgery operation execution and held using being arranged on
The operating theater instruments at row end are operated.Remote operating micro-wound operation robot passes through precision optical machinery control system and 3 D stereo becomes
As technology significantly improves stability and the precision of surgical operation.When doctor completes robotic surgery, only rely on endoscope and see
Examine organ-tissue, contrast ct, mri 2-D data judge lesions position it is impossible to intuitively obtain pathological tissues three-dimensional localization and
Accurately sense of touch positioning, thus increase potential risk.Meanwhile, compared with traditional operation, operating robot can eliminate doctor behaviour
Trembling in work, but the disappearance of tactile force feedback also causes huge image to doctor simultaneously, including the force feedback of operating theater instruments,
The soft durometer of tissue, and the operation of the especially dependence touch feedback such as special cutting and suture.For problem above, based on image
Augmented reality and force feedback bootstrap technique provide a kind of reliable auxiliary positioning and operation differentiation mechanism, can effectively improve
The performance accuracy of robotic surgery and safety.
Content of the invention
In view of this, it is an object of the invention to provide a kind of force feedback operating robot based on augmented reality controls system
System, completes being accurately positioned of tissue by augmented reality with force feedback guiding, allows doctor obtain essence when implementing robotic surgery
Really vision and haptic force auxiliary information, improve robotic surgery precision and reliability.
For reaching above-mentioned purpose, the present invention following technical scheme of offer:
A kind of force feedback surgery operation robot control system based on augmented reality, this system include operating side, 3d display,
Motion controller, augmented reality processing unit, virtual environment unit, actuating station and 3d endoscope；
Described operating side sends coordinate information to motion controller；Described motion controller sends motion control commands to holding
Row end；Actuating station sends displacement information to virtual environment unit；It is existing to strengthening that described virtual environment unit sends image information
Real processing unit；Described 3d endoscope sends the image video of operative site to augmented reality processing unit；Described augmented reality
Processing unit sends augmented reality video to 3d display.
Further, described system also includes force feedback unit, and described virtual environment unit sends tactile data to force feedback
Unit, described force feedback unit sends touch feedback to operating side.
Further, described augmented reality processing unit is used for receiving the image video of image information and operative site, passes through
Image processing algorithm completes three-dimensional virtual environment organ and operative site demarcation, registering and tracking superposition, points out pathological tissues, so
Afterwards augmented reality video is sent to 3d display.
Further, described virtual reality unit is used for for the image data of patient being converted into three dimensional virtual models, and root
Judge that operating theater instruments and the relative coordinates of pathological tissues in virtual environment and normal structure are closed according to the displacement information that actuating station sends
System, and generate tactile data and image information, tactile data is sent to force feedback unit, by the image information in virtual environment
Pass to augmented reality processing unit；
Described tactile data is to generate optimum operation pathway and force feedback guidance information according to anatomic model；Force feedback
Guidance information includes the force feedback information of normal structure and the force feedback information of pathological tissues and grasps in setting operation pathway
Haptic interaction information when making；Described image information includes virtual organization deformation analog information.
Further, described force feedback unit is used for judging surgical device according to the three-dimensional virtual environment of virtual environment cell formation
In tool and operation technique area virtual environment, the relative coordinates relation of pathological tissues and normal structure, anti-according to operation pathway and power
Feedback guidance information completes force feedback guiding by operating side.
The beneficial effects of the present invention is: a kind of force feedback operating robot control based on augmented reality that the present invention provides
System processed, based on Medical Imaging Technology and computer image processing technology, builds pathological tissues using three-dimensional reconstruction virtual
Model, by dummy model and corrective surgery position real-time calibration, registering and tracking superposition, completes the deformation simulation of pathological tissues
And three-dimensional complex imaging, accurately calculate surgical target and operating theater instruments spacing, planning operation technique path automatically, by power simultaneously
Feedback device realizes the guiding of operation technique tactile and the prompting of normal structure mechanics, assists doctor to complete in conjunction with vision and tactile specific
Operation technique, effectively improves performance accuracy, safety and the reliability of operating robot.
Brief description
In order that the object, technical solutions and advantages of the present invention are clearer, below in conjunction with accompanying drawing the present invention is made into
The detailed description of one step, wherein:
Fig. 1 is the force feedback surgery operation robot control system block diagram based on augmented reality of the present invention；
Fig. 2 is augmented reality force feedback system schematic diagram；
Wherein, 1 be operating side, 2 be 3d display, 3 be motion controller, 4 be force feedback unit, 5 be augmented reality at
Reason unit, 6 be virtual environment unit, 7 be actuating station, 8 be 3d endoscope, 6-1 be virtual environment in pathological tissues, 6-2 be just
Normal tissue, 6-3 are operation pathway, 7-1 and 7-2 is operating theater instruments.
Specific embodiment
Below in conjunction with accompanying drawing, the preferred embodiments of the present invention are described in detail.
A kind of force feedback surgery operation robot control system based on augmented reality that the present invention provides, as shown in figure 1, this is
The main kinetic control system including force feedback interactive system and operating robot actuating station of system；This system specifically includes operation
End 1,3d display 2, motion controller 3, augmented reality processing unit 5, virtual environment unit 6, actuating station 7 and 3d endoscope
8；
Described operating side 1 sends coordinate information to motion controller 3；Described motion controller 3 sends motion control commands
To actuating station 7；Actuating station 7 sends displacement information to virtual environment unit 6；Described virtual environment unit 6 sends image information
To augmented reality processing unit 5；Described 3d endoscope 8 sends the image video of operative site to augmented reality processing unit 5；Institute
State augmented reality processing unit 5 to send augmented reality video to 3d display 2.
System also includes force feedback unit 4, and described virtual environment unit 6 sends tactile data to force feedback unit 4, described
Force feedback unit 4 sends touch feedback to operating side 1.
Motion controller 3 is used for receiving the motion control commands of operating side 1 and being sent to actuating station 7.
Augmented reality processing unit 5 is for receiving image information and the image of virtual reality unit 6 and 3d endoscope 8 transmission
Video information, is demarcated by the real time imaging of three-dimensional virtual environment and patient body, registering and follow the tracks of superposition, prompting pathological changes group
Knit, then augmented reality image information is transferred to 3d display 2.
Augmented reality force feedback system schematic diagram, as shown in Fig. 2 virtual reality unit 6 be used for by the image of patient (ct,
Mri etc.) data is converted into three dimensional virtual models, according to actuating station positional information judge operating theater instruments (7-1,7-2) with virtual
The relative coordinates relation of pathological tissues 6-1 and normal structure 6-2 in environment, and generate tactile data and image information, by tactile
Information sends to force feedback unit 4, and the image information in virtual environment is passed to augmented reality processing unit 5；Described tactile
Information is to generate optimum operation pathway and force feedback guidance information according to anatomic model；Force feedback guidance information includes normally
The force feedback information organized and the force feedback information of pathological tissues and haptic interaction letter during operation in setting operation pathway
Breath；Described image information includes virtual organization deformation analog information.
The three-dimensional virtual environment that force feedback unit 4 is used for according to virtual environment unit 6 builds judges operating theater instruments (7-1,7-
2) with pathological tissues 6-1 in operation technique area virtual environment and normal structure 6-2 relative coordinates relation, thus according to handss
Art path and force feedback guidance information complete force feedback guiding by operating side 1.
Doctor during completing robotic surgery, generally cannot direct access tactile feedback information, therefore increased handss
The difficulty of art operation, by augmented reality force feedback surgical robot system proposed by the present invention, with visualization touch feedback,
Assist a physician complete organ and tissue be accurately positioned, effectively improve the safety and reliability of operating robot.
Finally illustrate, preferred embodiment above only in order to technical scheme to be described and unrestricted, although logical
Cross above preferred embodiment the present invention to be described in detail, it is to be understood by those skilled in the art that can be
In form and various changes are made to it, without departing from claims of the present invention limited range in details.
Claims (2)
1. a kind of force feedback surgery operation robot control system based on augmented reality it is characterised in that: this system includes operating side
(1), 3d display (2), motion controller (3), augmented reality processing unit (5), virtual environment unit (6), actuating station (7) with
And 3d endoscope (8)；
Described operating side (1) sends coordinate information to motion controller (3)；Described motion controller (3) sends motor control life
Make to actuating station (7)；Actuating station (7) sends displacement information to virtual environment unit (6)；Described virtual environment unit (6) is sent out
Sending information image is to augmented reality processing unit (5)；The image video that described 3d endoscope (8) sends operative site is existing to strengthening
Real processing unit (5)；Described augmented reality processing unit (5) sends augmented reality video to 3d display (2)；
Described system also includes force feedback unit (4), and described virtual environment unit (6) sends tactile data to force feedback unit
(4), described force feedback unit (4) sends touch feedback to operating side (1)；
Described augmented reality processing unit (5) is used for receiving the image video of image information and operative site, by image procossing
Algorithm completes three-dimensional virtual environment organ and operative site demarcation, registering and tracking superposition, points out pathological tissues, then will strengthen
Real video sends to 3d display；
Described virtual reality unit (6) is used for the image data of patient being converted into three dimensional virtual models, according to actuating station (7)
The displacement information sending judges the relative coordinates relation of operating theater instruments and pathological tissues in virtual environment and normal structure, and generates
Tactile data and image information, tactile data are sent to force feedback unit (4), the image information in virtual environment are passed to
Augmented reality processing unit (5)；
Described tactile data is to generate optimum operation pathway and force feedback guidance information according to anatomic model；Force feedback guides
When information includes the force feedback information of normal structure and the force feedback information of pathological tissues and operates in setting operation pathway
Haptic interaction information；Described image information includes virtual organization deformation analog information.
2. a kind of force feedback surgery operation robot control system based on augmented reality according to claim 1, its feature exists
In: described force feedback unit (4) be used for according to the three-dimensional virtual environment of virtual environment cell formation judge operating theater instruments with virtual
The relative coordinates relation of pathological tissues and normal structure in environment, passes through operation according to operation pathway and force feedback guidance information
End completes force feedback guiding.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510184133.3A CN104739519B (en) | 2015-04-17 | 2015-04-17 | Force feedback surgical robot control system based on augmented reality |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510184133.3A CN104739519B (en) | 2015-04-17 | 2015-04-17 | Force feedback surgical robot control system based on augmented reality |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104739519A CN104739519A (en) | 2015-07-01 |
| CN104739519B true CN104739519B (en) | 2017-02-01 |
Family
ID=53580163
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510184133.3A Expired - Fee Related CN104739519B (en) | 2015-04-17 | 2015-04-17 | Force feedback surgical robot control system based on augmented reality |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104739519B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109288540A (en) * | 2017-07-24 | 2019-02-01 | 云南师范大学 | A kind of long-distance ultrasonic diagnosis system with touch feedback |
Families Citing this family (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017151999A1 (en) * | 2016-03-04 | 2017-09-08 | Covidien Lp | Virtual and/or augmented reality to provide physical interaction training with a surgical robot |
| CN106078747B (en) * | 2016-08-11 | 2018-11-06 | 贵州翰凯斯智能技术有限公司 | A kind of delay industrial operation control system based on virtual reality |
| CN106326856A (en) * | 2016-08-18 | 2017-01-11 | 厚凯（天津）医疗科技有限公司 | Surgery image processing method and surgery image processing device |
| US20190231436A1 (en) * | 2016-09-30 | 2019-08-01 | Koninklijke Philips N.V. | Anatomical model for position planning and tool guidance of a medical tool |
| US10969874B2 (en) * | 2017-03-21 | 2021-04-06 | Pcms Holdings, Inc. | Method and system for the detection and augmentation of tactile interactions in augmented reality |
| CN106847036A (en) * | 2017-03-30 | 2017-06-13 | 武汉大学 | A kind of Replacement of Hip Joint teaching display systems based on virtual reality technology |
| CN106781719A (en) * | 2017-03-30 | 2017-05-31 | 武汉大学 | A kind of microvascular anastomosis operation teaching display systems based on virtual reality technology |
| CN106920451A (en) * | 2017-03-30 | 2017-07-04 | 武汉大学 | A kind of operation teaching display systems based on virtual reality technology |
| CN106974730A (en) * | 2017-04-01 | 2017-07-25 | 上海术理智能科技有限公司 | Surgical simulation method, device and equipment based on virtual reality and medical image |
| CN107296650A (en) * | 2017-06-01 | 2017-10-27 | 西安电子科技大学 | Intelligent operation accessory system based on virtual reality and augmented reality |
| CN108198247A (en) * | 2018-01-12 | 2018-06-22 | 福州大学 | A kind of lateral cerebral ventricle puncture operation teaching tool based on AR augmented realities |
| JP6774445B2 (en) * | 2018-02-05 | 2020-10-21 | 本田技研工業株式会社 | Mobile control system, mobile and mobile control method |
| JP2021062412A (en) * | 2018-02-20 | 2021-04-22 | ソニー株式会社 | Tactile presentation apparatus and tactile presentation system |
| CN110338907A (en) * | 2018-04-05 | 2019-10-18 | 云南师范大学 | A kind of haptic navigation system for medical image guidance operation |
| WO2020133097A1 (en) * | 2018-12-27 | 2020-07-02 | 北京维卓致远医疗科技发展有限责任公司 | Mixed-reality-based control system |
| CN110559083B (en) * | 2019-09-10 | 2020-08-25 | 深圳市精锋医疗科技有限公司 | Surgical robot and control method and control device for tail end instrument of surgical robot |
| CN111437033B (en) * | 2020-04-03 | 2021-03-02 | 天津理工大学 | Virtual sensor for vascular intervention surgical robot system |
| CN111643189A (en) * | 2020-05-13 | 2020-09-11 | 山东省千佛山医院 | Surgical robot master-slave control experimental system and method based on force feedback technology |
| CN111991088B (en) * | 2020-09-10 | 2022-02-11 | 苏州大学 | Minimally invasive surgery robot and tail end clamp holder thereof |
| CN112618026B (en) * | 2020-12-15 | 2022-05-31 | 清华大学 | Remote operation data fusion interactive display system and method |
| CN112914731A (en) * | 2021-03-08 | 2021-06-08 | 上海交通大学 | Interventional robot contactless teleoperation system based on augmented reality and calibration method |
| CN113633387B (en) * | 2021-06-21 | 2024-01-26 | 安徽理工大学 | Surgical field tracking supporting laparoscopic minimally invasive robot touch interaction method and system |
| CN113940755B (en) * | 2021-09-30 | 2023-05-02 | 南开大学 | Surgical planning and navigation method integrating surgical operation and image |
| CN114098981B (en) * | 2021-11-24 | 2024-05-07 | 东南大学 | Double-arm-coordinated head and neck auxiliary traction operation robot and control method thereof |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110306986A1 (en) * | 2009-03-24 | 2011-12-15 | Min Kyu Lee | Surgical robot system using augmented reality, and method for controlling same |
| CN203802428U (en) * | 2014-01-02 | 2014-09-03 | 中国科学院沈阳自动化研究所 | Control system for gastrointestinal endoscopy-assisted intervention robot |
| CN204033495U (en) * | 2014-08-14 | 2014-12-24 | 中国科学院重庆绿色智能技术研究院 | The reluctance type force-feedback control system of telesurgery robot |
| CN204542390U (en) * | 2015-04-17 | 2015-08-12 | 中国科学院重庆绿色智能技术研究院 | A kind of force feedback surgery operation robot control system based on augmented reality |
-
2015
- 2015-04-17 CN CN201510184133.3A patent/CN104739519B/en not_active Expired - Fee Related
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109288540A (en) * | 2017-07-24 | 2019-02-01 | 云南师范大学 | A kind of long-distance ultrasonic diagnosis system with touch feedback |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104739519A (en) | 2015-07-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104739519B (en) | Force feedback surgical robot control system based on augmented reality | |
| US11589937B2 (en) | Systems and methods for constraining a virtual reality surgical system | |
| US11931123B2 (en) | Robotic port placement guide and method of use | |
| CN204542390U (en) | A kind of force feedback surgery operation robot control system based on augmented reality | |
| US10064682B2 (en) | Collision avoidance during controlled movement of image capturing device and manipulatable device movable arms | |
| CN112370159A (en) | System for guiding a user to position a robot | |
| CN112618026B (en) | Remote operation data fusion interactive display system and method | |
| US20210315637A1 (en) | Robotically-assisted surgical system, robotically-assisted surgical method, and computer-readable medium | |
| EP2277441A1 (en) | Method for generating images of a human body zone undergoing a surgical operation by means of an apparatus for minimally invasive surgical procedures | |
| KR20120040687A (en) | Virtual measurement tool for minimally invasive surgery | |
| Ferraguti et al. | Augmented reality and robotic-assistance for percutaneous nephrolithotomy | |
| US20220096163A1 (en) | Camera control systems and methods for a computer-assisted surgical system | |
| WO2020236814A1 (en) | Systems and methods for generating workspace volumes and identifying reachable workspaces of surgical instruments | |
| Ho et al. | Supervised control for robot-assisted surgery using augmented reality | |
| US20230341932A1 (en) | Two-way communication between head-mounted display and electroanatomic system | |
| US20240033005A1 (en) | Systems and methods for generating virtual reality guidance | |
| US20240029858A1 (en) | Systems and methods for generating and evaluating a medical procedure | |
| WO2023150449A1 (en) | Systems and methods for remote mentoring in a robot assisted medical system | |
| EP4251087A1 (en) | Systems providing synthetic indicators in a user interface for a robot-assisted system | |
| CN115297799A (en) | System and method for optimizing configuration of a computer-assisted surgery system for accessibility of a target object |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20170201 |