CN115054374A

CN115054374A - AR-based surgical robot field troubleshooting system and method, and storage medium

Info

Publication number: CN115054374A
Application number: CN202210730988.1A
Authority: CN
Inventors: 不公告发明人
Original assignee: Shanghai Microport Medbot Group Co Ltd
Current assignee: Shanghai Microport Medbot Group Co Ltd
Priority date: 2022-06-24
Filing date: 2022-06-24
Publication date: 2022-09-16

Abstract

The invention provides an AR-based surgical robot field fault elimination system, an AR-based surgical robot field fault elimination method and a storage medium, wherein virtual fault elimination operation models corresponding to various fault reasons in a one-to-one mode can be established according to historical data, the current fault reason is deduced through data analysis such as video analysis results and fault codes, and then the corresponding virtual fault elimination operation model is found, so that the found virtual fault elimination operation model can be superposed on a surgical robot through AR equipment, that is, the actual fault elimination step is fused with a virtual three-dimensional model corresponding to a pre-planned fault elimination path in a virtual-real mode, related personnel can be effectively guided to eliminate faults in real time, the fault elimination time is shortened, the fault elimination efficiency is improved, the fault elimination error rate is reduced, and the surgical time is shortened.

Description

AR-based surgical robot field troubleshooting system and method, and storage medium

Technical Field

The invention relates to the technical field of medical instrument fault elimination, in particular to an AR-based surgical robot field fault elimination system and method and a storage medium.

Background

In recent years, surgical robot systems have become powerful tools for assisting doctors in completing operations (e.g., minimally invasive surgery) by virtue of their clear stereoscopic feedback, flexible manipulation, and the like, and have been widely used in the field of medical operations.

Referring to fig. 1, a current surgical robotic system generally includes a doctor-side console 10, an image display table 20, and a patient-side surgical table 30. the patient-side surgical table 30 generally includes a plurality of robotic arms (not labeled) including a vision robotic arm for mounting or connecting an endoscope and at least one surgical robotic arm for mounting or connecting a surgical instrument. The surgical robot system determines the initial position and state of the machine through self-inspection before operation, and after manually installing the devices such as a stamp card (not shown), a surgical instrument (not shown), an endoscope (not shown) and a sterile bag (not shown), the surgical robot system can enter a master-slave operation state, and a surgeon can input a control command from the surgeon console 10 and then control the surgeon console 30 at the patient end to drive a mechanical arm to perform an operation on the patient.

However, the current field troubleshooting solution for surgical robots has the following technical problems:

1. the surgical robot system has a safety fault alarm and relief method prompt, so that when a fault occurs, the surgical robot system can give a prompt fault code on display equipment, and a counter-following person needs to perform fault relief according to the fault code, however, as the types of the fault codes of the surgical robot system are many and complex, the fault relief person cannot quickly position and relieve the fault of the surgical robot;

2. the fault elimination method and the fault elimination steps depend on the skills and subjective judgment ability of professionals, and the skill training is long in time consumption and high in cost;

3. for the removal of doctors with low proficiency and unexpected problems, on-site counter-following personnel are required to remove faults timely and accurately, the labor cost is high, and the counter-following personnel carry out fault removal according to subjective judgment and experience, so that the fault removal efficiency is different from person to person.

4. The fault removal operation process cannot be monitored and responded, so that the fault cannot be accurately and efficiently removed.

Disclosure of Invention

The invention aims to provide an AR-based surgical robot field troubleshooting system, an AR-based surgical robot field troubleshooting method and a readable storage medium, which can solve at least part of technical problems in the prior art.

To achieve the above object, the present invention provides an AR-based surgical robot field troubleshooting system, comprising:

the fault processing system is used for establishing a virtual fault removal operation model corresponding to a plurality of fault reasons of the surgical robot according to historical data;

the video analysis system is used for acquiring and analyzing operation videos of the surgical robot before and during operation so as to obtain first behavior data before fault occurrence;

the data interaction system is used for performing data interaction with the video analysis system and the motion control system of the surgical robot so as to acquire state data of the surgical robot in real time;

the fault analysis system is used for analyzing the first behavior data and the state data so as to find out a virtual fault elimination operation model corresponding to the current fault reason from the fault processing system according to the analysis result;

and the AR equipment is used for overlaying the virtual troubleshooting operation model determined by the fault analysis system on the surgical robot in a virtual-real fusion mode through coordinate mapping so as to guide related personnel to carry out troubleshooting operation.

Optionally, the fault analysis system is further configured to issue each step of fault removal operation in the virtual fault removal operation model to the AR device in steps; the video analysis system is further configured to collect and analyze a video of the troubleshooting operation of the relevant person at the current step to obtain second behavior data, determine whether the troubleshooting operation of the relevant person at the current step is valid, and enable the fault analysis system to issue the next troubleshooting operation in the virtual troubleshooting operation model to the AR device only when the troubleshooting operation is valid.

Optionally, the fault analysis system is further configured to, when the video analysis system determines that the troubleshooting operation of the relevant person at the current step is invalid, enable the AR device to instruct the relevant person to repeat the troubleshooting operation at the current step, or perform comprehensive analysis on the second behavior data, the first behavior data, and the state data, so as to obtain a current fault cause and a corresponding virtual troubleshooting operation model again.

Optionally, the surgical robot includes patient end operation table and doctor end control cabinet, the vision analysis system includes first camera device and second camera device, first camera device is used for shooing each part state of operation video and patient end operation table in the patient end operation table region, second camera device is used for shooing each part state of operation video and doctor end control cabinet in doctor end control cabinet region.

Optionally, the AR device has a binocular vision module for establishing the coordinate mapping.

Optionally, the video analysis system or the fault handling system is further configured to: classifying historical operation videos according to different fault reasons to obtain corresponding classified videos, modeling a single-frame image and a multi-frame image of each classified video, and obtaining a feature model and a video model of each fault reason;

the video analysis system is further used for capturing the operation video of the surgical robot in real time, meanwhile, taking the fault occurrence moment as a starting point, capturing the operation video captured before the fault occurs by using the video model of the current fault reason, and analyzing behavior data of the captured operation video by using the feature model of the current fault reason to obtain first behavior data corresponding to the current fault reason.

Optionally, the surgical robot automatically generates a fault code corresponding to a current fault cause when a fault occurs, and the fault analysis system obtains the virtual troubleshooting operation model according to the fault code, the first behavior data, and the state data; or the AR device is provided with a trigger button used for manually triggering the fault analysis system when the surgical robot has a fault, so that the fault analysis system analyzes a fault code corresponding to the current fault reason.

Optionally, the surgical robot includes a pedal, a mechanical arm, an endoscope, a power box, a poking card, and a surgical instrument, wherein the mechanical arm includes a master hand on a doctor end console and a slave hand on a patient end console, and the failure cause includes at least one of mismatch of surgical postures, mismatch of master and slave postures, fixed view of endoscope failure, no output of energy when stepping on the pedal, no zero return of the power box, loosening of the poking card, reaching of a limit position by a telescopic joint of the mechanical arm, and no loosening of the instrument.

Based on the same invention concept, the invention also provides a field fault removal method for the surgical robot, which comprises the following steps:

according to historical data, virtual fault removal operation models corresponding to multiple fault reasons of the surgical robot are established in advance;

collecting and analyzing operation videos of the surgical robot before and during operation in real time to obtain first behavior data before a fault occurs;

acquiring state data of the surgical robot in real time;

analyzing the first behavior data and the state data to find out a virtual troubleshooting operation model of the current fault reason from all the virtual troubleshooting operation models established in advance according to the analysis result;

and overlaying the found virtual troubleshooting operation model on the surgical robot in a virtual-real fusion mode through coordinate mapping of the AR equipment so as to guide related personnel to carry out troubleshooting operation.

Optionally, the found virtual troubleshooting operation model is superimposed on the surgical robot in a virtual-real fusion manner according to the steps, a troubleshooting operation video of the relevant person in the current step is collected and analyzed at the same time to obtain second behavior data, whether the troubleshooting operation of the relevant person in the current step is effective or not is judged, and when the troubleshooting operation is effective, the next troubleshooting operation in the virtual troubleshooting operation model is issued to the AR device.

Optionally, the surgical robot field troubleshooting method further includes:

and when the fault removal operation of the current step of the related personnel is judged to be invalid, the AR equipment guides the related personnel to repeat the fault removal operation of the current step, or the second behavior data, the first behavior data and the state data are comprehensively analyzed, so that the corresponding current fault reason and the corresponding virtual fault removal operation model are obtained again.

Optionally, the surgical robot field troubleshooting method further includes:

classifying historical operation videos according to different fault reasons to obtain corresponding classified videos, modeling a single-frame image and a multi-frame image of each classified video, and obtaining a feature model and a video model of each fault reason;

the method comprises the steps of collecting operation videos of the surgical robot in real time, meanwhile, taking the fault occurrence time as a starting point, intercepting the operation videos collected before the fault occurs by using a video model of the current fault reason, and analyzing behavior data of the intercepted operation videos by using a feature model of the current fault reason to obtain first behavior data corresponding to the current fault reason.

Based on the same inventive concept, the invention also provides a storage medium, on which a computer program is stored, which, when executed by a processor, implements the surgical robot field troubleshooting method of the invention.

Compared with the prior art, the technical scheme of the invention at least has the following beneficial effects:

1. the technical scheme of the invention can establish the virtual fault removal operation model corresponding to a plurality of fault reasons in the surgical robot according to historical data, so that after the operation videos and the state data of the surgical robot before and during operation are collected and analyzed in real time, the obtained state data and the behavior data of the surgical robot before the fault occurs are integrated and analyzed, the virtual fault removal operation model corresponding to the current fault reason is found from all the established virtual fault removal operation models, the found virtual fault removal operation model is superposed on the surgical robot through the virtual-real fusion of AR equipment to guide relevant personnel to carry out fault removal operation, and the fault removal method can rapidly position the fault of the surgical robot and rapidly carry out fault removal under the assistance of the AR equipment, the method does not depend on the skills and subjective judgment ability of professionals, reduces the cost of troubleshooting and improves the accuracy of troubleshooting;

2. the found virtual troubleshooting operation model can be overlaid on the surgical robot according to the steps, so that the current operation of related personnel for troubleshooting can be subjected to real-time video acquisition and analysis, and the validity (namely correctness) of the current operation of the related personnel for troubleshooting can be judged in real time, so that the accuracy of the troubleshooting operation steps can be ensured, the troubleshooting time is saved, and the safety of the operation is improved.

3. When the current operation of the relevant personnel for troubleshooting is judged to be invalid, the current operation data (namely the second behavior data) of the relevant personnel and the obtained first behavior data and state data are further combined to re-analyze the failure reason and/or the virtual troubleshooting operation model, so that the problem that the failure which is newly generated due to the improper operation of the relevant personnel in the troubleshooting process cannot be debugged can be avoided, the troubleshooting error rate is further reduced, and the operation time is shortened.

Drawings

Fig. 1 is a schematic structural diagram of a surgical robot system according to the related art.

Fig. 2 is a schematic structural design diagram of a surgical robot field troubleshooting system according to an embodiment of the invention.

Fig. 3 is a schematic layout diagram of a surgical robot field troubleshooting system and a surgical robot according to an embodiment of the present invention.

Fig. 4 is a schematic diagram illustrating the classification of the behavior codes of the surgical robot according to the embodiment of the present invention.

Fig. 5 is a flow chart of a video analysis method according to an embodiment of the invention.

FIG. 6 is a schematic diagram of a method for video analysis of a joint of a master hand according to an embodiment of the present invention.

Fig. 7 is a schematic view of a video analysis method regarding the zero position of the power pack according to an embodiment of the present invention.

Fig. 8 is an illustration of the cause of a fault (i.e., the type of fault) that can be removed by the surgical robotic field troubleshooting system of an exemplary embodiment of the present invention.

Fig. 9 is a schematic structural diagram of an AR device in a surgical robot field troubleshooting system according to an embodiment of the present invention.

Fig. 10 is a flowchart illustrating a surgical robot field troubleshooting method according to an embodiment of the present invention.

FIG. 11 is a schematic illustration of a troubleshooting process for an instrument failure with respect to energy output, in accordance with an embodiment of the present invention.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. It will be apparent, however, to one skilled in the art, that the present invention may be practiced without one or more of these specific details. In other instances, well-known features have not been described in order to avoid obscuring the invention. It is to be understood that the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Like reference numerals refer to like elements throughout. It will be understood that when an element is referred to as being "connected," "coupled," or "coupled" to another element, it can be directly connected or intervening elements may be present. In contrast, when an element is referred to as being "directly connected to" other elements, there are no intervening elements present. Although the terms "left", "right", etc. may be used to describe various elements, components and/or sections, these elements, components and/or sections should not be limited by these terms. These terms are only used to distinguish one element, component or section from another element, component or section. Thus, a "left" element, component, or section discussed below could be termed a "right" element, component, or section without departing from the teachings of the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising" are used in an inclusive sense to specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof. As used herein, the term "and/or" includes any and all combinations of the associated listed items.

The technical solution proposed by the present invention will be further described in detail with reference to the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description. It is to be noted that the drawings are in a very simplified form and are not to precise scale, which is merely for the purpose of facilitating and distinctly claiming the embodiments of the present invention.

Referring to fig. 2 and 3, an embodiment of the present invention provides an AR-based surgical robot field troubleshooting system, which is used for troubleshooting a corresponding surgical robot (or surgical robot system). The surgical robot may be any surgical robot system that is currently available.

As an example, the surgical robot includes a doctor-side console 10, an imaging trolley 20, a patient-side surgical table 30, auxiliary equipment 40, and a patient bed 50. The doctor-side console 10, the imaging trolley 20, and the patient-side operating table 30 may be communicatively connected to each other by any suitable means (e.g., a wireless means such as a local area network or a wired means such as a data line).

The patient bed 50 is disposed at a suitable location near the patient end operating table 30 to facilitate the patient in providing a sitting, lying, etc. Auxiliary equipment 60 is also provided near the patient-end operating table 30 and may include, for example, a sterile table (on which surgical instruments to be used, etc. may be placed), a ventilator, an anesthesia machine, an extracorporeal blood circulation machine, or some vital sign detection device.

A patient-side surgical table 30 (also referred to as a slave cart) for performing related procedures on a patient includes a power pack (not shown), surgical instruments 32, an endoscope, a stab card, and at least one slave hand 31. The slave hand 31 may also be referred to as a tool arm, an adjusting arm, a slave mechanical arm, a slave manipulator, etc. for mounting or connecting a surgical instrument 32, a stab card (not shown) and an endoscope (not shown), one end of the slave hand 31 may be connected to the same passive adjusting arm, which may realize lifting and translation of the slave hand, and each slave hand has a plurality of joints, which may realize extension and retraction and rotation of the slave hand. The endoscope is used to acquire the image information of the operation environment such as the human tissue organ, the operation instrument 32, the blood vessel and the body fluid, and the operation instrument 32 may include a scalpel, a forceps, a puncture outfit, a suture needle, etc., which can perform different functions including clamping, cutting, suturing, anastomosing, etc. The stab card is typically secured to the slave hand and the surgical instrument 32 is mounted to the stab card, which facilitates installation and replacement of the surgical instrument 32. The power box (which can be a power motor) is a power source of the operation table at the patient end, and can drive the slave hand to move correspondingly according to the instruction of the control table at the doctor end, and in order to ensure the use accuracy of the operation robot, the power box is generally required to be set with a zero position, and the automatic return to the zero position is required to be realized.

The doctor end console 10 is used for controlling the patient end operation table 30 to drive the slave hand 31 and operate the surgical instrument 32 to perform corresponding operations on the patient under the operation of a surgeon. Wherein, doctor end control cabinet 10 mainly has 4 functional module, and from the top down is in proper order: 1) a stereoscopic image viewing window (not labeled) that can provide a surgeon with high-definition stereoscopic images during surgery for viewing the lesion and surgical instruments 32 at the patient-side surgical table 30; 2) a master hand 11, wherein the master hand 11 is an operation action instruction unit, and may be operated by one hand or both hands, a surgeon holds the end of the master hand 11 during an operation and sends a desired movement position and posture (clamping) instruction to the slave hand 31, and the master-slave mapping relationship between the operation of the surgeon on the master hand 11 on the surgeon console 10 and the operation generated by the slave hand 32 according to the instruction sent by the surgeon console 10 is established; 3) a control panel (not labeled) for adjusting surgical parameters of some preoperative surgical robots, such as endoscope angle, zoom, etc.; 4) the pedals 12 are usually provided in plural, and are used to realize the functions of adjusting the operation actions commonly used in the operation, disconnecting the master-slave motion mapping, controlling the endoscope motion, starting the patient-side operation table 30, and the like.

Before operation, the initial position and the state of a surgical machine are determined through machine self-inspection, the master-slave operation state can be entered after devices such as a poking card, a surgical instrument 32, an endoscope and a sterile bag are installed, and in the operation process, a doctor can frequently operate a master hand 11 to control a slave hand 31, so that the effect of the master-slave operation is achieved, and the doctor can realize functional operations such as master-slave clutch, endoscope control, slave hand switching and electrotome energy control through loosening and closing of pedals 12. Obviously, the field fault of the surgical robot can be caused by improper positioning before and during operation, master-slave operation or unexpected conditions (including operation posture mismatch, master-slave posture mismatch, endoscope fault view fixation, no energy output when an energy instrument pedal is stepped on, no zero position of a power box, poking and releasing, limit position reaching of a telescopic joint, no releasing of an instrument and the like) encountered by the surgical robot. One of the design objectives of the AR-based surgical robot field troubleshooting system of this embodiment is to ensure that field counter personnel can timely and accurately troubleshoot faults, reduce experience requirements on the troubleshooting personnel, and finally ensure the efficiency and safety of the surgery.

Specifically, the AR-based surgical robot field troubleshooting system of the present embodiment includes an AR device 60, a video analysis system 70, a fault processing system 80, a data interaction system 81, and a fault analysis system 82.

The fault handling system 80 is configured to establish virtual fault-removing operation models corresponding to various fault causes (also referred to as multiple fault types) of the surgical robot one-to-one according to historical data, so that once a fault occurs and a specific fault cause is analyzed, the corresponding virtual fault-removing operation model can be found and superimposed on a real object of the surgical robot in a virtual-real fusion manner through the AR device 60, and a person following the surgical robot is guided to handle the fault timely and correctly. The fault processing system 80 may include a data storage unit (not shown) and a fault model generation unit (not shown), wherein the fault model generation unit is configured to analyze historical data and establish a corresponding virtual fault-removing operation model according to an analysis result, the data storage unit may capture, update, and store the historical data (including clinical operation data, machine parameters, fault data, and the like), and may also store each fault cause analyzed by the fault model generation unit, each virtual fault-removing operation model, and a mapping relationship between the fault cause and the virtual fault-removing operation model.

The video analysis system 70 is used to collect and analyze the operation video of the surgical robot before, during or even after the operation to obtain the first behavior data before the occurrence of the fault. The video analysis system 70 may include a first camera 71, a second camera 72, and a video analysis device 73. The first camera 71 is arranged near the doctor end console 10 and is used for collecting the operation video of the doctor on the main hand 11, the control panel and the pedals 12 and the states of all the parts. The second camera device 72 is disposed near (e.g., above) the patient-side operating table 30, and is configured to collect operation videos and states of various components in the region of the patient-side operating table 30, including operation videos that are operated by the operator according to instructions from the doctor-side console 10, such as a hand, and operation videos that are troubleshooting performed on various components on the patient-side operating table 30 by the relevant operator. The video analyzing device 73 may be any suitable device, module, etc. such as a data processor, and may perform corresponding analysis on the videos collected by the first camera device 71 and the second camera device 72 to obtain corresponding behavior data, for example, capture a video within a preset time period before a fault occurs, so as to analyze whether an operation procedure on the patient-side operating table 30 is correct and effective.

Optionally, referring to fig. 4, after analyzing the operation video, the video analysis system 70 of this embodiment can obtain corresponding behavior codes, as shown in the second column of the table in fig. 4, and in these behavior codes, the first bit represents a corresponding module in the surgical robot, the second bit represents a joint, and the last three bits represent a behavior type.

Further optionally, referring to fig. 5, the video analysis system 70 is further configured to perform the following steps:

s1, classifying the historical operation videos according to different fault reasons to obtain corresponding classified videos, modeling single-frame images and multi-frame images of each classified video, and obtaining feature models and video models of the fault reasons;

s2, when the operation video of the surgical robot is collected in real time, the operation video collected before the fault occurs is intercepted by using the video model of the current fault reason with the moment of the fault occurrence as the starting point, or the collected operation video is segmented;

and S3, performing behavior data analysis on the intercepted operation video by using the characteristic model of the current fault reason to obtain first behavior data corresponding to the current fault reason.

As an example, the behavior of the surgical instrument 32 on the surgical robot within a certain time before the occurrence of the fault is a necessary condition for causing the occurrence of the current fault, so that the analysis and behavior recognition of the operation videos of the master hand and the slave hand can diagnose the cause of the fault (i.e., the fault cause) more accurately. Specifically, referring to fig. 5 and 6, in the process of analyzing and behavior recognizing the operation video of the main hand by the video analysis system 70, when the step S1 is executed, the historical operation videos are classified according to different failure reasons to obtain corresponding main hand classification videos, the main hand classification videos are classified, the main hand single-frame image is extracted for modeling, the main hand feature model T1 is generated, and the frame number (i.e., multi-frame image) and the node variation [ D ] of the frequently-swaying main hand posture joint are extracted]Modeling is carried out, and a master video model V1 is generated; when the step S2 is executed, capturing the operation videos of the joints 1-7 of the main hand in real time, taking the fault occurrence moment as a starting point, and intercepting the video data of a period of time before the fault occurs from the operation videos of the joints 1-7 of the main hand based on the duration/frame number of the video model V1 of the main hand; when step S3 is executed, position coordinates of 7 joint points are extracted from the captured video based on the master hand feature model T1, and the joint point coordinate of the ith joint is assumed to be P1 _i ＝(X _i ,Y _i ,Z _i ) And i is 1-7, calculating the distance variation of the joint points of two adjacent frames, and accumulating the distance variation to be recorded as [ D ] _i ]. When [ D ]]–[D _i ]When ≦ ε, the corresponding behavior code (e.g., 216XX) is determined and sent to the fault analysis system 82. Where P1 is the coordinates of the joint point of the first frame of the ith joint, P1' is the coordinates of the joint point of the second frame of the ith joint, and epsilon is a threshold value and is used to determine whether the distance difference is within the required range.

In this example, the video analysis system 70 may analyze and recognize the operation video of each slave hand by using the above-mentioned process of analyzing and recognizing the operation video of the master hand by the video analysis system 70.

As another example, analyzing and identifying the behavior of the operation video of the zero position of the power box can also more accurately diagnose the cause of the fault (i.e. the fault cause). Specifically, referring to fig. 5 and 7, the polar coordinates of the joints of the 4 separators of the power box are respectively predefined as P2 _i (r,θ _i ) (i ═ 1, 2, 3, 4), wherein:

therefore, in the process of analyzing and behavior recognizing the operation video at the zero position of the power box by the video analysis system 70, when the step S1 is executed, the historical operation videos are classified according to different fault reasons to obtain corresponding power box classification videos, the power box classification videos are subjected to modeling, single-frame images (including 4 partition board joints) of the power box are extracted, a power box characteristic model T2 is generated, and the image frame number and the polar coordinate variation cumulative value [ theta ] during self-test of the power box are extracted]Generating a power box video model V2; when the step S2 is executed, video data in a period of time before the fault occurs are intercepted on the basis of a power box video model V2 by taking the fault occurrence moment as a starting point of the power box partition board joint video collected in real time; in executing the above step S3, the power box is extracted from the cut video based on the power box feature model T2The polar coordinates of the joint points of the 4 isolation plates are integrated by the angle variation and are recorded as [ theta ] _i ]. When [ theta ], [ theta ]]–[θ _i ]And when the value is less than or equal to epsilon, determining that the self-checking is finished, judging whether the polar coordinate theta value of the last frame of image after the self-checking is finished is 0, if so, determining that the self-checking is successful, otherwise, determining a behavior code (such as 116XX) and sending the behavior code to a fault analysis system. Where P2 is the first frame joint polar coordinates of the ith spacer joint, P2' is the second frame joint polar coordinates of the ith spacer joint, and ε is a threshold and used to determine whether the angle difference is within this range.

As still another example, analyzing the operation video of the pedals and identifying the behavior can also more accurately diagnose the cause of the occurrence of the fault (i.e., the cause of the fault). Specifically, referring to fig. 5 and 8, in the process of analyzing and behavior recognizing the operation video of the pedals by the video analysis system 70, when the step S1 is executed, the historical operation video is classified according to different fault reasons to obtain a corresponding pedal classification video, the pedal classification video is modeled by extracting a pedal single-frame image to generate a pedal feature model T3, and a pedal multi-frame image and a pedal stroke variation are extracted to generate a pedal video model V3; when the step S2 is executed, the video data of the pedal operation video collected in real time in a period before the fault occurs is captured based on the pedal video model V3, with the fault occurrence time as a starting point; in the step S3, the captured video data is compared and analyzed based on the pedal feature model T3.

With continued reference to fig. 2 and 3, the data interaction system 81 is used for data interaction with the video analysis system 70 and a motion control system (not shown) of the surgical robot to obtain status data of the surgical robot in real time. Among them, the motion control system of the surgical robot is mainly used to control the slave hand to be movable according to the operation of the master hand.

With continued reference to fig. 2 and 3, the fault analysis system 82 is configured to integrate and analyze the behavior data obtained by the video analysis system 70 and the status data obtained by the data interaction system, so as to determine a current fault cause, and find a virtual troubleshooting operation model corresponding to the current fault cause from the fault processing system 80 according to the obtained current fault cause. Referring to fig. 8, the current failure cause includes at least one of an operation posture mismatch, a master-slave posture mismatch, a failure view of the endoscope, no energy output when stepping on the foot, a zero position failure of the power box, a release of the poking card, a limit position of the telescopic joint of the mechanical arm, and a failure of the instrument release.

As an example, referring to fig. 9, the fault analysis system 82 determines that the fault cause is "primary hand posture mismatch" according to the video analysis result "frequently shake primary hand posture joint" of the video analysis system 70 and the corresponding status data in the data interaction system 81, and further determines that the corresponding virtual fault elimination model (i.e. fault elimination method) "unplug and plug surgical instrument again" according to the video analysis result "frequently shake primary hand posture joint" of the video analysis system 70 and the obtained fault cause "primary hand posture mismatch". After the virtual troubleshooting model is overlaid on the surgical robot real object through the AR device 60, relevant personnel can see relevant guidance prompts about're-inserting surgical instruments' through the AR device 60.

With continued reference to fig. 2 and 3, AR device 60 is configured to superimpose the virtual troubleshooting operation model determined by failure analysis system 81 on the surgical robot through a virtual-real fusion manner by coordinate mapping so as to guide relevant personnel to perform troubleshooting operations. AR device 60 may be an AR glasses, AR helmet, or the like.

Referring to fig. 10, as an example of the present embodiment, the AR device 60 is an AR glasses having a binocular vision module 61 and a trigger button 62, the binocular vision module 61 is used for implementing a coordinate mapping relationship, and the trigger button 62 is used for implementing a manual trigger troubleshooting instruction. Wherein the coordinate system (X) of the binocular vision module 61 of the AR glasses ₅ 、Y ₅ 、Z ₅ ) Can pass through the mechanical position and the frame center coordinate system (X) of the AR glasses ₃ 、Y ₃ 、Z ₃ ) Coordinate mapping relation is established between the AR glasses and the center of the frame of the AR glasses is in a world coordinate system (X) ₀ 、Y ₀ 、Z ₀ ) Therein have corresponding coordinatesThus, the coordinate system (X) of the binocular vision module 61 of the AR glasses ₅ 、Y ₅ 、Z ₅ ) With the world coordinate system (X) ₀ 、Y ₀ 、Z ₀ ) Corresponding coordinate mapping relation can be established between the two. Further in conjunction with FIG. 3, the coordinate system (X) of the AR glasses ₅ 、Y ₅ 、Z ₅ ) In the world coordinate system (X) ₀ 、Y ₀ 、Z ₀ ) The seat (X) of the operating table 20 at the middle and patient end ₂ 、Y ₂ 、Z ₂ ) Coordinate mapping relation is established, and coordinate system (X) of AR glasses ₅ 、Y ₅ 、Z ₅ ) In the world coordinate system (X) ₀ 、Y ₀ 、Z ₀ ) Coordinate system (X) of the center and doctor end console 10 ₄ 、Y ₄ 、Z ₄ ) A coordinate mapping is established (this coordinate mapping can be known) whereby a patient-side coordinate system (X) can be realized ₂ 、Y ₂ 、Z ₂ ) Coordinate system (X) with the doctor's end console 10 ₄ 、Y ₄ 、Z ₄ ) And establishing a coordinate mapping relation.

Alternatively, after the trigger button 62 of the AR device 60 is manually triggered when the surgical robot fails, the fault analysis system 82 may also obtain a fault code corresponding to the cause of the current fault through analysis.

Of course, in other embodiments of the present invention, when the surgical robot has a function of alarming and prompting a fault, the trigger button 62 on the AR glasses may be omitted, and the surgical robot automatically generates a fault code corresponding to the current fault cause when a fault occurs, so that the fault analysis system 82 determines the current fault cause according to the fault code, and obtains the virtual troubleshooting operation model corresponding to the current fault cause according to the first behavior data of the video analysis system 70 and the status data of the data interaction system 81.

Optionally, the fault analysis system 82 is further configured to issue each step of fault removal operation in the determined virtual fault removal operation model to the AR device 60 step by step, so that relevant personnel can perform fault removal step by step, and therefore, relevant personnel can quickly locate and quickly remove a fault at a certain position of the surgical robot, which is beneficial to ensuring the accuracy of fault removal step by step, and under the condition that fault removal is complex, the skill and subjective judgment capability of a professional can not be relied on, the cost is reduced, the time is shortened, and the accuracy is improved.

In other embodiments of the present invention, on the basis that the failure analysis system 82 issues each step of failure removal operation in the virtual failure removal operation model determined by the failure analysis system in steps to the AR device 60, the video analysis system 70 is further configured to collect and analyze a video of the relevant person performing the current step of failure removal operation to obtain the second behavior data, and determine whether the current step of failure removal operation of the relevant person is valid, and when the determination is valid, the failure analysis system is enabled to issue the next step of failure removal operation in the virtual failure removal operation model to the AR device 60. When the video analysis system 70 determines that the troubleshooting operation of the relevant person at the current step is invalid, the failure analysis system 82 may enable the AR device 60 to instruct the relevant person to repeat the troubleshooting operation at the current step until the troubleshooting operation at the step is correctly performed, or the failure analysis system 82 may perform comprehensive analysis on the second behavior data, the first behavior data, and the status data obtained by the video analysis system 70 and the data interaction system 81 again, so as to obtain the current failure cause and the corresponding virtual failure troubleshooting operation model again, that is, update the virtual failure troubleshooting operation model.

That is, in this embodiment, the behavior data obtained by the video analysis system 70 includes first behavior data and second behavior data, the first behavior data is behavior data of the surgical robot before occurrence of the fault, which includes behavior data of the surgical robot generated before and during the operation to perform the operation, and the second behavior data is behavior data during troubleshooting, which includes operation behavior data of the relevant person and behavior data of the surgical robot when the relevant person troubleshoots the surgical robot.

The embodiment can realize real-time and accurate fault removal guidance for related personnel, and carries out real-time data feedback on the fault removal process, thereby ensuring the accuracy of fault removal operation steps, saving the fault removal time, increasing the safety of operations, and being capable of not depending on the skills and subjective judgment ability of professionals under the condition of more complex fault removal, reducing the cost and improving the accuracy.

It is understood that the AR device 60, the video analysis system 70, the fault processing system 80, the data interaction system 81, and the fault analysis system 82 may be combined and implemented in one module or apparatus, or any one of the AR device 60, the video analysis system 70, the fault processing system 80, the data interaction system 81, and the fault analysis system 82 may be split into a plurality of modules or apparatuses, or at least part of the functions of one or more of the AR device 60, the video analysis system 70, the fault processing system 80, the data interaction system 81, and the fault analysis system 82 may be combined with at least part of the functions of one or more of the other modules or apparatuses and implemented in one module or apparatus. According to an embodiment of the invention, at least one of the video analytics system 70, the fault handling system 80, the data interaction system 81, and the fault analysis system 82 may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or in hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or in a suitable combination of three implementations, software, hardware, and firmware. Alternatively, at least one of the video analysis system 70, the fault handling system 80, the data interaction system 81 and the fault analysis system 82 may be at least partially implemented as computer program modules which, when executed by a computer, may perform the functions of the respective module.

Referring to fig. 2 and fig. 11, an embodiment of the present invention further provides a surgical robot field troubleshooting method, which can be implemented by using the surgical robot field troubleshooting system of the present invention, where the method includes:

a1, the fault handling system 81 pre-establishes a virtual fault removal operation model corresponding to a plurality of fault reasons of the surgical robot according to historical data;

a2, acquiring and analyzing operation videos of the surgical robot before and during operation in real time by a video analysis system 70 to acquire first behavior data of the surgical robot before a fault occurs, and interacting the data interaction system 81 with the video analysis system 70 and a motion control system of the surgical robot to acquire state data of the surgical robot in real time; wherein step a3 may be further performed: is the data interaction system 81 or the video analysis system 70 or a human being judging from the acquired status data whether it is a malfunction of the motion control system of the surgical robot? If not, executing step a4, manually clicking the trigger button 62 of the AR device 60 to trigger the fault analysis system 82, if yes, executing step a5, where the surgical robot automatically triggers a fault code to trigger the fault analysis system 82;

a6, analyzing the fault code, the first behavior data and the status data by the fault analysis system 82 to determine a current fault cause according to an analysis result, and further finding out a virtual fault removal operation model of the current fault cause from all the virtual fault removal operation models established in advance;

and A7, overlaying the found virtual troubleshooting operation model on the surgical robot in a virtual-real fusion mode through the coordinate mapping of the AR device 60 so as to guide relevant personnel to carry out troubleshooting operation.

Optionally, in step a6, the failure analysis system 82 issues the found virtual failure-elimination operation model to the AR device 60 in steps, so that the virtual failure-elimination operation model is superimposed on the surgical robot by the AR device 60 through virtual-real fusion, and the relevant personnel may perform the failure-elimination operation in steps. Thus, the method further comprises step A8: the video analysis system 70 simultaneously collects and analyzes the video of the troubleshooting operation of the current step of the relevant person to obtain the second behavior data, and judges whether the troubleshooting operation of the current step of the relevant person is valid according to the second behavior data, when the judgment is valid, the fault analysis system 82 sends the next troubleshooting operation in the virtual troubleshooting operation model to the AR device 60, when the judgment is invalid, the AR device 60 guides the relevant person to repeat the troubleshooting operation of the current step, or the fault analysis system 82 comprehensively analyzes the second behavior data, the first behavior data and the state data to obtain the corresponding current fault reason and the corresponding virtual troubleshooting operation model again, until the troubleshooting operation of the current step of the surgical robot is effectively completed, the next troubleshooting operation is performed, and the process is circulated until all faults of the surgical robot are cleared, the trouble shooting work is ended, and the operation is continued.

Optionally, in the field troubleshooting method for a surgical robot of the embodiment, step a2 further includes:

firstly, classifying historical operation videos according to different fault reasons to obtain corresponding classified videos, and modeling a single-frame image and a multi-frame image of each classified video to obtain a feature model and a video model of each fault reason;

secondly, when the operation video of the surgical robot is collected in real time, the operation video collected before the fault occurs is intercepted by using the video model of the current fault reason as a starting point, and the behavior data analysis is carried out on the intercepted operation video by using the characteristic model of the current fault reason so as to obtain the first behavior data corresponding to the current fault reason.

It should be understood that, in the above embodiments, the videos collected by the video analysis system are mainly videos at the preoperative stage, the intraoperative stage and the troubleshooting stage, but the technical solution of the present invention is not limited thereto, and in other embodiments of the present invention, when the operation on the surgical robot after the operation may affect the safety of the next operation, the video analysis system may further collect the postoperative video for performing postoperative field troubleshooting, thereby ensuring the correct use of the postoperative surgical robot.

As an example, please further refer to fig. 4, 8-9 in combination, when the data interactive system 81 obtains a fault code without energy output, the motion control system obtains the pedal signal data S and the energy instrument data E, the video analysis system 70 analyzes and determines whether the pedal down-stroke before the fault occurs is greater than or equal to H (i.e. the pedal signal position is triggered), and the fault analysis system 82 provides a corresponding virtual fault-clearing operation model according to the fault code of the data interactive system 81, the data of the motion control system, and the result analyzed by the video analysis system 70, and further indicates a corresponding fault-clearing operation on the AR device, which is as follows:

when the analysis result indicates that the pedal has a signal, the energy apparatus exists, and the pedal is stepped in place, the current fault reason is determined to be no energy output, and finally the 'energy apparatus replacement' is indicated on the AR equipment by combining the analysis result;

when the analysis result indicates that the pedal has a signal, the energy instrument exists and the pedal is not stepped in place, the current fault reason is determined to be no energy output, and finally the pedal is indicated to be stepped downwards in place on the AR equipment according to the analysis result;

when the analysis result is that the pedal has no signal and has energy apparatus, and the pedal is stepped in place, determining that the current fault reason is no energy output, and finally indicating' replacing the pedal on the AR equipment according to the analysis result;

and when the analysis result indicates that the pedal has a signal and no energy instrument, and the pedal is stepped in place, determining that the current fault reason is no energy output, and finally indicating 'energy instrument replacement' on the AR equipment by combining the analysis result.

In addition, the failure cause, the analysis result, and the corresponding failure elimination method obtained in other examples may refer to fig. 4 and fig. 9, and are not described herein again.

Based on the same inventive concept, an embodiment of the present invention further provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements the field troubleshooting method for a surgical robot according to the present invention.

Further, the storage medium may be any medium that can contain, store, communicate, propagate, or transport the computer program. For example, the storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium, specific examples of which include magnetic storage devices such as magnetic tape or Hard Disk Drives (HDDs); optical storage devices, such as Compact Discs (CDROMs); a memory, such as a Random Access Memory (RAM) or a flash memory; and/or a wired, wireless communication link.

In summary, according to the technical scheme of the present invention, virtual troubleshooting operation models corresponding to various fault reasons one to one can be established according to historical data, and current fault reasons can be derived through data analysis such as analysis results of a video analysis system and machine fault codes, so as to find corresponding virtual troubleshooting operation models, and thus the found virtual troubleshooting operation models can be superimposed on the surgical robot through the AR device, that is, the virtual-real fusion is performed between the actual troubleshooting step and the virtual three-dimensional model corresponding to the pre-planned troubleshooting path, so that relevant personnel (which may be called platform following personnel) can be effectively guided to troubleshoot in real time, the troubleshooting time is shortened, the troubleshooting efficiency is improved, the troubleshooting error rate is reduced, and the operation time is shortened. Furthermore, whether the operation of troubleshooting of related personnel is effective or not can be fed back and analyzed in real time through the video analysis system, and the virtual three-dimensional model of the next troubleshooting step is overlaid on the AR equipment when the operation is effective, so that the accuracy of the troubleshooting operation step can be ensured, the troubleshooting time is saved, and the safety of the operation is improved through real-time data feedback.

The above description is only for the purpose of describing the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention, and any variations and modifications made by those skilled in the art according to the above disclosure are within the scope of the present invention.

Claims

1. An AR-based surgical robotic field troubleshooting system, comprising:

2. The surgical robotic field troubleshooting system of claim 1, wherein said failure analysis system is further for issuing each step of the troubleshooting operation in said virtual troubleshooting operation model to said AR device in steps; the video analysis system is further configured to collect and analyze a video of the troubleshooting operation of the relevant person at the current step to obtain second behavior data, determine whether the troubleshooting operation of the relevant person at the current step is valid, and enable the fault analysis system to issue the next troubleshooting operation in the virtual troubleshooting operation model to the AR device only when the troubleshooting operation is valid.

3. The surgical robot field troubleshooting system of claim 2, wherein said failure analysis system is further configured to, when said video analysis system determines that the troubleshooting operation of the relevant person at the current step is invalid, cause said AR device to instruct the relevant person to repeat the troubleshooting operation at the current step, or perform a comprehensive analysis on said second behavior data, said first behavior data, and said status data to retrieve the current failure cause and the corresponding virtual troubleshooting operation model.

4. The surgical robot field troubleshooting system of claim 1, wherein said surgical robot includes a patient-side operating table and a doctor-side console, said vision analysis system includes a first camera device for capturing an operating video of said patient-side operating table area and states of components of said patient-side operating table, and a second camera device for capturing an operating video of said doctor-side console area and states of components of said doctor-side console.

5. The surgical robotic field troubleshooting system according to claim 4, wherein said AR device has a binocular vision module for establishing said coordinate mapping.

6. The surgical robotic field troubleshooting system of any one of claims 1-5 wherein said video analysis system or said fault handling system is further for: classifying historical operation videos according to different fault reasons to obtain corresponding classified videos, modeling a single-frame image and a multi-frame image of each classified video, and obtaining a feature model and a video model of each fault reason;

7. A surgical robotic field troubleshooting system according to claim 6 wherein said surgical robot automatically generates a fault code corresponding to a current cause of a fault when a fault occurs, said fault analysis system deriving said virtual troubleshooting operational model based on said fault code and said first behavior data, said status data; or the AR device is provided with a trigger button used for manually triggering the fault analysis system when the surgical robot has a fault, so that the fault analysis system analyzes a fault code corresponding to the current fault reason.

8. A surgical robotic field troubleshooting system according to claim 6 wherein said surgical robot includes pedals, robotic arms, an endoscope, a power pack, a stab card, and surgical instruments, wherein the robotic arms include a master hand on a surgeon side console and a slave hand on a patient side surgical console, and wherein said causes of failure include at least one of surgical attitude mismatch, master-slave attitude mismatch, endoscope failure view fixation, no energy output when pedals are stepped on, no zero return of the power pack, disengagement of the stab card, reaching of a limit position by a telescopic joint of the robotic arms, and no instrument disengagement.

9. A surgical robot field troubleshooting method, comprising:

acquiring state data of the surgical robot in real time;

10. The surgical robot field troubleshooting method of claim 9, wherein the found virtual troubleshooting operational model is superimposed onto the surgical robot in a virtual-real fusion manner in steps, and a troubleshooting operational video of the relevant person at the current step is collected and analyzed at the same time to obtain second behavior data, and whether the troubleshooting operation of the relevant person at the current step is valid is judged, and when the troubleshooting operation is judged to be valid, a next troubleshooting operation in the virtual troubleshooting operational model is issued to the AR device.

11. The surgical robot field troubleshooting method of claim 10, wherein when it is determined that the troubleshooting operation of the relevant person at the current step is invalid, the relevant person is instructed by the AR device to repeat the troubleshooting operation at the current step, or the second behavior data, the first behavior data, and the status data are comprehensively analyzed to retrieve a corresponding current cause of failure and a corresponding virtual troubleshooting operation model.

12. A surgical robotic field troubleshooting method as claimed in any one of claims 9-11 further comprising:

13. A storage medium having stored thereon a computer program, characterized in that the computer program, when being executed by a processor, realizes the surgical robotic field troubleshooting method as recited in any one of claims 9-12.